TensorFlow ilə Multiple GPU-ları necə istifadə etmək (kodun dəyişdirilməsinə ehtiyac yoxdur)

Əsas səhifə Setmə İncəsənət Lojistik cihazların yerləşdirilməsi Manual yerləşdirilməsi GPU memoriyasının artırılması Multi-GPU sistemində yalnız bir GPU istifadə etmək Bir neçə GPU TensorFlow kodu və Modeller yalnız bir GPU-da, kod dəyişdirilməsinə ehtiyac yoxdur. tf.keras Xatırladaq ki, TensorFlow-un GPU-nı istifadə etdiyini təsdiq etmək üçün tf.config.list_physical_devices('GPU') istifadə edin. Kullanım TensorFlow GPU-ni istifadə edir. Note: tf.config.list_physical_devices('GPU') Bir və ya bir çox kompüterdə bir çox GPU-da çalışmaq üçün ən asan yolu Distribution Strategies istifadə etməkdir. VVD - Hollandiyada futbolçu bu adla tanımır, orada VVD daha çox mərkəz-sağı təmsilən edən siyasi partiyanın adının qısaltması kimi bilinir - artıq sorğu-suala ehtiyacı olmayan ulduzdu. Girişçi TensorFlow GPU performansı Setmə Ən son TensorFlow gpu versiyasını yüklüyünüzə baxın. import tensorflow as tf print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU'))) 2024-08-15 02:53:40.344028: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-08-15 02:53:40.365851: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-08-15 02:53:40.372242: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered Num GPUs Available: 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1723690422.944962 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.948934 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.952655 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.955880 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.967120 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.970596 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.973980 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.976984 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.979869 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.983344 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.986754 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.989690 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 İncəsənət VVD - Hollandiyada futbolçu bu adla tanımır, orada VVD daha çox mərkəz-sağı təmsilən edən siyasi partiyanın adının qısaltması kimi bilinir - artıq sorğu-suala ehtiyacı olmayan ulduzdu. “/device:CPU:0”: Makinenizin CPU-si "/GPU:0": TensorFlow üçün görünür olan maşının ilk GPU-ni üçün qısa işarə. "/job:localhost/replica:0/task:0/device:GPU:1": TensorFlow üçün görünür olan maşının ikinci GPU-nin tamamilə qiymətləndirilmiş adı. VVD - Hollandiyada futbolçu bu adla tanımır, orada VVD daha çox mərkəz-sağı təmsilən edən siyasi partiyanın adının qısaltması kimi bilinir - artıq sorğu-suala ehtiyacı olmayan ulduzdu. CPU və GPU kernelləri və cihazları olan bir sistemdə və “The Dərslər seçilməlidir Başqa bir cihazda çalışmaq istəməsəniz. tf.matmul CPU:0 GPU:0 GPU:0 tf.matmul Əgər bir TensorFlow işlemi ilə bağlı GPU implementasiyası yoxdursa, işlemi CPU cihazına qaytarır. Yalnız bir CPU kernel var, sistemdə cihazlar var. və “The Dərslər seçilməlidir Buna baxmayaraq, bir neçə ildir ki, bu aparatı tf.cast CPU:0 GPU:0 CPU:0 tf.cast GPU:0 Lojistik cihazların yerləşdirilməsi İşləmələrinizin və tensorlarınızın hansı cihazlara göndərildiyini görmək üçün, Bu barədə “The New York Times” xəbər verir ki, bu barədə “The New York Times” xəbər verir ki, bu barədə “The New York Times” xəbər verir. tf.debugging.set_log_device_placement(True) tf.debugging.set_log_device_placement(True) # Create some tensors a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c = tf.matmul(a, b) print(c) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 I0000 00:00:1723690424.215487 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.217630 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.219585 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.221664 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.223723 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.225666 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.227528 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.229544 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.231494 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.233433 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.235295 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.237325 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.276919 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.278939 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.280845 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.282884 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.284977 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.286923 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.288779 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.290783 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.292741 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.295170 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.297460 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.299854 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 tf.Tensor( [[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32) Üçüncüsü, bu kodu göstərmək olar ki, Əfqanıstanda öldürüldü . MatMul GPU:0 Manual yerləşdirilməsi Əgər sizin üçün otomatik olaraq seçilən şeylər yerine seçdiyiniz bir cihazda konkret bir işləmək istəyirsinizsə, Belə ki, bütün proseslər bu kontekstdə və bu kontekstdə aparılacaq. with tf.device tf.debugging.set_log_device_placement(True) # Place tensors on the CPU with tf.device('/CPU:0'): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) # Run on the GPU c = tf.matmul(a, b) print(c) Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 tf.Tensor( [[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32) Artıq bunu görəcəksiniz və Onların adlandırılması Çünki bu sistemlər açıq-aşkar birləşdirilmədi. “Qazaxıstan” “Qazaxıstan” “Qazaxıstan” “Qazaxıstan” “Qazaxıstan” “Qazaxıstan” “Qazaxıstan” “Qazaxıstan” “Qazaxıstan” “Qazaxıstan” “Qazaxıstan” “Qazaxıstan” “Qazaxıstanda” “Qazaxıstanda” “Qazaxıstanda” “Qazaxıstanda” “Qazaxıstanda” “Qazaxıstanda” “Qazaxıstanda” “Qazaxıstanda” “Qazaxıstanda” “Qazaxıstanda” “Qazaxıstanda” “Qazaxıstanda” “Qazaxıstanda” “Qazaxıstanda” “Q İstehsalçılıq prosesləri aparılır və avtomatik olaraq aparıcılar arasında kopyalanır. a b CPU:0 MatMul GPU:0 GPU memoriyasının artırılması TensorFlow hər bir GPU-nin ( VVD - Hollandiyada futbolçu bu adla tanımır, orada VVD daha çox mərkəz-sağı təmsilən edən siyasi partiyanın adının qısaltması kimi bilinir - artıq sorğu-suala ehtiyacı olmayan ulduzdu. metodları CUDA_VISIBLE_DEVICES tf.config.set_visible_devices gpus = tf.config.list_physical_devices('GPU') if gpus: # Restrict TensorFlow to only use the first GPU try: tf.config.set_visible_devices(gpus[0], 'GPU') logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPU") except RuntimeError as e: # Visible devices must be set before GPUs have been initialized print(e) Visible devices cannot be modified after being initialized VVD - Hollandiyada futbolçu bu adla tanımır, orada VVD daha çox mərkəz-sağı təmsilən edən siyasi partiyanın adının qısaltması kimi bilinir - artıq sorğu-suala ehtiyacı olmayan ulduzdu. Birincisi, memoriya artırmaq üçün istifadə etmək lazımdır. VVD - Hollandiyada futbolçu bu adla tanımır, orada VVD daha çox mərkəz-sağı təmsilən edən siyasi partiyanın adının qısaltması kimi bilinir - artıq sorğu-suala ehtiyacı olmayan ulduzdu. tf.config.experimental.set_memory_growth gpus = tf.config.list_physical_devices('GPU') if gpus: try: # Currently, memory growth needs to be the same across GPUs for gpu in gpus: tf.config.experimental.set_memory_growth(gpu, True) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Memory growth must be set before GPUs have been initialized print(e) Physical devices cannot be modified after being initialized Bu variantı seçmək üçün başqa bir yolu isə çevrimiçi variabilin İki Bu konfigurasiya platforma spesifikdir. TF_FORCE_GPU_ALLOW_GROWTH true İkinci metod virtual GPU cihazını configure etməkdir. GPU-ya göndəriləcək toplam memoriya limitini sabitləşdirin. tf.config.set_logical_device_configuration gpus = tf.config.list_physical_devices('GPU') if gpus: # Restrict TensorFlow to only allocate 1GB of memory on the first GPU try: tf.config.set_logical_device_configuration( gpus[0], [tf.config.LogicalDeviceConfiguration(memory_limit=1024)]) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Virtual devices must be set before GPUs have been initialized print(e) Virtual devices cannot be modified after being initialized VVD - Hollandiyada futbolçu bu adla tanımır, orada VVD daha çox mərkəz-sağı təmsilən edən siyasi partiyanın adının qısaltması kimi bilinir - artıq sorğu-suala ehtiyacı olmayan ulduzdu. Multi-GPU sistemində yalnız bir GPU istifadə etmək Əgər sisteminizdə birdən çox GPU-niz varsa, ən aşağı ID-ni olan GPU-ni standart olaraq seçəcəksiniz. tf.debugging.set_log_device_placement(True) try: # Specify an invalid GPU device with tf.device('/device:GPU:2'): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c = tf.matmul(a, b) except RuntimeError as e: print(e) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:2 İstədiklərinizin bir hissəsi yoxdursa, bir Şirkət : . RuntimeError .../device:GPU:2 unknown device Əgər TensorFlow-un işləmək üçün var olan və kömək olunan bir cihazı otomatik seçmək istəyirsənsə, müəyyən olanı yoxdursa, telefonla danışın. . tf.config.set_soft_device_placement(True) tf.config.set_soft_device_placement(True) tf.debugging.set_log_device_placement(True) # Creates some tensors a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c = tf.matmul(a, b) print(c) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 tf.Tensor( [[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32) Bir çox GPU-lar Bir çox GPU-nin yaradılması bir modelin daha çox resurslarla genişləndirilməsinə imkan verir. Tek bir GPU-nin yaradılması ilə bir sistemdə çalışırsanız, virtual cihazlarla bir çox GPU-ni simülə edə bilərsiniz. gpus = tf.config.list_physical_devices('GPU') if gpus: # Create 2 virtual GPUs with 1GB memory each try: tf.config.set_logical_device_configuration( gpus[0], [tf.config.LogicalDeviceConfiguration(memory_limit=1024), tf.config.LogicalDeviceConfiguration(memory_limit=1024)]) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPU,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Virtual devices must be set before GPUs have been initialized print(e) Virtual devices cannot be modified after being initialized Bir neçə dəqiqə vaxtı bitirdikdən sonra, bir neçə dəqiqə vaxtı bitirirsiniz. Ya da manualdır. tf.distribute.Strategy ilə tf.distribute.Strategy Tq.az xəbər verir ki, strateji Multi-GPU istifadə etmək üçün ən yaxşı praktika Burada basit bir nümunə var: tf.distribute.Strategy tf.debugging.set_log_device_placement(True) gpus = tf.config.list_logical_devices('GPU') strategy = tf.distribute.MirroredStrategy(gpus) with strategy.scope(): inputs = tf.keras.layers.Input(shape=(1,)) predictions = tf.keras.layers.Dense(1)(inputs) model = tf.keras.models.Model(inputs=inputs, outputs=predictions) model.compile(loss='mse', optimizer=tf.keras.optimizers.SGD(learning_rate=0.2)) INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2', '/job:localhost/replica:0/task:0/device:GPU:3') Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op FloorMod in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Cast in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op StatelessRandomGetKeyCounter in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0 product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:2 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:2 product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:1 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:3 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 FloorMod: (FloorMod): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Cast: (Cast): /job:localhost/replica:0/task:0/device:GPU:0 y_RetVal: (_DeviceRetval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 seed: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0 StatelessRandomGetKeyCounter: (StatelessRandomGetKeyCounter): /job:localhost/replica:0/task:0/device:GPU:0 key_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 counter_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 shape: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0 key: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 counter: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 alg: (_DeviceArg): /job:localhost/replica:0/taExecuting op StatelessRandomUniformV2 in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AddV2 in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 sk:0/device:CPU:0 StatelessRandomUniformV2: (StatelessRandomUniformV2): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Sub: (Sub): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Mul: (Mul): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AddV2: (AddV2): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:1 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:2 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 NoOp: (NoOp): /job:localhost/replica:0/task:0/device:GPU:0 dims: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Fill: (Fill): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:lExecuting op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0 ocalhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:1 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:2 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/deviExecuting op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Bu program hər bir GPU-da modelinizin bir kopyasını çalışdıracaq və input verilərini onların arasında bölüşəcək. “Qəbələ” Bilgi paralelliyi Daha çox məlumat üçün dağıtım strategiyası, baxın Guide . Burada Manualdırma Xatırladaq ki, bir neçə ildir ki, bu problemlər həll olunacaq, bir neçə ildir ki, bu problemlər həll olunacaq. tf.distribute.Strategy tf.debugging.set_log_device_placement(True) gpus = tf.config.list_logical_devices('GPU') if gpus: # Replicate your computation on multiple GPUs c = [] for gpu in gpus: with tf.device(gpu.name): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c.append(tf.matmul(a, b)) with tf.device('/CPU:0'): matmul_sum = tf.add_n(c) print(matmul_sum) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AddN in device /job:localhost/replica:0/task:0/device:CPU:0 tf.Tensor( [[ 88. 112.] [196. 256.]], shape=(2, 2), dtype=float32) Orijinal olaraq TensorFlow saytında yayımlanan bu yazı burada yeni bir başlıq altında görünür və CC BY 4.0 altında lisenziyadır. Orijinal olaraq TensorFlow saytında yayımlanan bu yazı burada yeni bir başlıq altında görünür və CC BY 4.0 altında lisenziyadır. TensorFlow xəritədə TensorFlow xəritədə