Com utilitzar múltiples GPU amb TensorFlow (no es requereixen canvis de codi)

Contingut general Establiment Visió general Col·locació de dispositius Instal·lació manual Limita el creixement de la memòria GPU Utilitzar una única GPU en un sistema multi-GPU Utilització de múltiples GPU El codi TensorFlow, i Els models s'executaran de manera transparent en una única GPU sense necessitat de canvis de codi. tf.keras Nota: Utilitzeu tf.config.list_physical_devices ('GPU') per confirmar que TensorFlow està utilitzant la GPU. utilitzar confirmar que TensorFlow està utilitzant la GPU. Note: tf.config.list_physical_devices('GPU') La manera més senzilla d'executar en múltiples GPU, en una o moltes màquines, és utilitzar Estratègies de distribució. Aquesta guia és per als usuaris que han provat aquests enfocaments i han trobat que necessiten un control fi sobre com TensorFlow utilitza la GPU. La guia. Optimització del rendiment de la GPU TensorFlow Establiment Assegureu-vos que teniu l'última versió de TensorFlow gpu instal·lada. import tensorflow as tf print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU'))) 2024-08-15 02:53:40.344028: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-08-15 02:53:40.365851: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-08-15 02:53:40.372242: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered Num GPUs Available: 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1723690422.944962 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.948934 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.952655 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.955880 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.967120 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.970596 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.973980 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.976984 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.979869 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.983344 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.986754 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.989690 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 Visió general TensorFlow és compatible amb l'execució de càlculs en una varietat de tipus de dispositius, incloent CPU i GPU. "/device:CPU:0": La CPU de la seva màquina. "/GPU:0": notació a mà curta per a la primera GPU de la màquina que és visible a TensorFlow. "/job:localhost/replica:0/task:0/device:GPU:1": Nom completament qualificat de la segona GPU de la seva màquina que és visible a TensorFlow. Si una operació TensorFlow té les dues implementacions de CPU i GPU, per defecte, el dispositiu GPU es prioritza quan s'assigna l'operació. disposa de nuclis de CPU i GPU i en un sistema amb dispositius i , el El dispositiu està seleccionat per executar llevat que sol·liciti explícitament que s'executi en un altre dispositiu. tf.matmul CPU:0 GPU:0 GPU:0 tf.matmul Si una operació TensorFlow no té una implementació de GPU corresponent, llavors l'operació cau de nou al dispositiu CPU. només té un nucli de la CPU, en un sistema amb dispositius i , el El dispositiu està seleccionat per executar Fins i tot si se li demana que surti al El dispositiu. tf.cast CPU:0 GPU:0 CPU:0 tf.cast GPU:0 Col·locació de dispositius Per saber a quins dispositius estan assignades les seves operacions i tensors, poseu Permetre el registre de la col·locació del dispositiu fa que s'imprimeixin totes les al·locacions o operacions del Tensor. tf.debugging.set_log_device_placement(True) tf.debugging.set_log_device_placement(True) # Create some tensors a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c = tf.matmul(a, b) print(c) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 I0000 00:00:1723690424.215487 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.217630 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.219585 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.221664 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.223723 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.225666 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.227528 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.229544 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.231494 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.233433 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.235295 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.237325 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.276919 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.278939 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.280845 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.282884 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.284977 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.286923 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.288779 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.290783 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.292741 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.295170 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.297460 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.299854 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 tf.Tensor( [[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32) El codi anterior indicarà la Va ser executat en . MatMul GPU:0 Instal·lació manual Si voleu que una operació específica s'executi en un dispositiu de la vostra elecció en lloc del que es selecciona automàticament per a vostè, podeu utilitzar per crear un context de dispositius, i totes les operacions dins d'aquest context s'executaran en el mateix dispositiu designat. with tf.device tf.debugging.set_log_device_placement(True) # Place tensors on the CPU with tf.device('/CPU:0'): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) # Run on the GPU c = tf.matmul(a, b) print(c) Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 tf.Tensor( [[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32) Ho veuràs ara i Estan assignats a Com que un dispositiu no es va especificar explícitament per a la l'operació, el temps d'execució de TensorFlow triarà un d'acord amb l'operació i els dispositius disponibles ( en aquest exemple) i copiar automàticament tensors entre dispositius si és necessari. a b CPU:0 MatMul GPU:0 Limita el creixement de la memòria GPU Per defecte, TensorFlow mapes gairebé tota la memòria de GPU de totes les GPU (subjecte a ) visible al procés. Això es fa per fer un ús més eficient dels recursos de memòria de GPU relativament valuosos en els dispositius mitjançant la reducció de la fragmentació de la memòria. El mètode. CUDA_VISIBLE_DEVICES tf.config.set_visible_devices gpus = tf.config.list_physical_devices('GPU') if gpus: # Restrict TensorFlow to only use the first GPU try: tf.config.set_visible_devices(gpus[0], 'GPU') logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPU") except RuntimeError as e: # Visible devices must be set before GPUs have been initialized print(e) Visible devices cannot be modified after being initialized En alguns casos, és desitjable que el procés només allotgi un subconjunt de la memòria disponible, o que només augmenti l'ús de la memòria segons sigui necessari pel procés. La primera opció és activar el creixement de la memòria trucant , que intenta allotjar només tanta memòria GPU com sigui necessari per a les alocacions de temps d'execució: comença a allotjar molt poca memòria, i a mesura que el programa s'executa i es necessita més memòria GPU, la regió de memòria GPU s'estén per al procés TensorFlow. tf.config.experimental.set_memory_growth gpus = tf.config.list_physical_devices('GPU') if gpus: try: # Currently, memory growth needs to be the same across GPUs for gpu in gpus: tf.config.experimental.set_memory_growth(gpu, True) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Memory growth must be set before GPUs have been initialized print(e) Physical devices cannot be modified after being initialized Una altra manera d'activar aquesta opció és establir la variable ambiental. Dues Aquesta configuració és específica de la plataforma. TF_FORCE_GPU_ALLOW_GROWTH true El segon mètode consisteix a configurar un dispositiu GPU virtual amb i establir un límit dur sobre la memòria total a allotjar a la GPU. tf.config.set_logical_device_configuration gpus = tf.config.list_physical_devices('GPU') if gpus: # Restrict TensorFlow to only allocate 1GB of memory on the first GPU try: tf.config.set_logical_device_configuration( gpus[0], [tf.config.LogicalDeviceConfiguration(memory_limit=1024)]) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Virtual devices must be set before GPUs have been initialized print(e) Virtual devices cannot be modified after being initialized Això és útil si voleu vincular realment la quantitat de memòria de GPU disponible al procés TensorFlow. Utilitzar una única GPU en un sistema multi-GPU Si teniu més d'una GPU al vostre sistema, la GPU amb la ID més baixa es seleccionarà per defecte. tf.debugging.set_log_device_placement(True) try: # Specify an invalid GPU device with tf.device('/device:GPU:2'): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c = tf.matmul(a, b) except RuntimeError as e: print(e) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:2 Si el dispositiu que heu especificat no existeix, obtindreu una · : . RuntimeError .../device:GPU:2 unknown device Si voleu que TensorFlow seleccioni automàticament un dispositiu existent i compatible per executar les operacions en cas que no existeixi, podeu trucar . tf.config.set_soft_device_placement(True) tf.config.set_soft_device_placement(True) tf.debugging.set_log_device_placement(True) # Creates some tensors a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c = tf.matmul(a, b) print(c) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 tf.Tensor( [[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32) Utilització de múltiples GPU Desenvolupar per a múltiples GPU permetrà que un model s'escalifiqui amb els recursos addicionals. Si es desenvolupa en un sistema amb una única GPU, es poden simular múltiples GPU amb dispositius virtuals. Això permet provar fàcilment les configuracions de múltiples GPU sense necessitar recursos addicionals. gpus = tf.config.list_physical_devices('GPU') if gpus: # Create 2 virtual GPUs with 1GB memory each try: tf.config.set_logical_device_configuration( gpus[0], [tf.config.LogicalDeviceConfiguration(memory_limit=1024), tf.config.LogicalDeviceConfiguration(memory_limit=1024)]) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPU,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Virtual devices must be set before GPUs have been initialized print(e) Virtual devices cannot be modified after being initialized Una vegada que hi hagi múltiples GPU lògics disponibles per al temps d'execució, podeu utilitzar les múltiples GPU amb o amb col·locació manual. tf.distribute.Strategy amb tf.distribute.Estratègia tf.distribute.Estratègia La millor pràctica per utilitzar múltiples GPU és utilitzar Aquí teniu un exemple senzill: tf.distribute.Strategy tf.debugging.set_log_device_placement(True) gpus = tf.config.list_logical_devices('GPU') strategy = tf.distribute.MirroredStrategy(gpus) with strategy.scope(): inputs = tf.keras.layers.Input(shape=(1,)) predictions = tf.keras.layers.Dense(1)(inputs) model = tf.keras.models.Model(inputs=inputs, outputs=predictions) model.compile(loss='mse', optimizer=tf.keras.optimizers.SGD(learning_rate=0.2)) INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2', '/job:localhost/replica:0/task:0/device:GPU:3') Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op FloorMod in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Cast in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op StatelessRandomGetKeyCounter in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0 product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:2 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:2 product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:1 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:3 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 FloorMod: (FloorMod): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Cast: (Cast): /job:localhost/replica:0/task:0/device:GPU:0 y_RetVal: (_DeviceRetval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 seed: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0 StatelessRandomGetKeyCounter: (StatelessRandomGetKeyCounter): /job:localhost/replica:0/task:0/device:GPU:0 key_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 counter_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 shape: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0 key: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 counter: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 alg: (_DeviceArg): /job:localhost/replica:0/taExecuting op StatelessRandomUniformV2 in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AddV2 in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 sk:0/device:CPU:0 StatelessRandomUniformV2: (StatelessRandomUniformV2): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Sub: (Sub): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Mul: (Mul): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AddV2: (AddV2): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:1 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:2 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 NoOp: (NoOp): /job:localhost/replica:0/task:0/device:GPU:0 dims: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Fill: (Fill): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:lExecuting op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0 ocalhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:1 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:2 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/deviExecuting op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Aquest programa executarà una còpia del vostre model en cada GPU, dividint les dades d'entrada entre ells, també conegudes com " » » Paral·lelisme de dades Per a més informació sobre les estratègies de distribució, consulteu la guia . Aquí Col·locació manual funciona sota el capó replicant la computació a través dels dispositius. Pots implementar manualment la replicació construint el teu model en cada GPU. tf.distribute.Strategy tf.debugging.set_log_device_placement(True) gpus = tf.config.list_logical_devices('GPU') if gpus: # Replicate your computation on multiple GPUs c = [] for gpu in gpus: with tf.device(gpu.name): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c.append(tf.matmul(a, b)) with tf.device('/CPU:0'): matmul_sum = tf.add_n(c) print(matmul_sum) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AddN in device /job:localhost/replica:0/task:0/device:CPU:0 tf.Tensor( [[ 88. 112.] [196. 256.]], shape=(2, 2), dtype=float32) Originalment publicat al lloc web de TensorFlow, aquest article apareix aquí sota un nou títol i està llicenciat sota CC BY 4.0. Originalment publicat al lloc web de TensorFlow, aquest article apareix aquí sota un nou títol i està llicenciat sota CC BY 4.0. TensorFlow TensorFlow