Kaip naudoti kelis GPU su „TensorFlow“ (reikia jokių kodo pakeitimų)

Turinio apžvalga Įdiegti Apžvalga Logging įrenginio vieta Rankinis įrenginio montavimas GPU atminties augimo apribojimas Vieno GPU naudojimas kelių GPU sistemoje Naudojant daug GPU „TensorFlow“ kodas ir modeliai bus skaidriai paleisti vienoje GPU be jokių kodo pakeitimų. tf.keras Pastaba : Naudokite tf.config.list_physical_devices ('GPU') norėdami patvirtinti, kad „TensorFlow“ naudoja GPU. Naudokite patvirtinti, kad „TensorFlow“ naudoja GPU. Note: tf.config.list_physical_devices('GPU') Paprasčiausias būdas paleisti kelias GPU, vienoje ar keliose mašinose, yra naudoti paskirstymo strategijas. Šis vadovas skirtas vartotojams, kurie išbandė šiuos metodus ir nustatė, kad jiems reikia smulkiai kontroliuoti, kaip TensorFlow naudoja GPU. Gydytojas TensorFlow GPU našumo optimizavimas Įdiegti Įsitikinkite, kad turite įdiegtą naujausią TensorFlow gpu versiją. import tensorflow as tf print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU'))) 2024-08-15 02:53:40.344028: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-08-15 02:53:40.365851: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-08-15 02:53:40.372242: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered Num GPUs Available: 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1723690422.944962 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.948934 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.952655 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.955880 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.967120 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.970596 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.973980 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.976984 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.979869 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.983344 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.986754 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.989690 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 Apžvalga TensorFlow palaiko skaičiavimus įvairių tipų įrenginiuose, įskaitant CPU ir GPU. „/device:CPU:0“: Jūsų kompiuterio procesorius. „/GPU:0“: trumpalaikis jūsų mašinos pirmojo GPU, matomo TensorFlow, ženklinimas. "/job:localhost/replica:0/task:0/device:GPU:1": Visiškai kvalifikuotas antrojo jūsų kompiuterio GPU pavadinimas, matomas TensorFlow. Jei TensorFlow operacijoje yra tiek CPU, tiek GPU įgyvendinimai, GPU įrenginys pagal numatytuosius nustatymus teikiamas pirmenybę, kai operacija priskiriama. turi CPU ir GPU branduolius ir sistemoje su įrenginiais ir Tačiau The įrenginys yra parinktas paleisti nebent aiškiai paprašysite jį paleisti kitu įrenginiu. tf.matmul CPU:0 GPU:0 GPU:0 tf.matmul Jei TensorFlow operacija neturi atitinkamo GPU įgyvendinimo, operacija grįžta į CPU įrenginį. turi tik vieną CPU branduolį, sistemoje su įrenginiais ir Tačiau The įrenginys yra parinktas paleisti Net jei buvo paprašyta paleisti į prietaisą . tf.cast CPU:0 GPU:0 CPU:0 tf.cast GPU:0 Logging įrenginio vieta Norėdami sužinoti, kuriems įrenginiams priskiriamos jūsų operacijos ir tensoriai, Įjungus įrenginio vietos įrašymą, atspausdinami bet kokie Tensor paskirstymai ar operacijos. tf.debugging.set_log_device_placement(True) tf.debugging.set_log_device_placement(True) # Create some tensors a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c = tf.matmul(a, b) print(c) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 I0000 00:00:1723690424.215487 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.217630 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.219585 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.221664 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.223723 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.225666 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.227528 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.229544 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.231494 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.233433 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.235295 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.237325 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.276919 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.278939 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.280845 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.282884 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.284977 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.286923 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.288779 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.290783 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.292741 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.295170 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.297460 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.299854 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 tf.Tensor( [[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32) Aukščiau esantis kodas spausdins nuorodą į buvo įvykdyta ant . MatMul GPU:0 Rankinis įrenginio montavimas Jei norite, kad tam tikra operacija būtų vykdoma jūsų pasirinktame įrenginyje, o ne tai, kas automatiškai parinkta jums, galite naudoti sukurti įrenginio kontekstą, ir visos operacijos toje kontekste bus vykdomos tame pačiame paskirtame įrenginyje. with tf.device tf.debugging.set_log_device_placement(True) # Place tensors on the CPU with tf.device('/CPU:0'): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) # Run on the GPU c = tf.matmul(a, b) print(c) Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 tf.Tensor( [[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32) Tai pamatysite dabar ir Jie skiriami Kadangi prietaisas nebuvo aiškiai nurodytas "TensorFlow" paleidimo laikas bus pasirinktas atsižvelgiant į operaciją ir turimus įrenginius ( šiame pavyzdyje) ir automatiškai kopijuoti tensorius tarp prietaisų, jei reikia. a b CPU:0 MatMul GPU:0 GPU atminties augimo apribojimas Pagal numatytuosius nustatymus „TensorFlow“ atpažįsta beveik visą visų GPU atmintį (atsižvelgiant į Tai daroma siekiant efektyviau panaudoti santykinai vertingus GPU atminties išteklius įrenginiuose, sumažinant atminties fragmentaciją. Metodą CUDA_VISIBLE_DEVICES tf.config.set_visible_devices gpus = tf.config.list_physical_devices('GPU') if gpus: # Restrict TensorFlow to only use the first GPU try: tf.config.set_visible_devices(gpus[0], 'GPU') logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPU") except RuntimeError as e: # Visible devices must be set before GPUs have been initialized print(e) Visible devices cannot be modified after being initialized Kai kuriais atvejais pageidautina, kad procesas paskirstytų tik dalį turimos atminties arba tik padidintų atminties naudojimą, kaip to reikia procesui. Pirmasis variantas yra įjungti atminties augimą skambinant , kuris bando paskirstyti tik tiek GPU atminties, kiek reikia paleidimo laiko paskirstymams: jis pradeda paskirstyti labai mažai atminties, o kai programa paleidžiama ir reikia daugiau GPU atminties, GPU atminties regionas išplečiamas TensorFlow procesui. Atmintis neišleidžiama, nes tai gali sukelti atminties fragmentaciją. tf.config.experimental.set_memory_growth gpus = tf.config.list_physical_devices('GPU') if gpus: try: # Currently, memory growth needs to be the same across GPUs for gpu in gpus: tf.config.experimental.set_memory_growth(gpu, True) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Memory growth must be set before GPUs have been initialized print(e) Physical devices cannot be modified after being initialized Kitas būdas įgalinti šią parinktį yra nustatyti aplinkos kintamąjį. dviejų Ši konfigūracija yra platforma specifinė. TF_FORCE_GPU_ALLOW_GROWTH true Antrasis metodas yra sukonfigūruoti virtualų GPU įrenginį su ir nustatykite kietą bendros atminties ribą, kurią reikia paskirstyti GPU. tf.config.set_logical_device_configuration gpus = tf.config.list_physical_devices('GPU') if gpus: # Restrict TensorFlow to only allocate 1GB of memory on the first GPU try: tf.config.set_logical_device_configuration( gpus[0], [tf.config.LogicalDeviceConfiguration(memory_limit=1024)]) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Virtual devices must be set before GPUs have been initialized print(e) Virtual devices cannot be modified after being initialized Tai naudinga, jei norite tikrai susieti GPU atminties kiekį su TensorFlow procesu. Vieno GPU naudojimas kelių GPU sistemoje Jei jūsų sistemoje yra daugiau nei vienas GPU, GPU su mažiausiu ID bus pasirinktas pagal numatytuosius nustatymus. tf.debugging.set_log_device_placement(True) try: # Specify an invalid GPU device with tf.device('/device:GPU:2'): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c = tf.matmul(a, b) except RuntimeError as e: print(e) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:2 Jei nurodytas įrenginys neegzistuoja, gausite : į . RuntimeError .../device:GPU:2 unknown device Jei norite, kad „TensorFlow“ automatiškai pasirinktų esamą ir palaikomą įrenginį operacijoms atlikti, jei nurodytas įrenginys neegzistuoja, galite paskambinti . tf.config.set_soft_device_placement(True) tf.config.set_soft_device_placement(True) tf.debugging.set_log_device_placement(True) # Creates some tensors a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c = tf.matmul(a, b) print(c) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 tf.Tensor( [[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32) Naudojant daug GPU Kelių GPU kūrimas leis modeliui išplėsti papildomus išteklius.Jei kuriate sistemą su vienu GPU, galite imituoti kelis GPU su virtualiais įrenginiais.Tai leidžia lengvai išbandyti daugelio GPU nustatymus nereikalaujant papildomų išteklių. gpus = tf.config.list_physical_devices('GPU') if gpus: # Create 2 virtual GPUs with 1GB memory each try: tf.config.set_logical_device_configuration( gpus[0], [tf.config.LogicalDeviceConfiguration(memory_limit=1024), tf.config.LogicalDeviceConfiguration(memory_limit=1024)]) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPU,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Virtual devices must be set before GPUs have been initialized print(e) Virtual devices cannot be modified after being initialized Kai paleidimo metu yra keli loginiai GPU, galite naudoti kelis GPU su arba rankiniu būdu. tf.distribute.Strategy su tf.distribute.Strategy Tf.distribute.Strategija Geriausia praktika naudojant kelis GPU yra naudoti Štai paprastas pavyzdys: tf.distribute.Strategy tf.debugging.set_log_device_placement(True) gpus = tf.config.list_logical_devices('GPU') strategy = tf.distribute.MirroredStrategy(gpus) with strategy.scope(): inputs = tf.keras.layers.Input(shape=(1,)) predictions = tf.keras.layers.Dense(1)(inputs) model = tf.keras.models.Model(inputs=inputs, outputs=predictions) model.compile(loss='mse', optimizer=tf.keras.optimizers.SGD(learning_rate=0.2)) INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2', '/job:localhost/replica:0/task:0/device:GPU:3') Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op FloorMod in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Cast in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op StatelessRandomGetKeyCounter in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0 product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:2 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:2 product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:1 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:3 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 FloorMod: (FloorMod): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Cast: (Cast): /job:localhost/replica:0/task:0/device:GPU:0 y_RetVal: (_DeviceRetval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 seed: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0 StatelessRandomGetKeyCounter: (StatelessRandomGetKeyCounter): /job:localhost/replica:0/task:0/device:GPU:0 key_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 counter_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 shape: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0 key: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 counter: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 alg: (_DeviceArg): /job:localhost/replica:0/taExecuting op StatelessRandomUniformV2 in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AddV2 in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 sk:0/device:CPU:0 StatelessRandomUniformV2: (StatelessRandomUniformV2): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Sub: (Sub): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Mul: (Mul): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AddV2: (AddV2): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:1 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:2 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 NoOp: (NoOp): /job:localhost/replica:0/task:0/device:GPU:0 dims: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Fill: (Fill): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:lExecuting op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0 ocalhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:1 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:2 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/deviExecuting op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Ši programa paleis jūsų modelio kopiją kiekviename GPU, padalindama įvesties duomenis tarp jų, taip pat žinomų kaip " „ Duomenų paralelizmas Norėdami gauti daugiau informacijos apie platinimo strategijas, peržiūrėkite vadovą . čia Rankinė išdėstymas veikia pagal dangtelį, kopijuojant skaičiavimus įvairiuose įrenginiuose. Galite rankiniu būdu įgyvendinti replikaciją, sukurdami modelį kiekviename GPU. Pavyzdžiui: tf.distribute.Strategy tf.debugging.set_log_device_placement(True) gpus = tf.config.list_logical_devices('GPU') if gpus: # Replicate your computation on multiple GPUs c = [] for gpu in gpus: with tf.device(gpu.name): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c.append(tf.matmul(a, b)) with tf.device('/CPU:0'): matmul_sum = tf.add_n(c) print(matmul_sum) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AddN in device /job:localhost/replica:0/task:0/device:CPU:0 tf.Tensor( [[ 88. 112.] [196. 256.]], shape=(2, 2), dtype=float32) Iš pradžių paskelbtas „TensorFlow“ svetainėje, šis straipsnis čia pasirodo pagal naują antraštę ir yra licencijuotas pagal CC BY 4.0. Iš pradžių paskelbtas „TensorFlow“ svetainėje, šis straipsnis čia pasirodo pagal naują antraštę ir yra licencijuotas pagal CC BY 4.0. „TensorFlow“ „TensorFlow“