Πώς να χρησιμοποιήσετε πολλαπλές GPUs με TensorFlow (δεν απαιτούνται αλλαγές κώδικα)

Περιεχόμενο Επισκόπηση εγκατάσταση Επισκόπηση Λογισμικό τοποθέτησης Εγχειρίδιο τοποθέτησης Περιορισμός της ανάπτυξης μνήμης GPU Χρήση μίας μονάδας GPU σε ένα σύστημα πολλαπλών GPU Χρήση πολλαπλών GPU Ο κώδικας TensorFlow, και Τα μοντέλα θα τρέχουν με διαφάνεια σε μία μόνο GPU χωρίς να απαιτούνται αλλαγές κώδικα. tf.keras Σημείωση: Χρησιμοποιήστε tf.config.list_physical_devices ('GPU') για να επιβεβαιώσετε ότι το TensorFlow χρησιμοποιεί την GPU. Χρησιμοποιήστε για να επιβεβαιώσετε ότι το TensorFlow χρησιμοποιεί την GPU. Note: tf.config.list_physical_devices('GPU') Ο ευκολότερος τρόπος για να τρέξετε σε πολλαπλές GPUs, σε ένα ή περισσότερα μηχανήματα, είναι να χρησιμοποιήσετε Στρατηγικές Διανομής. Αυτός ο οδηγός είναι για χρήστες που έχουν δοκιμάσει αυτές τις προσεγγίσεις και διαπίστωσαν ότι χρειάζονται λεπτομερή έλεγχο του τρόπου με τον οποίο το TensorFlow χρησιμοποιεί την GPU. Οδηγός . Βελτιστοποίηση της απόδοσης της GPU TensorFlow εγκατάσταση Βεβαιωθείτε ότι έχετε εγκαταστήσει την τελευταία έκδοση του TensorFlow gpu. import tensorflow as tf print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU'))) 2024-08-15 02:53:40.344028: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-08-15 02:53:40.365851: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-08-15 02:53:40.372242: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered Num GPUs Available: 4 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1723690422.944962 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.948934 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.952655 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.955880 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.967120 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.970596 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.973980 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.976984 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.979869 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.983344 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.986754 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690422.989690 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 Επισκόπηση Το TensorFlow υποστηρίζει την εκτέλεση υπολογισμών σε διάφορους τύπους συσκευών, συμπεριλαμβανομένων των CPU και GPU. «/device:CPU:0»: Η CPU του υπολογιστή σας. "/GPU:0": Σύντομη σήμανση για την πρώτη GPU της μηχανής σας που είναι ορατή στο TensorFlow. "/job:localhost/replica:0/task:0/device:GPU:1": Πλήρως κατάλληλο όνομα της δεύτερης GPU του υπολογιστή σας που είναι ορατή στο TensorFlow. Εάν μια λειτουργία TensorFlow έχει τόσο τις εφαρμογές CPU όσο και GPU, από προεπιλογή, η συσκευή GPU δίνεται προτεραιότητα όταν εκχωρείται η λειτουργία. έχει και τους πυρήνες CPU και GPU και σε ένα σύστημα με συσκευές και Η Η συσκευή έχει επιλεγεί για να τρέξει Εκτός αν ζητήσετε ρητά να εκτελεστεί σε άλλη συσκευή. tf.matmul CPU:0 GPU:0 GPU:0 tf.matmul Εάν μια λειτουργία TensorFlow δεν έχει αντίστοιχη υλοποίηση GPU, τότε η λειτουργία επιστρέφει στη συσκευή CPU. έχει μόνο έναν πυρήνα CPU, σε ένα σύστημα με συσκευές και Η Η συσκευή έχει επιλεγεί για να τρέξει Ακόμη και αν ζητηθεί να τρέξει στο Η συσκευή. tf.cast CPU:0 GPU:0 CPU:0 tf.cast GPU:0 Λογισμικό τοποθέτησης Για να μάθετε σε ποιες συσκευές ανατίθενται οι λειτουργίες και οι τεντωτές σας, τοποθετήστε Η ενεργοποίηση της καταγραφής της τοποθέτησης της συσκευής προκαλεί την εκτύπωση οποιωνδήποτε κατανομών ή λειτουργιών του Tensor. tf.debugging.set_log_device_placement(True) tf.debugging.set_log_device_placement(True) # Create some tensors a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c = tf.matmul(a, b) print(c) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 I0000 00:00:1723690424.215487 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.217630 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.219585 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.221664 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.223723 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.225666 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.227528 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.229544 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.231494 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.233433 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.235295 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.237325 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.276919 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.278939 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.280845 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.282884 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.284977 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.286923 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.288779 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.290783 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.292741 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.295170 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.297460 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723690424.299854 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 tf.Tensor( [[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32) Ο ανωτέρω κώδικας θα εκτυπώσει μια ένδειξη της Εκτελέστηκε σε . MatMul GPU:0 Εγχειρίδιο τοποθέτησης Εάν θέλετε μια συγκεκριμένη λειτουργία να εκτελείται σε μια συσκευή της επιλογής σας αντί για αυτό που έχει επιλεγεί αυτόματα για εσάς, μπορείτε να χρησιμοποιήσετε να δημιουργήσετε ένα πλαίσιο συσκευής και όλες οι λειτουργίες σε αυτό το πλαίσιο θα εκτελούνται στην ίδια καθορισμένη συσκευή. with tf.device tf.debugging.set_log_device_placement(True) # Place tensors on the CPU with tf.device('/CPU:0'): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) # Run on the GPU c = tf.matmul(a, b) print(c) Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 tf.Tensor( [[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32) Θα το δείτε τώρα και Ανατίθενται σε Δεδομένου ότι μια συσκευή δεν καθορίστηκε ρητά για την η λειτουργία, ο χρόνος εκτέλεσης του TensorFlow θα επιλέξει έναν με βάση τη λειτουργία και τις διαθέσιμες συσκευές ( σε αυτό το παράδειγμα) και να αντιγράψετε αυτόματα τους τενζόντες μεταξύ των συσκευών, εάν είναι απαραίτητο. a b CPU:0 MatMul GPU:0 Περιορισμός της ανάπτυξης μνήμης GPU Από προεπιλογή, το TensorFlow χαρτογραφεί σχεδόν όλη τη μνήμη GPU όλων των GPU (υπό την επιφύλαξη Αυτό γίνεται για να χρησιμοποιήσετε πιο αποτελεσματικά τους σχετικά πολύτιμους πόρους μνήμης GPU στις συσκευές μειώνοντας τον κατακερματισμό της μνήμης.Για να περιορίσετε το TensorFlow σε ένα συγκεκριμένο σύνολο GPU, χρησιμοποιήστε το τη μέθοδο. CUDA_VISIBLE_DEVICES tf.config.set_visible_devices gpus = tf.config.list_physical_devices('GPU') if gpus: # Restrict TensorFlow to only use the first GPU try: tf.config.set_visible_devices(gpus[0], 'GPU') logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPU") except RuntimeError as e: # Visible devices must be set before GPUs have been initialized print(e) Visible devices cannot be modified after being initialized Σε ορισμένες περιπτώσεις, είναι επιθυμητό για τη διαδικασία να εκχωρήσει μόνο ένα υποσύνολο της διαθέσιμης μνήμης, ή να αυξήσει μόνο τη χρήση μνήμης όπως απαιτείται από τη διαδικασία. Η πρώτη επιλογή είναι να ενεργοποιήσετε την ανάπτυξη μνήμης κάνοντας κλήση , η οποία προσπαθεί να εκχωρήσει μόνο τόσο μνήμη GPU όσο χρειάζεται για τις κατανομές χρόνου εκτέλεσης: αρχίζει να εκχωρεί πολύ λίγη μνήμη και καθώς το πρόγραμμα εκτελείται και χρειάζεται περισσότερη μνήμη GPU, η περιοχή μνήμης GPU επεκτείνεται για τη διαδικασία TensorFlow. Μνήμη δεν απελευθερώνεται καθώς μπορεί να οδηγήσει σε κατακερματισμό μνήμης. tf.config.experimental.set_memory_growth gpus = tf.config.list_physical_devices('GPU') if gpus: try: # Currently, memory growth needs to be the same across GPUs for gpu in gpus: tf.config.experimental.set_memory_growth(gpu, True) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Memory growth must be set before GPUs have been initialized print(e) Physical devices cannot be modified after being initialized Ένας άλλος τρόπος για να ενεργοποιήσετε αυτήν την επιλογή είναι να ορίσετε την περιβαλλοντική μεταβλητή Δύο Αυτή η διαμόρφωση είναι συγκεκριμένη πλατφόρμα. TF_FORCE_GPU_ALLOW_GROWTH true Η δεύτερη μέθοδος είναι να ρυθμίσετε μια εικονική συσκευή GPU με και ορίστε ένα σκληρό όριο για τη συνολική μνήμη που πρέπει να διατεθεί στην GPU. tf.config.set_logical_device_configuration gpus = tf.config.list_physical_devices('GPU') if gpus: # Restrict TensorFlow to only allocate 1GB of memory on the first GPU try: tf.config.set_logical_device_configuration( gpus[0], [tf.config.LogicalDeviceConfiguration(memory_limit=1024)]) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Virtual devices must be set before GPUs have been initialized print(e) Virtual devices cannot be modified after being initialized Αυτό είναι χρήσιμο εάν θέλετε πραγματικά να δεσμεύσετε την ποσότητα της μνήμης GPU που είναι διαθέσιμη στη διαδικασία TensorFlow. Χρήση μίας μονάδας GPU σε ένα σύστημα πολλαπλών GPU Εάν έχετε περισσότερες από μία GPU στο σύστημά σας, η GPU με το χαμηλότερο ID θα επιλεγεί από προεπιλογή. tf.debugging.set_log_device_placement(True) try: # Specify an invalid GPU device with tf.device('/device:GPU:2'): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c = tf.matmul(a, b) except RuntimeError as e: print(e) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:2 Εάν η συσκευή που καθορίσατε δεν υπάρχει, θα λάβετε ένα : : . RuntimeError .../device:GPU:2 unknown device Εάν θέλετε το TensorFlow να επιλέξει αυτόματα μια υπάρχουσα και υποστηριζόμενη συσκευή για να εκτελέσει τις λειτουργίες σε περίπτωση που η καθορισμένη δεν υπάρχει, μπορείτε να καλέσετε . tf.config.set_soft_device_placement(True) tf.config.set_soft_device_placement(True) tf.debugging.set_log_device_placement(True) # Creates some tensors a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c = tf.matmul(a, b) print(c) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 tf.Tensor( [[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32) Χρησιμοποιώντας πολλαπλά GPUs Η ανάπτυξη για πολλαπλές GPUs θα επιτρέψει σε ένα μοντέλο να κλιμακωθεί με τους πρόσθετους πόρους. Εάν η ανάπτυξη σε ένα σύστημα με μία GPU, μπορείτε να προσομοιώσετε πολλαπλές GPUs με εικονικές συσκευές. gpus = tf.config.list_physical_devices('GPU') if gpus: # Create 2 virtual GPUs with 1GB memory each try: tf.config.set_logical_device_configuration( gpus[0], [tf.config.LogicalDeviceConfiguration(memory_limit=1024), tf.config.LogicalDeviceConfiguration(memory_limit=1024)]) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPU,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Virtual devices must be set before GPUs have been initialized print(e) Virtual devices cannot be modified after being initialized Μόλις υπάρχουν πολλαπλές λογικές GPU διαθέσιμες για το runtime, μπορείτε να χρησιμοποιήσετε τις πολλαπλές GPU με ή με χειροκίνητη τοποθέτηση. tf.distribute.Strategy Με tf.distribute.Strategy Στρατηγική διανομής Η καλύτερη πρακτική για τη χρήση πολλαπλών GPUs είναι να χρησιμοποιείτε Εδώ είναι ένα απλό παράδειγμα: tf.distribute.Strategy tf.debugging.set_log_device_placement(True) gpus = tf.config.list_logical_devices('GPU') strategy = tf.distribute.MirroredStrategy(gpus) with strategy.scope(): inputs = tf.keras.layers.Input(shape=(1,)) predictions = tf.keras.layers.Dense(1)(inputs) model = tf.keras.models.Model(inputs=inputs, outputs=predictions) model.compile(loss='mse', optimizer=tf.keras.optimizers.SGD(learning_rate=0.2)) INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2', '/job:localhost/replica:0/task:0/device:GPU:3') Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op FloorMod in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Cast in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op StatelessRandomGetKeyCounter in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0 product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:2 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:2 product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:1 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:3 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 FloorMod: (FloorMod): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Cast: (Cast): /job:localhost/replica:0/task:0/device:GPU:0 y_RetVal: (_DeviceRetval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0 _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 seed: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0 StatelessRandomGetKeyCounter: (StatelessRandomGetKeyCounter): /job:localhost/replica:0/task:0/device:GPU:0 key_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 counter_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 shape: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0 key: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 counter: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 alg: (_DeviceArg): /job:localhost/replica:0/taExecuting op StatelessRandomUniformV2 in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AddV2 in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 sk:0/device:CPU:0 StatelessRandomUniformV2: (StatelessRandomUniformV2): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Sub: (Sub): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Mul: (Mul): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AddV2: (AddV2): /job:localhost/replica:0/task:0/device:GPU:0 z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:1 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:2 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 NoOp: (NoOp): /job:localhost/replica:0/task:0/device:GPU:0 dims: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 Fill: (Fill): /job:localhost/replica:0/task:0/device:GPU:0 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:lExecuting op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0 ocalhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:1 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:2 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2 input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3 output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3 AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0 resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0 ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0 value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3 resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0 VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/deviExecuting op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0 Αυτό το πρόγραμμα θα εκτελέσει ένα αντίγραφο του μοντέλου σας σε κάθε GPU, διαιρώντας τα δεδομένα εισόδου μεταξύ τους, επίσης γνωστά ως " « Παράλληλες πληροφορίες Για περισσότερες πληροφορίες σχετικά με τις στρατηγικές διανομής, δείτε τον οδηγό . ΕΔΩ Εγχειρίδιο τοποθέτησης λειτουργεί κάτω από το καπάκι με την αναπαραγωγή υπολογισμού σε όλες τις συσκευές. Μπορείτε να εφαρμόσετε χειροκίνητα την αναπαραγωγή κατασκευάζοντας το μοντέλο σας σε κάθε GPU. tf.distribute.Strategy tf.debugging.set_log_device_placement(True) gpus = tf.config.list_logical_devices('GPU') if gpus: # Replicate your computation on multiple GPUs c = [] for gpu in gpus: with tf.device(gpu.name): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c.append(tf.matmul(a, b)) with tf.device('/CPU:0'): matmul_sum = tf.add_n(c) print(matmul_sum) Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:1 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:2 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:3 Executing op AddN in device /job:localhost/replica:0/task:0/device:CPU:0 tf.Tensor( [[ 88. 112.] [196. 256.]], shape=(2, 2), dtype=float32) Αρχικά δημοσιευμένο στην ιστοσελίδα TensorFlow, αυτό το άρθρο εμφανίζεται εδώ κάτω από μια νέα επικεφαλίδα και είναι αδειοδοτημένο υπό την άδεια CC BY 4.0. Αρχικά δημοσιευμένο στην ιστοσελίδα TensorFlow, αυτό το άρθρο εμφανίζεται εδώ κάτω από μια νέα επικεφαλίδα και είναι αδειοδοτημένο υπό την άδεια CC BY 4.0. TensorFlow TensorFlow