Kāpēc GradientTape ir visvairāk nenovērtētā funkcija TensorFlow

Satura pārskats Automātiskā diferenciācija un gradienti iestatīšana Gradientu datori Gradientu tapas Gradienti attiecībā pret modeli Kontrolēt, ko skatās lente Starpposma rezultāti Piezīmes par sniegumu Ne-skalaru mērķu gradienti Pārvaldīt plūsmu Automātiskā diferenciācija un gradienti ir noderīga mašīntulkošanas algoritmu ieviešanai, piemēram, Neironu tīklu apmācība. Automātiskā diferenciācija Atpakaļizplatīšana Šajā rokasgrāmatā jūs izpētīsiet veidus, kā aprēķināt gradientus ar TensorFlow, jo īpaši vērienīgā izpildē. iestatīšana import numpy as np import matplotlib.pyplot as plt import tensorflow as tf 2024-08-15 01:30:07.003169: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-08-15 01:30:07.023862: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-08-15 01:30:07.029954: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered Gradientu datori Lai automātiski atšķirtos, TensorFlow ir jāatceras, kādas operācijas notiek kādā secībā Tātad, pēc tam, laikā, kad , TensorFlow šķērso šo darbību sarakstu apgrieztā secībā, lai aprēķinātu gradientus. Uz priekšu Atpakaļceļš Gradientu tapas TensorFlow nodrošina API automātiskai diferenciācijai; tas ir, aprēķinot aprēķina gradientu attiecībā uz dažiem ievadiem, parasti s. TensorFlow "reģistrē" attiecīgās operācijas, kas tiek veiktas kontekstā ar TensorFlow pēc tam izmanto šo joslu, lai aprēķinātu "reģistrētā" aprēķina gradientus, izmantojot . tf.GradientTape tf.Variable tf.GradientTape Atgriezeniskā diferenciācija Šeit ir vienkāršs piemērs: x = tf.Variable(3.0) with tf.GradientTape() as tape: y = x**2 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1723685409.408818 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685409.412555 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685409.416343 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685409.420087 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685409.431667 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685409.435229 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685409.438777 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685409.442350 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685409.445712 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685409.449141 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685409.452491 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685409.456034 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.687389 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.689411 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.691490 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.693542 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.695541 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.697441 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.699432 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.701351 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.703333 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.705229 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.707222 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.744994 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.747037 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.749507 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.751538 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.753500 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.755501 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.757421 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.759404 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.761363 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.763858 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.766199 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.768560 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 Kad esat ierakstījis dažas operācijas, izmantojiet lai aprēķinātu kādu mērķa (bieži zaudējumu) gradientu attiecībā pret kādu avotu (bieži modeļa mainīgos): GradientTape.gradient(target, sources) # dy = 2x * dx dy_dx = tape.gradient(y, x) dy_dx.numpy() 6.0 Iepriekš minētais piemērs izmanto skalārus, bet darbojas tikpat viegli kā uz jebkuru tensoru: tf.GradientTape w = tf.Variable(tf.random.normal((3, 2)), name='w') b = tf.Variable(tf.zeros(2, dtype=tf.float32), name='b') x = [[1., 2., 3.]] with tf.GradientTape(persistent=True) as tape: y = x @ w + b loss = tf.reduce_mean(y**2) Lai iegūtu gradientu no Attiecībā uz abām mainīgajām, jūs varat nodot abas kā avotus Tape ir elastīga par to, kā avoti tiek nodoti, un pieņems jebkuru sarakstu vai vārdnīcu apvienojumu un atgriezīs gradientu strukturētu tādā pašā veidā (skatīt Tātad ) loss gradient tf.nest [dl_dw, dl_db] = tape.gradient(loss, [w, b]) Gradients attiecībā pret katru avotu ir avota formas: print(w.shape) print(dl_dw.shape) (3, 2) (3, 2) Šeit atkal ir gradientu aprēķins, šoreiz iziet mainīgo vārdnīcu: my_vars = { 'w': w, 'b': b } grad = tape.gradient(loss, my_vars) grad['b'] Gradienti attiecībā pret modeli Parasti tiek savākti Iekšā a vai viena no tās apakšgrupām ( , ) par un . tf.Variables tf.Module layers.Layer keras.Model Pārbaudes punkts Eksportēšana Vairumā gadījumu jūs vēlaties aprēķināt gradientus attiecībā uz modeļa apmācāmām mainīgajām. Apvienojiet savas izmaiņas ar īpašumu, jūs varat aprēķināt šos gradientus dažās kodu rindās: tf.Module Module.trainable_variables layer = tf.keras.layers.Dense(2, activation='relu') x = tf.constant([[1., 2., 3.]]) with tf.GradientTape() as tape: # Forward pass y = layer(x) loss = tf.reduce_mean(y**2) # Calculate gradients with respect to every trainable variable grad = tape.gradient(loss, layer.trainable_variables) for var, g in zip(layer.trainable_variables, grad): print(f'{var.name}, shape: {g.shape}') kernel, shape: (3, 2) bias, shape: (2,) Kontrolēt, ko skatās lente Parastā uzvedība ir ierakstīt visas operācijas pēc tam, kad piekļūstat Iemesli tam ir: tf.Variable Sloksnei ir jāzina, kuras operācijas jāreģistrē priekšā, lai aprēķinātu gradientus atpakaļceļā. Tape satur atsauces uz starpposma izejas, tāpēc jūs nevēlaties ierakstīt nevajadzīgas operācijas. Visbiežāk lietojamais gadījums ietver zaudējumu gradienta aprēķināšanu attiecībā uz visiem modeļa apmācāmiem mainīgajiem. Piemēram, nav iespējams aprēķināt gradientu, jo nav “skatoties” pēc noklusējuma, un Nav piemēroti treniņiem: tf.Tensor tf.Variable # A trainable variable x0 = tf.Variable(3.0, name='x0') # Not trainable x1 = tf.Variable(3.0, name='x1', trainable=False) # Not a Variable: A variable + tensor returns a tensor. x2 = tf.Variable(2.0, name='x2') + 1.0 # Not a variable x3 = tf.constant(3.0, name='x3') with tf.GradientTape() as tape: y = (x0**2) + (x1**2) + (x2**2) grad = tape.gradient(y, [x0, x1, x2, x3]) for g in grad: print(g) tf.Tensor(6.0, shape=(), dtype=float32) None None None Jūs varat uzskaitīt mainīgos, kurus skatās lente, izmantojot Metodiskā : GradientTape.watched_variables [var.name for var in tape.watched_variables()] ['x0:0'] nodrošina hooks, kas dod lietotājam kontroli pār to, kas ir vai netiek skatīts. tf.GradientTape Lai reģistrētu gradientus attiecībā pret a Jums ir nepieciešams zvanīt : tf.Tensor GradientTape.watch(x) x = tf.constant(3.0) with tf.GradientTape() as tape: tape.watch(x) y = x**2 # dy = 2x * dx dy_dx = tape.gradient(y, x) print(dy_dx.numpy()) 6.0 Pretēji, lai atspējot noklusējuma uzvedību skatīties visu Sērija Šis aprēķins izmanto divus mainīgos, bet savieno tikai gradientu vienam no mainīgajiem: tf.Variables watch_accessed_variables=False x0 = tf.Variable(0.0) x1 = tf.Variable(10.0) with tf.GradientTape(watch_accessed_variables=False) as tape: tape.watch(x1) y0 = tf.math.sin(x0) y1 = tf.nn.softplus(x1) y = y0 + y1 ys = tf.reduce_sum(y) Kopš Nav aicināts uz , nav aprēķināts gradients attiecībā uz to: GradientTape.watch x0 # dys/dx1 = exp(x1) / (1 + exp(x1)) = sigmoid(x1) grad = tape.gradient(ys, {'x0': x0, 'x1': x1}) print('dy/dx0:', grad['x0']) print('dy/dx1:', grad['x1'].numpy()) dy/dx0: None dy/dx1: 0.9999546 Starpposma rezultāti Jūs varat arī pieprasīt izejas gradientus attiecībā uz starpposma vērtībām, kas aprēķinātas Konteksts tf.GradientTape x = tf.constant(3.0) with tf.GradientTape() as tape: tape.watch(x) y = x * x z = y * y # Use the tape to compute the gradient of z with respect to the # intermediate value y. # dz_dy = 2 * y and y = x ** 2 = 9 print(tape.gradient(z, y).numpy()) 18.0 Pēc noklusējuma līdzekļi, kas pieder pie a Tiek atbrīvots, tiklīdz Lai aprēķinātu vairākus gradientus vienā un tajā pašā aprēķinā, izveidojiet gradienta joslu ar Tas ļauj veikt vairākus zvanus uz metodi kā resursi tiek atbrīvoti, kad lentes objekts ir atkritumi savākti. GradientTape GradientTape.gradient persistent=True gradient x = tf.constant([1, 3.0]) with tf.GradientTape(persistent=True) as tape: tape.watch(x) y = x * x z = y * y print(tape.gradient(z, x).numpy()) # [4.0, 108.0] (4 * x**3 at x = [1.0, 3.0]) print(tape.gradient(y, x).numpy()) # [2.0, 6.0] (2 * x at x = [1.0, 3.0]) [ 4. 108.] [2. 6.] del tape # Drop the reference to the tape Piezīmes par sniegumu There is a tiny overhead associated with doing operations inside a gradient tape context. For most eager execution this will not be a noticeable cost, but you should still use tape context around the areas only where it is required. Gradient tapes use memory to store intermediate results, including inputs and outputs, for use during the backwards pass. For efficiency, some ops (like ) don't need to keep their intermediate results and they are pruned during the forward pass. However, if you use on your tape, and your peak memory usage will be higher. ReLU persistent=True nothing is discarded Ne-skalaru mērķu gradienti Gradients būtībā ir operācija uz skalaru. x = tf.Variable(2.0) with tf.GradientTape(persistent=True) as tape: y0 = x**2 y1 = 1 / x print(tape.gradient(y0, x).numpy()) print(tape.gradient(y1, x).numpy()) 4.0 -0.25 Tādējādi, ja jūs pieprasāt vairāku mērķu gradientu, katra avota rezultāts ir: Mērķu summas gradients vai ekvivalents Katra mērķa gradientu summa. x = tf.Variable(2.0) with tf.GradientTape() as tape: y0 = x**2 y1 = 1 / x print(tape.gradient({'y0': y0, 'y1': y1}, x).numpy()) 3.75 Līdzīgi, ja mērķis(-i) nav skalārs, tiek aprēķināts summas gradients: x = tf.Variable(2.) with tf.GradientTape() as tape: y = x * [3., 4.] print(tape.gradient(y, x).numpy()) 7.0 Tas padara to vienkāršu, lai ņemtu zaudējumu kopas summas gradientu vai elementārā zaudējumu aprēķina summas gradientu. Ja jums ir nepieciešams atsevišķs gradients katram elementam, atsaucieties uz . Jacobiāns Dažos gadījumos jūs varat izlaist Jacobian. elementārā aprēķināšanai summas gradients dod katra elementa atvasinājumu attiecībā pret tā ievades elementu, jo katrs elements ir neatkarīgs: x = tf.linspace(-10.0, 10.0, 200+1) with tf.GradientTape() as tape: tape.watch(x) y = tf.nn.sigmoid(x) dy_dx = tape.gradient(y, x) plt.plot(x, y, label='y') plt.plot(x, dy_dx, label='dy/dx') plt.legend() _ = plt.xlabel('x') Pārvaldīt plūsmu Tā kā gradientu josla ieraksta operācijas, kad tās tiek izpildītas, Python kontroles plūsma tiek dabiski apstrādāta (piemēram, un Saskaņā ar paziņojumu). if while Šeit katrā filiālē tiek izmantots atšķirīgs mainīgais Gradients savieno tikai ar izmantoto mainīgo: if x = tf.constant(1.0) v0 = tf.Variable(2.0) v1 = tf.Variable(2.0) with tf.GradientTape(persistent=True) as tape: tape.watch(x) if x > 0.0: result = v0 else: result = v1**2 dv0, dv1 = tape.gradient(result, [v0, v1]) print(dv0) print(dv1) tf.Tensor(1.0, shape=(), dtype=float32) None Vienkārši atcerieties, ka paši kontroles paziņojumi nav diferencējami, tāpēc tie ir neredzami gradientu optimizatoriem. Atkarībā no vērtības iepriekš minētajā piemērā, kas ieraksta vai Gradients attiecībā pret Vienmēr ir . x result = v0 result = v1**2 x None dx = tape.gradient(result, x) print(dx) None Sākotnēji publicēts TensorFlow tīmekļa vietnē, šis raksts šeit parādās ar jaunu virsrakstu un ir licencēts saskaņā ar CC BY 4.0. Sākotnēji publicēts TensorFlow tīmekļa vietnē, šis raksts šeit parādās ar jaunu virsrakstu un ir licencēts saskaņā ar CC BY 4.0. TensorFlow TensorFlow