TensorFlow’s Experimental NumPy Interface Brings Familiarity to Deep Learning

Content Overview Overview Setup Enabling NumPy behavior TensorFlow NumPy ND array Type promotion Broadcasting Indexing Example Model TensorFlow NumPy and NumPy NumPy interoperability Buffer copies Operator precedence Overview Setup Enabling NumPy behavior TensorFlow NumPy ND array Type promotion Broadcasting Indexing Example Model TensorFlow NumPy and NumPy NumPy interoperability Buffer copies Operator precedence Overview TensorFlow implements a subset of the NumPy API, available as tf.experimental.numpy. This allows running NumPy code, accelerated by TensorFlow, while also allowing access to all of TensorFlow's APIs. NumPy API tf.experimental.numpy Setup import matplotlib.pyplot as plt import numpy as np import tensorflow as tf import tensorflow.experimental.numpy as tnp import timeit print("Using TensorFlow version %s" % tf.__version__) import matplotlib.pyplot as plt import numpy as np import tensorflow as tf import tensorflow.experimental.numpy as tnp import timeit print("Using TensorFlow version %s" % tf.__version__) 2024-08-15 01:31:55.452313: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-08-15 01:31:55.473711: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-08-15 01:31:55.480014: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered Using TensorFlow version 2.17.0 2024-08-15 01:31:55.452313: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-08-15 01:31:55.473711: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-08-15 01:31:55.480014: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered Using TensorFlow version 2.17.0 Enabling NumPy behavior In order to use tnp as NumPy, enable NumPy behavior for TensorFlow: tnp tnp.experimental_enable_numpy_behavior() tnp.experimental_enable_numpy_behavior() This call enables type promotion in TensorFlow and also changes type inference, when converting literals to tensors, to more strictly follow the NumPy standard. Note: This call will change the behavior of entire TensorFlow, not just the tf.experimental.numpy module. Note: This call will change the behavior of entire TensorFlow, not just the tf.experimental.numpy module. Note: tf.experimental.numpy TensorFlow NumPy ND array An instance of tf.experimental.numpy.ndarray, called ND Array, represents a multidimensional dense array of a given dtype placed on a certain device. It is an alias to tf.Tensor. Check out the ND array class for useful methods like ndarray.T, ndarray.reshape, ndarray.ravel and others. tf.experimental.numpy.ndarray ND Array dtype tf.Tensor ndarray.T ndarray.reshape ndarray.ravel First create an ND array object, and then invoke different methods. # Create an ND array and check out different attributes. ones = tnp.ones([5, 3], dtype=tnp.float32) print("Created ND array with shape = %s, rank = %s, " "dtype = %s on device = %s\n" % ( ones.shape, ones.ndim, ones.dtype, ones.device)) # `ndarray` is just an alias to `tf.Tensor`. print("Is `ones` an instance of tf.Tensor: %s\n" % isinstance(ones, tf.Tensor)) # Try commonly used member functions. print("ndarray.T has shape %s" % str(ones.T.shape)) print("narray.reshape(-1) has shape %s" % ones.reshape(-1).shape) # Create an ND array and check out different attributes. ones = tnp.ones([5, 3], dtype=tnp.float32) print("Created ND array with shape = %s, rank = %s, " "dtype = %s on device = %s\n" % ( ones.shape, ones.ndim, ones.dtype, ones.device)) # `ndarray` is just an alias to `tf.Tensor`. print("Is `ones` an instance of tf.Tensor: %s\n" % isinstance(ones, tf.Tensor)) # Try commonly used member functions. print("ndarray.T has shape %s" % str(ones.T.shape)) print("narray.reshape(-1) has shape %s" % ones.reshape(-1).shape) Created ND array with shape = (5, 3), rank = 2, dtype = on device = /job:localhost/replica:0/task:0/device:GPU:0 Is `ones` an instance of tf.Tensor: True ndarray.T has shape (3, 5) narray.reshape(-1) has shape (15,) WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1723685517.895102 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.898954 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.902591 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.906282 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.918004 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.921334 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.924792 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.928195 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.931648 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.935182 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.938672 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.942206 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.174780 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.176882 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.178882 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.180984 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.183497 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.185413 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.187391 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.189496 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.191405 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.193373 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.195387 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.197409 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.235845 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.237893 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.239848 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.241962 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.243923 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.245869 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.247783 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.249797 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.251701 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.254112 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.256530 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.258985 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 Created ND array with shape = (5, 3), rank = 2, dtype = on device = /job:localhost/replica:0/task:0/device:GPU:0 Is `ones` an instance of tf.Tensor: True ndarray.T has shape (3, 5) narray.reshape(-1) has shape (15,) WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1723685517.895102 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.898954 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.902591 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.906282 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.918004 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.921334 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.924792 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.928195 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.931648 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.935182 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.938672 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685517.942206 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.174780 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.176882 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.178882 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.180984 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.183497 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.185413 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.187391 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.189496 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.191405 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.193373 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.195387 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.197409 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.235845 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.237893 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.239848 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.241962 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.243923 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.245869 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.247783 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.249797 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.251701 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.254112 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.256530 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685519.258985 23752 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 Type promotion There are 4 options for type promotion in TensorFlow. By default, TensorFlow raises errors instead of promoting types for mixed type operations. Running tf.numpy.experimental_enable_numpy_behavior() switches TensorFlow to use NumPy type promotion rules (described below). After TensorFlow 2.15, there are two new options (refer to TF NumPy Type Promotion for details): tf.numpy.experimental_enable_numpy_behavior(dtype_conversion_mode="all") uses Jax type promotion rules. tf.numpy.experimental_enable_numpy_behavior(dtype_conversion_mode="safe") uses Jax type promotion rules, but disallows certain unsafe promotions. By default, TensorFlow raises errors instead of promoting types for mixed type operations. Running tf.numpy.experimental_enable_numpy_behavior() switches TensorFlow to use NumPy type promotion rules (described below). tf.numpy.experimental_enable_numpy_behavior() NumPy After TensorFlow 2.15, there are two new options (refer to TF NumPy Type Promotion for details): tf.numpy.experimental_enable_numpy_behavior(dtype_conversion_mode="all") uses Jax type promotion rules. tf.numpy.experimental_enable_numpy_behavior(dtype_conversion_mode="safe") uses Jax type promotion rules, but disallows certain unsafe promotions. TF NumPy Type Promotion tf.numpy.experimental_enable_numpy_behavior(dtype_conversion_mode="all") uses Jax type promotion rules. tf.numpy.experimental_enable_numpy_behavior(dtype_conversion_mode="safe") uses Jax type promotion rules, but disallows certain unsafe promotions. tf.numpy.experimental_enable_numpy_behavior(dtype_conversion_mode="all") uses Jax type promotion rules. tf.numpy.experimental_enable_numpy_behavior(dtype_conversion_mode="all") tf.numpy.experimental_enable_numpy_behavior(dtype_conversion_mode="safe") uses Jax type promotion rules, but disallows certain unsafe promotions. tf.numpy.experimental_enable_numpy_behavior(dtype_conversion_mode="safe") NumPy Type Promotion TensorFlow NumPy APIs have well-defined semantics for converting literals to ND array, as well as for performing type promotion on ND array inputs. Please see np.result_type for more details. np.result_type TensorFlow APIs leave tf.Tensor inputs unchanged and do not perform type promotion on them, while TensorFlow NumPy APIs promote all inputs according to NumPy type promotion rules. In the next example, you will perform type promotion. First, run addition on ND array inputs of different types and note the output types. None of these type promotions would be allowed by TensorFlow APIs. tf.Tensor print("Type promotion for operations") values = [tnp.asarray(1, dtype=d) for d in (tnp.int32, tnp.int64, tnp.float32, tnp.float64)] for i, v1 in enumerate(values): for v2 in values[i + 1:]: print("%s + %s => %s" % (v1.dtype.name, v2.dtype.name, (v1 + v2).dtype.name)) print("Type promotion for operations") values = [tnp.asarray(1, dtype=d) for d in (tnp.int32, tnp.int64, tnp.float32, tnp.float64)] for i, v1 in enumerate(values): for v2 in values[i + 1:]: print("%s + %s => %s" % (v1.dtype.name, v2.dtype.name, (v1 + v2).dtype.name)) Type promotion for operations int32 + int64 => int64 int32 + float32 => float64 int32 + float64 => float64 int64 + float32 => float64 int64 + float64 => float64 float32 + float64 => float64 Type promotion for operations int32 + int64 => int64 int32 + float32 => float64 int32 + float64 => float64 int64 + float32 => float64 int64 + float64 => float64 float32 + float64 => float64 Finally, convert literals to ND array using ndarray.asarray and note the resulting type. ndarray.asarray print("Type inference during array creation") print("tnp.asarray(1).dtype == tnp.%s" % tnp.asarray(1).dtype.name) print("tnp.asarray(1.).dtype == tnp.%s\n" % tnp.asarray(1.).dtype.name) print("Type inference during array creation") print("tnp.asarray(1).dtype == tnp.%s" % tnp.asarray(1).dtype.name) print("tnp.asarray(1.).dtype == tnp.%s\n" % tnp.asarray(1.).dtype.name) Type inference during array creation tnp.asarray(1).dtype == tnp.int64 tnp.asarray(1.).dtype == tnp.float64 Type inference during array creation tnp.asarray(1).dtype == tnp.int64 tnp.asarray(1.).dtype == tnp.float64 When converting literals to ND array, NumPy prefers wide types like tnp.int64 and tnp.float64. In contrast, tf.convert_to_tensor prefers tf.int32 and tf.float32 types for converting constants to tf.Tensor. TensorFlow NumPy APIs adhere to the NumPy behavior for integers. As for floats, the prefer_float32 argument of experimental_enable_numpy_behavior lets you control whether to prefer tf.float32 over tf.float64 (default to False). For example: tnp.int64 tnp.float64 tf.convert_to_tensor tf.int32 tf.float32 tf.Tensor prefer_float32 experimental_enable_numpy_behavior tf.float32 tf.float64 False tnp.experimental_enable_numpy_behavior(prefer_float32=True) print("When prefer_float32 is True:") print("tnp.asarray(1.).dtype == tnp.%s" % tnp.asarray(1.).dtype.name) print("tnp.add(1., 2.).dtype == tnp.%s" % tnp.add(1., 2.).dtype.name) tnp.experimental_enable_numpy_behavior(prefer_float32=False) print("When prefer_float32 is False:") print("tnp.asarray(1.).dtype == tnp.%s" % tnp.asarray(1.).dtype.name) print("tnp.add(1., 2.).dtype == tnp.%s" % tnp.add(1., 2.).dtype.name) tnp.experimental_enable_numpy_behavior(prefer_float32=True) print("When prefer_float32 is True:") print("tnp.asarray(1.).dtype == tnp.%s" % tnp.asarray(1.).dtype.name) print("tnp.add(1., 2.).dtype == tnp.%s" % tnp.add(1., 2.).dtype.name) tnp.experimental_enable_numpy_behavior(prefer_float32=False) print("When prefer_float32 is False:") print("tnp.asarray(1.).dtype == tnp.%s" % tnp.asarray(1.).dtype.name) print("tnp.add(1., 2.).dtype == tnp.%s" % tnp.add(1., 2.).dtype.name) When prefer_float32 is True: tnp.asarray(1.).dtype == tnp.float32 tnp.add(1., 2.).dtype == tnp.float32 When prefer_float32 is False: tnp.asarray(1.).dtype == tnp.float64 tnp.add(1., 2.).dtype == tnp.float64 When prefer_float32 is True: tnp.asarray(1.).dtype == tnp.float32 tnp.add(1., 2.).dtype == tnp.float32 When prefer_float32 is False: tnp.asarray(1.).dtype == tnp.float64 tnp.add(1., 2.).dtype == tnp.float64 Broadcasting Similar to TensorFlow, NumPy defines rich semantics for "broadcasting" values. You can check out the NumPy broadcasting guide for more information and compare this with TensorFlow broadcasting semantics. NumPy broadcasting guide TensorFlow broadcasting semantics x = tnp.ones([2, 3]) y = tnp.ones([3]) z = tnp.ones([1, 2, 1]) print("Broadcasting shapes %s, %s and %s gives shape %s" % ( x.shape, y.shape, z.shape, (x + y + z).shape)) x = tnp.ones([2, 3]) y = tnp.ones([3]) z = tnp.ones([1, 2, 1]) print("Broadcasting shapes %s, %s and %s gives shape %s" % ( x.shape, y.shape, z.shape, (x + y + z).shape)) Broadcasting shapes (2, 3), (3,) and (1, 2, 1) gives shape (1, 2, 3) Broadcasting shapes (2, 3), (3,) and (1, 2, 1) gives shape (1, 2, 3) Indexing NumPy defines very sophisticated indexing rules. See the NumPy Indexing guide. Note the use of ND arrays as indices below. NumPy Indexing guide x = tnp.arange(24).reshape(2, 3, 4) print("Basic indexing") print(x[1, tnp.newaxis, 1:3, ...], "\n") print("Boolean indexing") print(x[:, (True, False, True)], "\n") print("Advanced indexing") print(x[1, (0, 0, 1), tnp.asarray([0, 1, 1])]) x = tnp.arange(24).reshape(2, 3, 4) print("Basic indexing") print(x[1, tnp.newaxis, 1:3, ...], "\n") print("Boolean indexing") print(x[:, (True, False, True)], "\n") print("Advanced indexing") print(x[1, (0, 0, 1), tnp.asarray([0, 1, 1])]) Basic indexing tf.Tensor( [[[16 17 18 19] [20 21 22 23]]], shape=(1, 2, 4), dtype=int64) Boolean indexing tf.Tensor( [[[ 0 1 2 3] [ 8 9 10 11]] [[12 13 14 15] [20 21 22 23]]], shape=(2, 2, 4), dtype=int64) Advanced indexing tf.Tensor([12 13 17], shape=(3,), dtype=int64) Basic indexing tf.Tensor( [[[16 17 18 19] [20 21 22 23]]], shape=(1, 2, 4), dtype=int64) Boolean indexing tf.Tensor( [[[ 0 1 2 3] [ 8 9 10 11]] [[12 13 14 15] [20 21 22 23]]], shape=(2, 2, 4), dtype=int64) Advanced indexing tf.Tensor([12 13 17], shape=(3,), dtype=int64) # Mutation is currently not supported try: tnp.arange(6)[1] = -1 except TypeError: print("Currently, TensorFlow NumPy does not support mutation.") # Mutation is currently not supported try: tnp.arange(6)[1] = -1 except TypeError: print("Currently, TensorFlow NumPy does not support mutation.") Currently, TensorFlow NumPy does not support mutation. Currently, TensorFlow NumPy does not support mutation. Example Model Next, you can see how to create a model and run inference on it. This simple model applies a relu layer followed by a linear projection. Later sections will show how to compute gradients for this model using TensorFlow's GradientTape. GradientTape class Model(object): """Model with a dense and a linear layer.""" def __init__(self): self.weights = None def predict(self, inputs): if self.weights is None: size = inputs.shape[1] # Note that type `tnp.float32` is used for performance. stddev = tnp.sqrt(size).astype(tnp.float32) w1 = tnp.random.randn(size, 64).astype(tnp.float32) / stddev bias = tnp.random.randn(64).astype(tnp.float32) w2 = tnp.random.randn(64, 2).astype(tnp.float32) / 8 self.weights = (w1, bias, w2) else: w1, bias, w2 = self.weights y = tnp.matmul(inputs, w1) + bias y = tnp.maximum(y, 0) # Relu return tnp.matmul(y, w2) # Linear projection model = Model() # Create input data and compute predictions. print(model.predict(tnp.ones([2, 32], dtype=tnp.float32))) class Model(object): """Model with a dense and a linear layer.""" def __init__(self): self.weights = None def predict(self, inputs): if self.weights is None: size = inputs.shape[1] # Note that type `tnp.float32` is used for performance. stddev = tnp.sqrt(size).astype(tnp.float32) w1 = tnp.random.randn(size, 64).astype(tnp.float32) / stddev bias = tnp.random.randn(64).astype(tnp.float32) w2 = tnp.random.randn(64, 2).astype(tnp.float32) / 8 self.weights = (w1, bias, w2) else: w1, bias, w2 = self.weights y = tnp.matmul(inputs, w1) + bias y = tnp.maximum(y, 0) # Relu return tnp.matmul(y, w2) # Linear projection model = Model() # Create input data and compute predictions. print(model.predict(tnp.ones([2, 32], dtype=tnp.float32))) tf.Tensor( [[-0.8292594 0.75780904] [-0.8292594 0.75780904]], shape=(2, 2), dtype=float32) tf.Tensor( [[-0.8292594 0.75780904] [-0.8292594 0.75780904]], shape=(2, 2), dtype=float32) TensorFlow NumPy and NumPy TensorFlow NumPy implements a subset of the full NumPy spec. While more symbols will be added over time, there are systematic features that will not be supported in the near future. These include NumPy C API support, Swig integration, Fortran storage order, views and stride_tricks, and some dtypes (like np.recarray and np.object). For more details, please see the TensorFlow NumPy API Documentation. stride_tricks dtype np.recarray np.object TensorFlow NumPy API Documentation NumPy interoperability TensorFlow ND arrays can interoperate with NumPy functions. These objects implement the __array__ interface. NumPy uses this interface to convert function arguments to np.ndarray values before processing them. __array__ np.ndarray Similarly, TensorFlow NumPy functions can accept inputs of different types including np.ndarray. These inputs are converted to an ND array by calling ndarray.asarray on them. np.ndarray ndarray.asarray Conversion of the ND array to and from np.ndarray may trigger actual data copies. Please see the section on buffer copies for more details. np.ndarray buffer copies # ND array passed into NumPy function. np_sum = np.sum(tnp.ones([2, 3])) print("sum = %s. Class: %s" % (float(np_sum), np_sum.__class__)) # `np.ndarray` passed into TensorFlow NumPy function. tnp_sum = tnp.sum(np.ones([2, 3])) print("sum = %s. Class: %s" % (float(tnp_sum), tnp_sum.__class__)) # ND array passed into NumPy function. np_sum = np.sum(tnp.ones([2, 3])) print("sum = %s. Class: %s" % (float(np_sum), np_sum.__class__)) # `np.ndarray` passed into TensorFlow NumPy function. tnp_sum = tnp.sum(np.ones([2, 3])) print("sum = %s. Class: %s" % (float(tnp_sum), tnp_sum.__class__)) sum = 6.0. Class: sum = 6.0. Class: sum = 6.0. Class: sum = 6.0. Class: # It is easy to plot ND arrays, given the __array__ interface. labels = 15 + 2 * tnp.random.randn(1, 1000) _ = plt.hist(labels) # It is easy to plot ND arrays, given the __array__ interface. labels = 15 + 2 * tnp.random.randn(1, 1000) _ = plt.hist(labels) Buffer copies Intermixing TensorFlow NumPy with NumPy code may trigger data copies. This is because TensorFlow NumPy has stricter requirements on memory alignment than those of NumPy. When a np.ndarray is passed to TensorFlow NumPy, it will check for alignment requirements and trigger a copy if needed. When passing an ND array CPU buffer to NumPy, generally the buffer will satisfy alignment requirements and NumPy will not need to create a copy. np.ndarray ND arrays can refer to buffers placed on devices other than the local CPU memory. In such cases, invoking a NumPy function will trigger copies across the network or device as needed. Given this, intermixing with NumPy API calls should generally be done with caution and the user should watch out for overheads of copying data. Interleaving TensorFlow NumPy calls with TensorFlow calls is generally safe and avoids copying data. See the section on TensorFlow interoperability for more details. TensorFlow interoperability Operator precedence TensorFlow NumPy defines an __array_priority__ higher than NumPy's. This means that for operators involving both ND array and np.ndarray, the former will take precedence, i.e., np.ndarray input will get converted to an ND array and the TensorFlow NumPy implementation of the operator will get invoked. __array_priority__ np.ndarray np.ndarray x = tnp.ones([2]) + np.ones([2]) print("x = %s\nclass = %s" % (x, x.__class__)) x = tnp.ones([2]) + np.ones([2]) print("x = %s\nclass = %s" % (x, x.__class__)) x = tf.Tensor([2. 2.], shape=(2,), dtype=float64) class = x = tf.Tensor([2. 2.], shape=(2,), dtype=float64) class = Originally published on the TensorFlow website, this article appears here under a new headline and is licensed under CC BY 4.0. Code samples shared under the Apache 2.0 License. Originally published on the TensorFlow website, this article appears here under a new headline and is licensed under CC BY 4.0. Code samples shared under the Apache 2.0 License. Originally published on the TensorFlow website, this article appears here under a new headline and is licensed under CC BY 4.0. Code samples shared under the Apache 2.0 License. TensorFlow TensorFlow