I’ve been trying to make Tensorflow 2.8.0 work with my Windows GPU (GeForce GTX 1650 Ti), and even though it detects my GPU, any model that I make will be stuck at Epoch 1 indefinitely when I try to use the fit method till the kernel (I’ve tried on jupyter notebook and spyder) hangs and restarts.

Based on Tensorflow’s website, I’ve downloaded the respective cuDNN and CUDA versions, for which I’ve further verified (together with tensorflow’s detection of my GPU) by running the various commands:

CUDA (Supposed to be 11.2)

(on command line)
nvcc --version
Build cuda_11.2.r11.2/compiler.29373293_0

(In python)
import tensorflow.python.platform.build_info as build
print(build.build_info['cuda_version'])
Output: '64_112'

cuDNN (Supposed to be 8.1)

import tensorflow.python.platform.build_info as build
print(build.build_info['cuda_version'])
Output: '64_8' # Looks like v8 but I've actually installed v8.1 (cuDNN v8.1.1 (Feburary 26th, 2021), for CUDA 11.0,11.1 and 11.2) so I think it's fine?

GPU Checks

tf.config.list_physical_devices('GPU')
Output: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

tf.test.is_gpu_available()
Output: True

tf.test.gpu_device_name()
Output: This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Created device /device:GPU:0 with 2153 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1650 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5

When I then try to fit any sort of model, it just fails following what I described above. What is surprising is that even though it can’t load code such as that described in Tensorflow’s CNN Tutorial, the only time it ever works is if I run the chunk of code from this stackoverflow question. This chunk of code looks almost the same as every other chunk that failed.

Can someone help me with this issue? I’ve been desperately testing TensorFlow with every chunk of code that I came across for the past couple of hours, and the only time where it does not get stuck at Epoch 1 is with the link above.

**(I’ve also tried running only on my CPU via os.environ['CUDA_VISIBLE_DEVICES'] = '-1' and everything seems to work fine)

Answer

Update (Solution)

It seems like the suggestions from this post helped – I’ve copied the following files from the zipped cudnn bin sub folder (cudnn-11.2-windows-x64-v8.1.1.33cudabin) into my cuda bin folder (C:Program FilesNVIDIA GPU Computing ToolkitCUDAv11.2bin)

cudnn_adv_infer64_8.dll
cudnn_adv_train64_8.dll
cudnn_cnn_infer64_8.dll
cudnn_cnn_train64_8.dll
cudnn_ops_infer64_8.dll
cudnn_ops_train64_8.dll

It seems like I initially misinterpreted the copy all cudnn*.dll files as only copying over the cudnn64_8.dll file, rather than copying every other file listed above.

(Tensorflow) Stuck at Epoch 1 during model.fit()

Advertisement

Answer

Update (Solution)