How to load models trained on GPU into CPU (system) memory?

I trained a model on a GPU and now I am trying to evaluate it on CPU (GPU is being used for a different train run). However when I try to load it using:

with tf.device('/cpu:0'):
    model = tf.keras.models.load_model('models/resnet50_untrained.h5', compile=False)

I am getting a CUDA_ERROR_OUT_OF_MEMORY:

2019-10-17 09:25:23.088408: W tensorflow/compiler/xla/service/platform_util.cc:256] unable to create StreamExecutor for CUDA:0: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY: out of memory; total memory reported: 7981694976
2019-10-17 09:25:23.088505: F tensorflow/stream_executor/lib/statusor.cc:34] Attempting to fetch value instead of handling error Internal: no supported devices found for platform CUDA

(I tried also setting compile=True with the same result.)

It seems that the model is being loaded into GPU which is already used by another instance. How can I force keras/tensorflow to load it into system memory and execute it on CPU?

Answer

Can you define everything inside with tf.device('/cpu:0'): except the libraries import part and test.

If that doesn’t work then create a virtual envoironment and install normal tensorflow not the gpu version and then try it. If it is still OOM error, then it is from the CPU being utilized and it has not enough memory to load this trained model.