Tensorflow’ pb and pbtxt files don’t work with OpenCV after retraining MobileNet SSD V1 COCO

Tags: , , , ,



I have followed this tutorial to retrain MobileNet SSD V1 using Tensorflow GPU as described and got 0.5 loss after training using GPU (below more info about config) and got model.ckpt.

This is the command I used for Training:

python ../models/research/object_detection/legacy/train.py --logtostderr --train_dir=./data/ --pipeline_config_path=./ssd_mobilenet_v1_pets.config

And this is the command for freezing (generate pb file):

python ../models/research/object_detection/export_inference_graph.py --input_type image_tensor --pipeline_config_path ./ssd_mobilenet_v1_pets.config --trained_checkpoint_prefix ./data/model.ckpt-1407 --output_directory ./data/

This is the error I get when I use frozen pb and pbtxt:

Traceback (most recent call last):
File "Object_detection_image.py", line 29, in <module>
    cvOut = cvNet.forward()
cv2.error: OpenCV(3.4.3) C:projectsopencv-pythonopencvmodulesdnnsrcdnn.cpp:565: error: (-215:Assertion failed) inputs.size() == requiredOutputs in function 'cv::dnn::experimental_dnn_34_v7::DataLayer::getMemoryShapes'

This is the Object_detection_image.py file I used:

import cv2 as cv
import os 
import time 
import logging

logger = logging.getLogger()
fh = logging.FileHandler('xyz.log')
fh.setLevel(logging.DEBUG)    
logger.addHandler(fh)

cvNet = cv.dnn.readNetFromTensorflow('frozen_inference_graph.pb', 'object_detection.pbtxt')
dir_x  = "C:\Users\Omen\Desktop\LP_dataset\anno"
for filename in os.listdir(dir_x):
    print(filename)
    if not (filename.endswith(".png") or filename.endswith(".jpg")):
        continue
    print('daz')
    img = cv.imread(os.path.join(dir_x,filename))
    img = cv.resize(img, (300,300))
    #cv.imshow('i',img)
    #cv.waitKey(0)
    img = cv.cvtColor(img,cv.COLOR_BGR2GRAY)
    img = cv.cvtColor(img,cv.COLOR_GRAY2RGB)
    rows = img.shape[0]
    cols = img.shape[1]
    #cvNet.setInput(cv.dnn.blobFromImage(img, size=(cols,rows), swapRB=True, crop=False))
    cvNet.setInput(cv.dnn.blobFromImage(img, size=(300, 300), crop=False))
    t0  = time.time()
    cvOut = cvNet.forward()
    print(time.time() - t0)
    for detection in cvOut[0,0,:,:]:
        score = float(detection[2])
        #print(score)
        if score > 0.80:
            left = detection[3] * cols
            top = detection[4] * rows
            right = detection[5] * cols
            bottom = detection[6] * rows
            cv.rectangle(img, (int(left), int(top)), (int(right), int(bottom)), (23, 230, 210), thickness=2)

    cv.imshow('img', img)
    cv.waitKey(0)

This is pbtxt file (also I tried the exported pbtxt and generated pbtxt from pb but not working):

item {
  id: 1
  name: 'licenseplate'
}

Config:

What is the top-level directory of the model you are using: object_detetion

Have I written custom code: no

OS Platform and Distribution: win10

TensorFlow installed from: binary

TensorFlow GPU version: 1.13.0

CUDA/cuDNN version: 10

GPU model: 1050 GTX

I can provide any files you ask, please help me. In tensorflow’s github they told me to ask in Stackoverflow…

Update:

I got the problem solved thanks to the answer, here is the content of cvOut:

  [[[[-0.00476191 -0.00361736  0.          0.25361738 -0.07576995
     0.03405379  0.40910327]
   [ 0.21594621  0.04544836  0.          0.28788495  0.30689242
    -0.13025634  0.05074273]
   [ 0.46358964  0.19925728  0.         -0.09778295  0.26563603
     0.34778297 -0.02014329]
   [-0.01515752  0.3534766   0.          0.32857144 -0.00361736
     0.67142856  0.25361738]
   [ 0.25756338  0.03405379  0.          0.21594621  0.3787817
    -0.05689242  0.6212183 ]
   [ 0.30689242  0.203077    0.          0.796923    0.19925728
     0.40103063 -0.09778295]
   [ 0.5989694   0.34778297  0.         -0.01515752  0.68680996
     0.26515752  0.66190475]
   [-0.00361736  1.0047619   0.          0.59089667  0.03405379
     1.0757699   0.21594621]
   [ 0.712115   -0.05689242  0.          0.30689242  0.53641033
     0.05074273  1.1302563 ]
   [ 0.19925728  0.7343639   0.          0.93230265  0.34778297
     0.64652336 -0.01515752]
   [ 1.0201433   0.26515752  0.          0.24638264  0.33809522
     0.50361735 -0.07576995]
   [ 0.2840538   0.40910327  0.          0.04544836  0.19310758
     0.28788495  0.5568924 ]
   [-0.13025634  0.30074272  0.          0.44925728  0.06769729
     0.15221705  0.26563603]
   [ 0.59778297 -0.02014329  0.          0.3534766   0.5151575
     0.32857144  0.24638264]
   [ 0.67142856  0.50361735  0.          0.2840538   0.7424366
     0.4659462   0.3787817 ]
   [ 0.19310758  0.6212183   0.          0.203077    0.30074272
     0.796923    0.44925728]
   [ 0.40103063  0.15221705  0.          0.59778297  0.31319004
     0.23484248  0.68680996]
   [ 0.5151575   0.66190475  0.          1.0047619   0.50361735
     0.59089667  0.2840538 ]
   [ 1.0757699   0.4659462   0.          0.19310758  0.95455164
     0.5568924   0.53641033]
   [ 0.30074272  1.1302563   0.          0.7343639   0.15221705
     0.93230265  0.59778297]
   [ 0.64652336  0.23484248  0.          0.5151575  -0.00476191
     0.49638262  0.33809522]
   [ 0.75361735 -0.07576995  0.          0.40910327  0.7159462
     0.04544836  0.44310758]
   [ 0.28788495  0.8068924   0.          0.55074275  0.46358964
     0.69925725  0.06769729]
   [ 0.40221703  0.26563603  0.         -0.02014329  0.48484248
     0.3534766   0.7651575 ]
   [ 0.32857144  0.49638262  0.          0.75361735  0.25756338
     0.5340538   0.7424366 ]
   [ 0.7159462   0.3787817   0.          0.6212183   0.8068924
     0.203077    0.55074275]
   [ 0.796923    0.69925725  0.          0.40221703  0.5989694
     0.84778297  0.31319004]
   [ 0.48484248  0.68680996  0.          0.66190475  0.49638262
     1.0047619   0.75361735]
   [ 0.59089667  0.5340538   0.          0.7159462   0.712115
     0.44310758  0.95455164]
   [ 0.8068924   0.53641033  0.          1.1302563   0.69925725
     0.7343639   0.40221703]
   [ 0.93230265  0.84778297  0.          0.48484248  1.0201433
     0.7651575  -0.00476191]
   [ 0.74638265  0.33809522  0.         -0.07576995  0.7840538
     0.40910327  0.9659462 ]
   [ 0.04544836  0.6931076   0.          1.0568924  -0.13025634
     0.80074275  0.46358964]
   [ 0.94925725  0.06769729  0.          0.26563603  1.0977829
    -0.02014329  0.7348425 ]
   [ 0.3534766   1.0151576   0.          0.74638265  0.67142856
     1.0036174   0.25756338]
   [ 0.7840538   0.7424366   0.          0.3787817   0.6931076
     0.6212183   1.0568924 ]
   [ 0.203077    0.80074275  0.          0.94925725  0.40103063
     0.65221703  0.5989694 ]
   [ 1.0977829   0.31319004  0.          0.68680996  1.0151576
     0.66190475  0.74638265]
   [ 1.0047619   1.0036174   0.          0.7840538   1.0757699
     0.9659462   0.712115  ]
   [ 0.6931076   0.95455164  0.          0.53641033  0.80074275
     1.1302563   0.94925725]
   [ 0.7343639   0.65221703  0.          1.0977829   0.64652336
     0.7348425   1.0201433 ]
   [ 1.0151576   0.1         0.          0.2         0.2
     0.1         0.1       ]
   [ 0.2         0.2         0.          0.1         0.2
     0.2         0.1       ]
   [ 0.1         0.2         0.          0.1         0.1
     0.2         0.2       ]
   [ 0.1         0.1         0.          0.2         0.1
     0.1         0.2       ]
   [ 0.2         0.1         0.          0.2         0.2
     0.1         0.1       ]
   [ 0.2         0.2         0.          0.1         0.2
     0.2         0.1       ]
   [ 0.1         0.2         0.          0.1         0.1
     0.2         0.2       ]
   [ 0.1         0.1         0.          0.2         0.1
     0.1         0.2       ]
   [ 0.2         0.1         0.          0.2         0.2
     0.1         0.1       ]
   [ 0.2         0.2         0.          0.1         0.2
     0.2         0.1       ]
   [ 0.1         0.2         0.          0.1         0.1
     0.2         0.2       ]
   [ 0.1         0.1         0.          0.2         0.1
     0.1         0.2       ]
   [ 0.2         0.1         0.          0.2         0.2
     0.1         0.1       ]
   [ 0.2         0.2         0.          0.1         0.2
     0.2         0.1       ]
   [ 0.1         0.2         0.          0.1         0.1
     0.2         0.2       ]
   [ 0.1         0.1         0.          0.2         0.1
     0.1         0.2       ]
   [ 0.2         0.1         0.          0.2         0.2
     0.1         0.1       ]
   [ 0.2         0.2         0.          0.1         0.2
     0.2         0.1       ]
   [ 0.1         0.2         0.          0.1         0.1
     0.2         0.2       ]
   [ 0.1         0.1         0.          0.2         0.1
     0.1         0.2       ]
   [ 0.2         0.1         0.          0.2         0.2
     0.1         0.1       ]
   [ 0.2         0.2         0.          0.1         0.2
     0.2         0.1       ]
   [ 0.1         0.2         0.          0.1         0.1
     0.2         0.2       ]
   [ 0.1         0.1         0.          0.2         0.1
     0.1         0.2       ]
   [ 0.2         0.1         0.          0.2         0.2
     0.1         0.1       ]
   [ 0.2         0.2         0.          0.1         0.2
     0.2         0.1       ]
   [ 0.1         0.2         0.          0.1         0.1
     0.2         0.2       ]
   [ 0.1         0.1         0.          0.2         0.1
     0.1         0.2       ]
   [ 0.2         0.1         0.          0.2         0.2
     0.1         0.1       ]
   [ 0.2         0.2         0.          0.1         0.2
     0.2         0.1       ]
   [ 0.1         0.2         0.          0.1         0.1
     0.2         0.2       ]
   [ 0.1         0.1         0.          0.2         0.1
     0.1         0.2       ]
   [ 0.2         0.1         0.          0.2         0.2
     0.1         0.1       ]
   [ 0.2         0.2         0.          0.1         0.2
     0.2         0.1       ]
   [ 0.1         0.2         0.          0.1         0.1
     0.2         0.2       ]
   [ 0.1         0.1         0.          0.2         0.1
     0.1         0.2       ]
   [ 0.2         0.1         0.          0.2         0.2
     0.1         0.1       ]
   [ 0.2         0.2         0.          0.1         0.2
     0.2         0.1       ]
   [ 0.1         0.2         0.          0.1         0.1
     0.2         0.2       ]
   [ 0.1         0.1         0.          0.2         0.1
     0.1         0.2       ]
   [ 0.2         0.1         0.          0.2         0.2
     0.1         0.1       ]
   [ 0.2         0.2         0.          0.8479438   0.67317617
     0.5581815   0.1778345 ]
   [-0.9215721   1.5896183   0.          0.6099795   0.5955366
    -0.46569395 -0.8461083 ]
   [ 1.6129647   1.4244858   0.          0.5209342   0.17585325
    -0.8687666   1.7872683 ]
   [ 1.3389692   0.8533131   0.         -0.00590521 -0.7195761
     1.6236191   1.1828533 ]
   [ 1.1838211   0.6728102   0.         -0.785988    1.2751837
     1.1616383   0.933811  ]
   [ 0.4684658   0.2719049   0.          1.2093123   0.66612804
     0.66964823  0.55971766]
   [ 0.17104894 -1.0688283   0.          0.6494252   0.6844874
     0.66586125  0.01329695]
   [-1.2607187  -0.22749203  0.         -0.8741171  -0.9443728
    -0.9659323  -0.03422031]
   [-0.0364061   0.54829746  0.          0.6263525   0.66758543
     0.04167109 -0.11780822]
   [ 0.48400337  0.4685324   0.         -0.04594427  0.02469592
    -0.3487326   0.08831279]
   [ 0.4161314   0.23332608  0.         -0.13553022 -0.31008872
     0.04969648  0.5674252 ]
   [ 0.36492363 -0.07475745  0.         -0.03859219  0.2016789
    -0.39845943 -0.07058203]
   [-0.08173721  0.1720942   0.          0.02323131  0.07122216
     0.07469177  0.12792486]
   [-0.24689877  0.196296    0.          0.5564647   0.535513
     0.22528338 -0.37152448]
   [-1.7235181  -1.8204601   0.         -1.5040898  -1.8099409
    -1.8550183  -1.1855855 ]
   [-1.6341007  -1.3448519   0.         -1.6656716  -1.6564709
    -1.2735447  -1.3357594 ]
   [-1.2829769  -1.2869868   0.         -1.6657944  -1.4066424
    -1.4230443  -1.4196167 ]
   [-1.3691044  -1.656098    0.         -1.4339573  -1.5685135
    -1.633306   -1.4437945 ]]]]

Answer

The error was caused by the wrong input .pbtxt file passed into the function readNetFromTensorflow because the .pbtxt has to be geneated by tf_text_graph_ssd.py as describe here:

Run this script to get a text graph of SSD model from TensorFlow Object Detection API. Then pass it with .pb file to cv::dnn::readNetFromTensorflow function.

For other models such as faster r-cnn and mask r-cnn, there are also corresponding scripts.

PS: I just found there is a very good official tutorial here.



Source: stackoverflow