How to get multiprocessing for subtask work with tensorflow-gpu?

Basically I use tf-gpu 2.3rc0 to perform image detection on a video stream. For each loop, there’s a subtask runs separately. So I try to use Pool from multiprocessing The structure is like:

while VideoCapture.isOpenn():
    # first: backrgound calculation
    pool = Pool(num)
    res0 = pool.starmap(func, args)
    pool.close()

    # then,  perform detection in the meanwhile
    # return with res1
    
    # at last make use of these two results
    pool.join()
    func(res0, res1)

It runs without error, but the tf seems to re-initiate for each frame, resulting to a extremely slow speed… Specifically, the same as follows shows for each loop:

2021-02-15 21:27:57.546408: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2021-02-15 21:27:57.546698: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2021-02-15 21:28:00.999529: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2021-02-15 21:28:00.999846: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2021-20022-115- 0212-1:528 2:10:1.2083:10014.2: I tensorflow/core/common_runtime/gp031041: I tensorflow/core/common_runtime/ug/gpu/pgu_pu_ddevevice.iccce:.1c7c1:617] Foun1d6 ]d eFvoice 0 with properties: 
pciBusIDu:nd 00 0de0:0vice 0 w1i:th properties: 
pci0B0usI.D0 name: :G 0e0F0o0:0r1:0c0e. GT0X name:  GeFor1c060 we GTX 1i0t6h0  Mwaixt-hQ  MDaexs-iQg nD ecsoimgn pcuomtepCuatepabCialitypability: :6 6.1
.c1
orceClock: 1.3415GHz coreCount: 1o0re deviceMemorCyloSizce: k6:. 010.G3i4B1 5dGeHvzi ceMemoryBandwidth: 178.99GiB/s
coreCount2:0 2110- 0d2e-v1i5c e2M1e:m2o8ryS:ize0:1 .6031.90903G:i BI deviceMemoryBandwidth: 178.99 GteiB/ns
2021-02-15 21:28:01.032s1orfl47:o wI/ tstreenasorflow/ms_terxeame_ceuxteocru/tporlatf/orm/defapulalt/tdfsoor_m/loadder.cec:fault/4d8s]o _Suloacderc.ecscs:f48] Successufllullyy o oppeened dynamic lniedbra rdyy ncuamdiac libraryr tcud6a4rt_6104_110.d1.dll
ll
20212-02-0152 1-201:2-15 228:01:28:10.1.03073718104: I 3t:e I tensorflnsorofwl/streoaw/strema_execm_exuetcour/ptlatofr/ploatfrom/rm/default/dso_lodefault/adsdoer.cc:48] Succe_slsfully ooader.cpce:ne4d dy8] namiScu lccessifullbyr openaed dynramic lyi cubblarsary cublas64_10.d6l4l
_10.dll
2021-02-15 21:28:01.043209: I tensorflo2w02/stream_executor1/-02-15 p2l1a:t2f8o:0r1m.043/2d09efault/dso_lo: I taeder.nsorflow/stream_executor/cplc:48a]t fSuccessfully opened doyrnammic /ldiebfraaurlyt /cdusfof_tl6o4a_d1e0r..dcll
c:48] Successfully opened dynamic library cufft64_10.dll
2021-02-15 21:28:01.045029: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2021-02-15 21:28:01.045362: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2021-02-15 21:28:01.049708: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] S202u1-0ccessfully o2p-e15ned  dy21:n2a8:01.m0i4981c9: libra rI y ctuensolvers6o4r_f1l0.ow/dll
stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2021-02-15 21:28:01.053052: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2021-02-15 21:28:01.053398: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
202210-21-002-15 21:2-152 8:0211.06:12138:021: .0I6 1te133: In stenosrforflloow/w/ssttrreeamam_exec_uetxoerc/uptloatr/pfloratfmor/dmefault//ddseof_aluoladt/edr.soc_c:loa4der8] Su.cccec:ssfully 4o8]pened dyn aSuccmesic slfuibrlaly operned yd ycunamic dlninb6rary cud4n_7n64_7..ddlll
l
2021-022021-0-2-15 125 1:221:28:081:.010.606198169:8 I6 :t eI tensnorflosworflo/wc/ocroree//ccoommmmoon_runtinm_re/gupnut/gimep/gpu/ug_dpu_deviecvei.cecc.:cc:18158858]]  AAddddiinng gv isvisiiblble gep ug pudev deviceisces: 0
: 0

Normally this info only appears once when it the model is loaded. I don’t know how to fix it – is it that I used it wrong or is there some settings of tf I should be aware of?

Answer

it seems likely you’re not only re-initializing tf for each frame, but you’re starting a new process for each frame, then killing it after it’s done. One of the benefits of a Pool is letting some initialization happen in the child process, then keeping it around to execute multiple tasks. I think a simple solution would probably be re-arranging your code to look something like this, though it’s hard to know with what you’ve posted.

with Pool(num) as pool:
    while VideoCapture.isOpenn():
        # first: backrgound calculation
        res0 = pool.starmap(func, args)

        # then,  perform detection in the meanwhile
        # return with res1
    
        # at last make use of these two results
        func(res0, res1)