Saving and Loading a Vowpal Wabbit model in python with –safe_resume –cb_explore

I would like to do online training with a contextual Vowpal Wabbit model, so I need to save and reload frequently. However, whenever I try to reload a model (which was initialized with –save_resume), I get an exception with:

Error: Model content is corrupted, weight vector index 1072693248 must be less than total vector length 262144
Traceback (most recent call last):
  File "/home/alex/projects/experiment/vw_minimal_example_fail.py", line 15, in <module>
    vw = pyvw.vw("--quiet -i model.vw")
  File "/home/alex/projects/datascience/lib/python3.8/site-packages/vowpalwabbit/pyvw.py", line 347, in __init__
    super(vw, self).__init__(" ".join(l))
RuntimeError: Model content is corrupted, weight vector index 1072693248 must be less than total vector length 262144

JavaScript
​x
 
Error: Model content is corrupted, weight vector index 1072693248 must be less than total vector length 262144
Traceback (most recent call last):
  File "/home/alex/projects/experiment/vw_minimal_example_fail.py", line 15, in <module>
    vw = pyvw.vw("--quiet -i model.vw")
  File "/home/alex/projects/datascience/lib/python3.8/site-packages/vowpalwabbit/pyvw.py", line 347, in __init__
    super(vw, self).__init__(" ".join(l))
RuntimeError: Model content is corrupted, weight vector index 1072693248 must be less than total vector length 262144
​

Example Code to reproduce:

from vowpalwabbit import pyvw

print('# test some save/load behavior')
example = "feature1:f feature2:f feature3:y feature4:f feature5:f feature6:f feature7:c feature8:b feature9:h feature10:e feature11:b feature12:k feature13:k feature14:b feature15:b feature16:p feature17:w feature18:o feature19:l feature20:h feature21:v feature22:g"
vw = pyvw.vw("--cb_explore 2 --quiet --save_resume") # removing --save_resume will prevent exception
vw.learn(f"1:1:0.25 | {example}")
before_save = vw.predict(f"| {example}")
print('before saving, prediction =', before_save)
vw.save("model.vw")

# now re-start vw by loading that model
vw = pyvw.vw("--quiet -i model.vw")
after_save = vw.predict(f"| {example}")
print(' after saving, prediction =', after_save)

JavaScript
 
from vowpalwabbit import pyvw
​
print('# test some save/load behavior')
example = "feature1:f feature2:f feature3:y feature4:f feature5:f feature6:f feature7:c feature8:b feature9:h feature10:e feature11:b feature12:k feature13:k feature14:b feature15:b feature16:p feature17:w feature18:o feature19:l feature20:h feature21:v feature22:g"
vw = pyvw.vw("--cb_explore 2 --quiet --save_resume") # removing --save_resume will prevent exception
vw.learn(f"1:1:0.25 | {example}")
before_save = vw.predict(f"| {example}")
print('before saving, prediction =', before_save)
vw.save("model.vw")
​
# now re-start vw by loading that model
vw = pyvw.vw("--quiet -i model.vw")
after_save = vw.predict(f"| {example}")
print(' after saving, prediction =', after_save)
​

Python 3.8.5 vowpalwabbit==8.10.1

If I don’t use –save_resume, load and save work, however the model performance is not as good. I would love to just do pickle.dump(vw) but that gives me a RuntimeError: RuntimeError: Pickling of "vowpalwabbit.pyvw.vw" instances is not enabled

Answer

The problem was resolved in a bugfix: https://github.com/VowpalWabbit/vowpal_wabbit/issues/3062

vowpalwabbit 8.10.2 has been released on PyPi

Advertisement

Answer