xlearn predictions error give a different mse than output by the function

the xlearn predict function gives a different mse than what you get by looking at the predictions and calculating it yourself. Here is code to do this; you can run it by cloning the xlearn repository and copying the below code in demo/regression/house_price in the repository

# Copyright (c) 2018 by contributors. All Rights Reserved.                                                                                                   
#                                                                                                                                                            
# Licensed under the Apache License, Version 2.0 (the "License");                                                                                            
# you may not use this file except in compliance with the License.                                                                                           
# You may obtain a copy of the License at                                                                                                                    
#                                                                                                                                                            
#     http://www.apache.org/licenses/LICENSE-2.0                                                                                                             
#                                                                                                                                                            
# Unless required by applicable law or agreed to in writing, software                                                                                        
# distributed under the License is distributed on an "AS IS" BASIS,                                                                                          
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.                                                                                   
# See the License for the specific language governing permissions and                                                                                        
# limitations under the License.                                                                                                                             
                                                                                                                                                             
import xlearn as xl                                                                                                                                          
import pandas as pd                                                                                                                                          
from sklearn.metrics import mean_squared_error                                                                                                               
                                                                                                                                                             
# Training task                                                                                                                                              
                                                                                                                                                             
# param:                                                                                                                                                     
#  0. regression task                                                                                                                                        
#  1. learning rate: 0.2                                                                                                                                     
#  2. regular lambda: 0.002                                                                                                                                  
#  3. evaluation metric: mae                                                                                                                                 
fm_model = xl.create_linear()  # Use factorization machine                                                                                                   
fm_model.setTrain("./house_price_train.txt")    # Training data                                                                                              
fm_model.setValidate("./house_price_test.txt")  # Validation data                                                                                            
fm_model.disableNorm()                                                                                                                                       
# fm_model.setSigmoid()                                                                                                                                      
# fm_model.disableEarlyStop()                                                                                                                                
param = {'task':'reg', 'lr':0.0002,                                                                                                                          
        'lambda':0.00001, 'metric':'rmse', 'epoch':100}                                                                                                      
                                                                                                                                                             
# Start to train                                                                                                                                             
# The trained model will be stored in model.out                                                                                                              
print("here")                                                                                                                                                
fm_model.fit(param, './model.out')                                                                                                                           
fm_model.setTest("./house_price_test.txt")  # Test data                                                                                                      
                                                                                                                                                             
# Prediction task                                                                                                                                            
                                                                                                                                                             
# Start to predict                                                                                                                                           
# The output result will be stored in output.txt                                                                                                             
outs = fm_model.predict("./model.out", "./output.txt")                                                                                                       
true = pd.read_csv("./house_price_test.txt", sep='t', header=None)[0]                                                                                       
# print(true)                                                                                                                                                
preds = pd.read_csv("./output.txt", header=None)[0]                                                                                                          
                                                                                                                                                             
# Calculate using sklearn                                                                                                                                    
print(mean_squared_error(true, preds))                                                                                                                       
# Self calculate                                                                                                                                             
sq = 0.0                                                                                                                                                     
for t, p in zip(true, preds):                                                                                                                                
    sq += (t - p) ** 2                                                                                                                                       
print(sq/len(true))                                                                                                                                          

If you save it as min_eg.py, run it (after installing xlearn) as python min_eg.py simply.

Here is the output you get:

Interestingly the MSE you get is always exactly double of the reported mse from the predict function.

Any help is greatly appreciated; and I wonder if other folks have encountered the same issue.

Answer

A lot of people use 1/2 MSE for the loss because it makes the derivative “easier”. Given that they use the word “loss” rather than “MSE” or something like that, I’d bet this is what’s going on.

For clarity, if your loss is

1/2n * [(y_1 – p_1)^2 + … + (y_n – p_n)^2]

then the derivative (wrt p) would be

-1/n * [(y_1 – p_1) + … + (y_n – p_n)]

The 2 goes away because you end up multiplying by 2 for the power rule.

pardon the formatting… I don’t know how to do math stuff here.