the xlearn predict function gives a different mse than what you get by looking at the predictions and calculating it yourself. Here is code to do this; you can run it by cloning the xlearn repository and copying the below code in demo/regression/house_price in the repository
# Copyright (c) 2018 by contributors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import xlearn as xl
import pandas as pd
from sklearn.metrics import mean_squared_error
# Training task
# param:
# 0. regression task
# 1. learning rate: 0.2
# 2. regular lambda: 0.002
# 3. evaluation metric: mae
fm_model = xl.create_linear() # Use factorization machine
fm_model.setTrain("./house_price_train.txt") # Training data
fm_model.setValidate("./house_price_test.txt") # Validation data
fm_model.disableNorm()
# fm_model.setSigmoid()
# fm_model.disableEarlyStop()
param = {'task':'reg', 'lr':0.0002,
'lambda':0.00001, 'metric':'rmse', 'epoch':100}
# Start to train
# The trained model will be stored in model.out
print("here")
fm_model.fit(param, './model.out')
fm_model.setTest("./house_price_test.txt") # Test data
# Prediction task
# Start to predict
# The output result will be stored in output.txt
outs = fm_model.predict("./model.out", "./output.txt")
true = pd.read_csv("./house_price_test.txt", sep='t', header=None)[0]
# print(true)
preds = pd.read_csv("./output.txt", header=None)[0]
# Calculate using sklearn
print(mean_squared_error(true, preds))
# Self calculate
sq = 0.0
for t, p in zip(true, preds):
sq += (t - p) ** 2
print(sq/len(true))
If you save it as min_eg.py, run it (after installing xlearn) as
python min_eg.py simply.
Here is the output you get:
Interestingly the MSE you get is always exactly double of the reported mse from the predict function.
Any help is greatly appreciated; and I wonder if other folks have encountered the same issue.
Advertisement
Answer
A lot of people use 1/2 MSE for the loss because it makes the derivative “easier”. Given that they use the word “loss” rather than “MSE” or something like that, I’d bet this is what’s going on.
For clarity, if your loss is
1/2n * [(y_1 – p_1)^2 + … + (y_n – p_n)^2]
then the derivative (wrt p) would be
-1/n * [(y_1 – p_1) + … + (y_n – p_n)]
The 2 goes away because you end up multiplying by 2 for the power rule.
pardon the formatting… I don’t know how to do math stuff here.