I thought of implementing kappaScore metrics using sklearn.metrics.cohen_kappa_score
def kappaScore(y_true,y_pred): k = cohen_kappa_score(y_true,y_pred,weights='quadratic') return k
Error I get when I try to run this code:
OperatorNotAllowedInGraphError: in user code: /opt/conda/lib/python3.7/site-packages/keras/engine/training.py:853 train_function * return step_function(self, iterator) /tmp/ipykernel_33/1006337667.py:2 kappaScore * k = cohen_kappa_score(y_true,y_pred,weights='quadratic') /opt/conda/lib/python3.7/site-packages/sklearn/utils/validation.py:555 inner_f * return f(**kwargs) /opt/conda/lib/python3.7/site-packages/sklearn/metrics/_classification.py:600 cohen_kappa_score * confusion = confusion_matrix(y1, y2, labels=labels, /opt/conda/lib/python3.7/site-packages/sklearn/utils/validation.py:555 inner_f * return f(**kwargs) /opt/conda/lib/python3.7/site-packages/sklearn/metrics/_classification.py:276 confusion_matrix * y_type, y_true, y_pred = _check_targets(y_true, y_pred) /opt/conda/lib/python3.7/site-packages/sklearn/metrics/_classification.py:81 _check_targets * check_consistent_length(y_true, y_pred) /opt/conda/lib/python3.7/site-packages/sklearn/utils/validation.py:253 check_consistent_length * uniques = np.unique(lengths) <__array_function__ internals>:6 unique ** /opt/conda/lib/python3.7/site-packages/numpy/lib/arraysetops.py:261 unique ret = _unique1d(ar, return_index, return_inverse, return_counts) /opt/conda/lib/python3.7/site-packages/numpy/lib/arraysetops.py:322 _unique1d ar.sort() /opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:900 __bool__ self._disallow_bool_casting() /opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:504 _disallow_bool_casting "using a `tf.Tensor` as a Python `bool`") /opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:491 _disallow_when_autograph_enabled " indicate you are trying to use an unsupported feature.".format(task)) OperatorNotAllowedInGraphError: using a `tf.Tensor` as a Python `bool` is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.
Here the type of y_true
and y_pred
requires to be in list or numpy array
But the type of y_true
and y_pred
are,
y_true: <class 'tensorflow.python.framework.ops.Tensor'> y_pred: <class 'tensorflow.python.framework.ops.Tensor'>
When directly try to print it (i.e, without type() function), it shows like this:
y_true: Tensor("IteratorGetNext:1", shape=(None, None), dtype=float32) y_pred: Tensor("sequential_5/dense_5/Softmax:0", shape=(None, 5), dtype=float32)
Unable to use y_true.numpy()
(Convert a tensor to numpy array in Tensorflow?) and tf.make_ndarray(y_true)
(https://www.tensorflow.org/api_docs/python/tf/make_ndarray#:~:text=tf.make_ndarray(proto_tensor)) tried it..
How can I convert these datatypes in a way that, it can be passed to sklearn.metrics.cohen_kappa_score function? I don’t want to write a code for kappa score. Is it possible to convert?
Advertisement
Answer
There’s a way to solve this problem wrapping cohen_kappa_score in tf.py_function
. It’s available in tensorflow 2.x, but I don’t know since which version of framework; py_function
does all heavy lifting for you, wrapping a Python function into a TensorFlow operation that executes it eagerly.
import tensorflow as tf import numpy as np from sklearn.metrics import cohen_kappa_score def cohen_kappa_score_wrapper(y_true, y_pred): y_pred = np.argmax(y_pred, axis=1) return cohen_kappa_score(y_true, y_pred, weights='quadratic') def kappaScore(y_true, y_pred): return tf.py_function( func=cohen_kappa_score_wrapper, inp=[y_true, y_pred], Tout=tf.float32 ) model.compile('adam', 'sparse_categorical_crossentropy', metrics=[kappaScore])
First, define cohen_kappa_score_wrapper
. It’s important, because last dense layer usually returns an array of probabilities of each class for each sample. But Cohen kappa score accepts integer labels of classes, so one has to convert probabilities into labels with np.argmax()
. We’re still on the Python’s territory, so could just use numpy functions.
Then wrap cohen_kappa_score_wrapper
with py_function
: see kappaScore
.
Complete example using MNIST:
import keras import tensorflow as tf from keras.layers import Input, LSTM, RepeatVector, TimeDistributed, Dense from keras import Model import numpy as np from sklearn.metrics import cohen_kappa_score from keras.datasets import mnist (x_train, x_test), (y_train, y_test) = mnist.load_data() model = keras.Sequential([ keras.layers.Flatten(), keras.layers.Dense(128, activation='relu'), keras.layers.Dense(10, activation='softmax'), ]) def cohen_kappa_score_wrapper(y_true, y_pred): y_pred = np.argmax(y_pred, axis=1) return cohen_kappa_score(y_true, y_pred, weights='quadratic') def kappaScore(y_true, y_pred): return tf.py_function( func=cohen_kappa_score_wrapper, inp=[y_true, y_pred], # weights='quadratic'] Tout=tf.float32 ) model.compile('adam', 'sparse_categorical_crossentropy', metrics=[kappaScore]) model.fit(x_train / 255., x_test)
60000/60000 [==============================] - 5s 90us/sample - loss: 0.2612 - kappaScore: 0.9202 <keras.callbacks.History at 0x7fcea289cc50>
Note
Despite the docs states that wrapped function executes eagerly, it still works for me if I turn off eager execution: tf.compat.v1.disable_eager_execution()
. I use tf2.7
. But I’m not so confident about other versions of the framework/different environments. It could be tricky sometimes. Also, if you use tf1.x
, it could be a different story.