Skip to content
Advertisement

IndexError: tuple index out of range in LabelEncoder Sklearn

I would like to train a DecisionTree using sklearn Pipeline. My goal is to predict the ‘language’ column, using the ‘tweet’ as ngram transformed features. However I am not able to make the LabelEncoder transformation works for the ‘language’ column inside a pipeline. I saw that there is a common error, but also if I try the suggested method to reshape I am still no able to overcame the problem. This is my df:

JavaScript
JavaScript
JavaScript

But the error is still the same.

JavaScript

Many thanks!

Advertisement

Answer

Imo there are a couple of main issues linked to the way you’re dealing with your CountVectorizer instance.

First off, CountVectorizer requires 1D input, in which case (I mean with such transformers) ColumnTransformer requires parameter column to be passed as a scalar string or int; you might find a detailed explanation in sklearn .fit transformers , IndexError: tuple index out of range. Therefore you should pass 'tweet' rather than ['tweet'] (or 0 rather than [0] in case you’d specify columns positionally) to the ColumnTransformer instance.

Then, according to the documentation of CountVectorizer,

input: {‘filename’, ‘file’, ‘content’}, default=’content’

  • If ‘filename’, the sequence passed as an argument to fit is expected to be a list of filenames that need reading to fetch the raw content to analyze.

  • If ‘file’, the sequence items must have a ‘read’ method (file-like object) that is called to fetch the bytes in memory.

  • If ‘content’, the input is expected to be a sequence of items that can be of type string or byte.

you should better pass to its instance an input which is a sequence of items that can be of type string or byte. That’s the point of passing X_train.values rather than X_train only: you’ll get away from a pandas series, which would not respect the specification given in the docs. Eventually, I’m not completely sure why reshaping is needed given that you’d obtain an iterable of strings even with X_train.values, but effectively X_train.values.reshape(-1, 1) should be your way to go in such case.

Last point, personally I wouldn’t deal with target transformation in a ColumnTransformer or in a Pipeline: they’re meant to transform the features only. Some details can be found in my answer to Why should LabelEncoder from sklearn be used only for the target variable?.

All in all, this might be one way of solving your problem:

JavaScript
User contributions licensed under: CC BY-SA
5 People found this is helpful
Advertisement