Extract sentence embeddings features with Pandas and spaCy

I’m currently learning spaCy, and I have an exercise on word and sentence embeddings. Sentences are stored in a pandas DataFrame columns, and, we’re requested to train a classifier based on the vector of these sentences.

I have a dataframe that looks like this:

+---+---------------------------------------------------+
|   |                                          sentence |
+---+---------------------------------------------------+
| 0 | "Whitey on the Moon" is a 1970 spoken word poe... |
+---+---------------------------------------------------+
| 1 | St Anselm's Church is a Roman Catholic church ... |
+---+---------------------------------------------------+
| 2 | Nymphargus grandisonae (common name: giant gla... |
+---+---------------------------------------------------+

Next, I apply an NLP function to these sentences:

import en_core_web_md
nlp = en_core_web_md.load()
df['tokenized'] = df['sentence'].apply(nlp)

Now, if I understand correctly, each item in df[‘tokenized’] has an attribute that returns the vector of the sentence in a 2D array.

print(type(df['tokenized'][0].vector))
print(df['tokenized'][0].vector.shape)

yields

<class 'numpy.ndarray'>
(300,)

How do I add the content of this array (300 rows) as columns to the df dataframe for the corresponding sentence, ignoring stop words?

Thanks!

Answer

Actually, using a single value averaging all vectors does yield good results in a classification model. What was needed was indeed a dataframe of 300 columns per sentence (since 300 is the standard length of spaCy word embeddings:

So, to continue @Sergey’s code:

sents = ["'Whitey on the Moon' is a 1970 spoken word"
         , "St Anselm's Church is a Roman Catholic church"
         , "Nymphargus grandisonae (common name: giant)"]

df=pd.DataFrame({"sentence":sents})

df['tokenized'] = df['sentence'].apply(nlp)
df['sent_vectors'] = df['tokenized'].apply(lambda x: x.vector)
vectors = 0['sent_vector'].apply(pd.Series)

With this, vectors contains the features of which a model can be trained. For instance, assuming each sentence has a sentiment attached to it:

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X = vectors
y = df['sentiment']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

clf = LogisticRegression()
clf.fit(X_train,y_train)
y_pred = clf.predict(X_test)

What I couldn’t do is to remove stopwords from the DataFrame entries (i.e. remove each Token object from the Doc parent object stored in the dataframe where is_stop is False.