Here’s some data from another question: What I would do first is to add quotes across all words, and then: Is there a smarter way to do this? Answer Lists of strings For basic structures you can use yaml without having to add quotes: Lists of numeric data Under certain conditions, you can read your list…
Tag: dataframe
plot multiple pandas dataframes in one graph
I have created 6 different dataframes that eliminate the outliers of their own original data frames. Now, I’m trying to plot all of the dataframes that eliminate the outliers on the same graph. This is my code that eliminates the outliers in each data frame: If I remove the comment newdf.plot() I will b…
How to read a list of parquet files from S3 as a pandas dataframe using pyarrow?
I have a hacky way of achieving this using boto3 (1.4.4), pyarrow (0.4.1) and pandas (0.20.3). First, I can read a single parquet file locally like this: I can also read a directory of parquet files locally like this: Both work like a charm. Now I want to achieve the same remotely with files stored in a S3 bu…
Join pandas dataframes based on column values
I’m quite new to pandas dataframes, and I’m experiencing some troubles joining two tables. The first df has just 3 columns: DF1: And the second has exactly same two columns (and plenty of others): DF2: What I need is to perform an operation which, in SQL, would look as follows: And, as a result, I…
python pandas how to compute on rows with same index values
I have a dataframe called resulttable that looks like: where df Index values are the index values when resulttable is printed or exported to xls, Tag = str, and Exp. m/z, Intensity, and Norm_Intensity are float64. The tag values will be coming from the file names in a specified folder, so they can vary. As yo…
Comparison of a Dataframe column values with a list
Consider this Dataframe: This is the code to get values of column C, where it is the first row of each group (Column A): So first will be: (100, 200, 300). Now I want to add new column which it will be 1 if value of column C for row is in firsts otherwise it will be 0. A B
Pandas Dataframe – Shifting rows down and maintaining data
My original Dataframe (df): I want to shift the values down by 6 like so: When I use df = df.shift(6), I end up loosing data. I found this post (How to shift a column in Pandas DataFrame without losing value) but it only seems to work if the values are shifted down by 1. How can I shift multiple
PySpark: Get first Non-null value of each column in dataframe
I’m dealing with different Spark DataFrames, which have lot of Null values in many columns. I want to get any one non-null value from each of the column to see if that value can be converted to datetime. I tried doing df.na.drop().first() in a hope that it’ll drop all rows with any null value, and…
Python Pandas iterate over rows and access column names
I am trying to iterate over the rows of a Python Pandas dataframe. Within each row of the dataframe, I am trying to to refer to each value along a row by its column name. Here is what I have: I used this approach to iterate, but it is only giving me part of the solution – after selecting a
Remove ‘seconds’ and ‘minutes’ from a Pandas dataframe column
Given a dataframe like: I would like to remove the ‘minutes’ and ‘seconds’ information. The following (mostly stolen from: How to remove the ‘seconds’ of Pandas dataframe index?) works okay, but it feels strange to convert a datetime to a string then back to a datetime. Is …