I have a list which contains some words and I need to extract matching words from a text line, I found this, but it only extracts one word.
keys file content
this is a keyword
part_description file content
32015 this is a keyword hello world
Code
import pyspark.sql.functions as F keywords = sc.textFile('file:///home/description_search/keys') #1 part_description = sc.textFile('file:///description_search/part_description') #2 keywords = keywords.map(lambda x: x.split(' ')) #3 keywords = keywords.collect()[0] #4 df = part_description.map(lambda r: Row(r)).toDF(['line']) #5 df.withColumn('extracted_word', F.regexp_extract(df['line'],'|'.join(keywords), 0)).show() #6
Outputs
+--------------------+--------------+ | line|extracted_word| +--------------------+--------------+ |32015 this is a...| this| +--------------------+--------------+
Expected output
+--------------------+-----------------+ | line| extracted_word| +--------------------+-----------------+ |32015 this is a...|this,is,a,keyword| +--------------------+-----------------+
I want to
return all matching keyword and their count
and if
step #4
is the most effecient way
Reproducible example:
keywords = ['this','is','a','keyword'] l = [('32015 this is a keyword hello world' , ), ('keyword this' , ), ('32015 this is a keyword hello world 32015 this is a keyword hello world' , ), ('keyword keyword' , ), ('is a' , )] columns = ['line'] df=spark.createDataFrame(l, columns)
Advertisement
Answer
I managed to solve it by using UDF instead as below
def build_regex(keywords): res = '(' for key in keywords: res += '\b' + key + '\b|' res = res[0:len(res) - 1] + ')' return res def get_matching_string(line, regex): matches = re.findall(regex, line) return matches if matches else None udf_func = udf(lambda line, regex: get_matching_string(line, regex), ArrayType(StringType())) df = df.withColumn('matched', udf_func(df['line'], F.lit(build_regex(keywords)))).withColumn('count', F.size('matched'))
Result
+--------------------+--------------------+-----+ | line| matched|count| +--------------------+--------------------+-----+ |32015 this is ...|[this, is, this, ...| 5| |12832 Shb is a...| [is, a]| 2| |35015 this is ...| [this, is]| 2| +--------------------+--------------------+-----+