Count the resulting words of the stopWord list

I have performed the data cleaning of my dataframe with pyspark, including the removal of the Stop-Words. Removing the Stop-Word produces a list for each line, containing words that are NOT Stop-Words. Now I would like to count all the words left in that column, to make the Word-Cloud or the Word-Frequency.
This is my pyspark dataframe:

+--------------------+-----+-----+-----------+--------------------+--------------------+--------------------+
|             content|score|label|classWeigth|               words|            filtered|       terms_stemmed|
+--------------------+-----+-----+-----------+--------------------+--------------------+--------------------+
|absolutely love d...|    5|    1|       0.48|[absolutely, love...|[absolutely, love...|[absolut, love, d...|
|absolutely love t...|    5|    1|       0.48|[absolutely, love...|[absolutely, love...|[absolut, love, g...|
|absolutely phenom...|    5|    1|       0.48|[absolutely, phen...|[absolutely, phen...|[absolut, phenome...|
|absolutely shocki...|    1|    0|       0.52|[absolutely, shoc...|[absolutely, shoc...|[absolut, shock, ...|
|accept the phone ...|    1|    0|       0.52|[accept, the, pho...|[accept, phone, n...|[accept, phone, n...|
+--------------------+-----+-----+-----------+--------------------+--------------------+--------------------+

terms_stemmed is the final column, from which I would like to get a new data frame like the following:

+-------------+--------+
|terms_stemmed| count  |
+-------------+--------+
|app          |  592059|
|use          |  218178|
|good         |  187671|
|like         |  155304|
|game         |  149941|
|....         |   .... |

Someone can help me?

Answer

One option is to use explode

import pyspark.sql.functions as F

new_df = df
  .withColumn('terms_stemmed', F.explode('terms_stemmed'))
  .groupby('terms_stemmed')
  .count()

Example

import pyspark.sql.functions as F

df = spark.createDataFrame([
  (1, ["Apple", "Banana"]),
  (2, ["Banana", "Orange", "Banana"]),
  (3, ["Orange"])
], ("id", "terms_stemmed"))

df.show(truncate=False)

+---+------------------------+
|id |terms_stemmed           |
+---+------------------------+
|1  |[Apple, Banana]         |
|2  |[Banana, Orange, Banana]|
|3  |[Orange]                |
+---+------------------------+



new_df = df
  .withColumn('terms_stemmed', F.explode('terms_stemmed'))
  .groupby('terms_stemmed')
  .count()

new_df.show()

+-------------+-----+
|terms_stemmed|count|
+-------------+-----+
|       Banana|    3|
|        Apple|    1|
|       Orange|    2|
+-------------+-----+