Sorted dataframe returns data in unordered way while joining the two dataframes

I have two dataframes players and striker_details. The striker_details df is sorted by striker_grade and looks like this:

+-------------+-------------+
|player_api_id|striker_grade|
+-------------+-------------+
|        20276|        89.25|
|        37412|         89.0|
|        38817|        88.75|
|        32118|        88.25|
|        31921|         87.0|
+-------------+-------------+ 

The other dataframe players is an unordered dataframe and looks like this:

+---+-------------+------------------+------------------+-------------------+------+------+
| id|player_api_id|       player_name|player_fifa_api_id|           birthday|height|weight|
+---+-------------+------------------+------------------+-------------------+------+------+
|  1|       505942|Aaron Appindangoye|            218353|1992-02-29 00:00:00|182.88|   187|
|  2|       155782|   Aaron Cresswell|            189615|1989-12-15 00:00:00|170.18|   146|
|  3|       162549|       Aaron Doran|            186170|1991-05-13 00:00:00|170.18|   163|
|  4|        30572|     Aaron Galindo|            140161|1982-05-08 00:00:00|182.88|   198|
|  5|        23780|      Aaron Hughes|             17725|1979-11-08 00:00:00|182.88|   154|
+---+-------------+------------------+------------------+-------------------+------+------+

When I am trying to join the two dataframes using pyspark join, it is returning an unordered data:

+-------------+-----------------+------------------+-------------------+------+------+
|player_api_id|    striker_grade|       player_name|           birthday|height|weight|
+-------------+-----------------+------------------+-------------------+------+------+
|       309726|75.38888888888889|    Andrea Belotti|1993-12-20 00:00:00|180.34|   159|
|        38433|          72.5625|      Borja Valero|1985-01-12 00:00:00|175.26|   161|
|        41157|             82.0|Giovani dos Santos|1989-05-11 00:00:00|175.26|   163|
|        40740|           70.375|      Jeremy Morel|1984-04-02 00:00:00|172.72|   157|
|       109653|             73.5|     John Goossens|1988-07-25 00:00:00|175.26|   150|
+-------------+-----------------+------------------+-------------------+------+------+

I used the command: ss = striker_details.join(players, ["player_api_id"], "inner")

How can I achieve sorted data?

Answer

Try chaining orderBy('striker_grade,ascending=False) at the end of your join:

ss = striker_details.join(players, ["player_api_id"], "inner").orderBy('striker_grade',ascending=False)

ss.display()

Also, the default join is inner, so you don’t need to specify it. It is good to show it for explicitness though.