#Function
In PySpark, the concat_ws() function is used to concatenate multiple columns or strings into a single string, with a specified delimiter between them.
Syntax: -
from pyspark.sql.functions import concat_ws
concat_ws(delimiter, *cols)
Example 1: Concatenate two columns with a space separator
Assume you have a DataFrame with the columns First_Name and Last_Name.
from pyspark.sql.functions import concat_ws, col
# Concatenate First_Name and Last_Name with a space separator
df1 = df.withColumn("Full_Name", concat_ws(" ", col("First_Name"), col("Last_Name")))
df1.show()
| Feature | concat |
concat_ws |
|---|---|---|
| Separator | No separator is used. | Requires a separator. |
| Input | Columns or literals only. | Separator + columns/literals. |
| Use Case | Merge values directly. | Merge values with a delimiter. |