pyspark.sql.functions.regexp_replace#

pyspark.sql.functions.regexp_replace(string, pattern, replacement, position=None)[source]#

Replace all substrings of the specified string value that match regexp with replacement.

Added in version 1.5.0.

Changed in version 3.4.0: Supports Spark Connect.

Changed in version 4.3.0: Supports the position parameter.

Parameters:

stringColumn or str: column name or column containing the string value. A column that evaluates to a string.
patternColumn or str: column object or str containing the regexp pattern. A column that evaluates to a string.
replacementColumn or str: column object or str containing the replacement. A column that evaluates to a string.
positionColumn or int, optional: position to start replacement. The first position is 1. A column that evaluates to an integer. Must be a constant.

Returns:

Column: string with all substrings replaced. Returns a column that evaluates to a string.

Examples

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame(
...      [("100-200", r"(\d+)", "--")],
...      ["str", "pattern", "replacement"]
... )

Example 1: Replaces all the substrings in the str column name that match the regex pattern (d+) (one or more digits) with the replacement string “–“.

>>> df.select('*', sf.regexp_replace('str', r'(\d+)', '--')).show()
+-------+-------+-----------+---------------------------------+
|    str|pattern|replacement|regexp_replace(str, (\d+), --, 1)|
+-------+-------+-----------+---------------------------------+
|100-200|  (\d+)|         --|                            -----|
+-------+-------+-----------+---------------------------------+

Example 2: Replaces all the substrings in the str Column that match the regex pattern in the pattern Column with the string in the replacement column.

>>> df.select('*', \
...     sf.regexp_replace(sf.col("str"), sf.col("pattern"), sf.col("replacement")) \
... ).show()
+-------+-------+-----------+--------------------------------------------+
|    str|pattern|replacement|regexp_replace(str, pattern, replacement, 1)|
+-------+-------+-----------+--------------------------------------------+
|100-200|  (\d+)|         --|                                       -----|
+-------+-------+-----------+--------------------------------------------+

Example 3: Replaces substrings starting from the specified position. For the input string “100-200”, position 5 starts replacement after “100-“.

>>> df.select(sf.regexp_replace("str", r"(\d+)", "--", 5).alias("d")).show()
+------+
|     d|
+------+
|100---|
+------+