Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Python string multireplacement
def multireplace(string, replacements, ignore_case=False):
"""
Given a string and a replacement map, it returns the replaced string.
:param str string: string to execute replacements on
:param dict replacements: replacement dictionary {value to find: value to replace}
:param bool ignore_case: whether the match should be case insensitive
:rtype: str
"""
# If case insensitive, we need to normalize the old string so that later a replacement
# can be found. For instance with {"HEY": "lol"} we should match and find a replacement for "hey",
# "HEY", "hEy", etc.
if ignore_case:
def normalize_old(s):
return s.lower()
re_mode = re.IGNORECASE
else:
def normalize_old(s):
return s
re_mode = 0
replacements = {normalize_old(key): val for key, val in replacements.items()}
# Place longer ones first to keep shorter substrings from matching where the longer ones should take place
# For instance given the replacements {'ab': 'AB', 'abc': 'ABC'} against the string 'hey abc', it should produce
# 'hey ABC' and not 'hey ABc'
rep_sorted = sorted(replacements, key=len, reverse=True)
rep_escaped = map(re.escape, rep_sorted)
# Create a big OR regex that matches any of the substrings to replace
pattern = re.compile("|".join(rep_escaped), re_mode)
# For each match, look up the new string in the replacements, being the key the normalized old string
return pattern.sub(lambda match: replacements[normalize_old(match.group(0))], string)
@derdav3

This comment has been minimized.

Copy link

derdav3 commented Apr 6, 2017

how would you modify this to have a case-insensitive match?

@HatScripts

This comment has been minimized.

Copy link

HatScripts commented Apr 27, 2017

@derdav3

Edit 2019-11-25:
Removed my code since it had bugs, as pointed out by @bgusach (OP) and @sidscry, and as fixed by @thorfi.
Props to OP for updating their code to support case-insensitive matching.

@bgusach

This comment has been minimized.

Copy link
Owner Author

bgusach commented May 10, 2017

@derdav3, as @HatScripts suggested, just pass the ignore-case flag to re.compile.

@HatScripts, I haven't tested your proposal, but... aren't you sorting the strings by the length of the first character (i.e. always 1?)

@sidscry

This comment has been minimized.

Copy link

sidscry commented Feb 6, 2018

@HatScripts This case fails
string = "original text is here"
replacements = {
"original": "text",
"text" : "fake",
"Is hEre": "was there"
}
ignore_case = True

@thorfi

This comment has been minimized.

Copy link

thorfi commented Jul 11, 2018

@sidscry @HatScripts @bgusach:
Bugfixes for the above, replace: rep_sorted = ... with:

    if ignore_case:
        replacements = dict((pair[0].lower(), pair[1]) for pair in sorted(replacements.iteritems()))
    rep_sorted = sorted(replacements, key=lambda s: (len(s), s), reverse=True)
    ...
    return pattern.sub(lambda match: replacements[match.group(0).lower() if ignore_case else match.group(0)], string)
@Gutsu7

This comment has been minimized.

Copy link

Gutsu7 commented Sep 2, 2019

How is it possible to use a pandas DataFrame for replacement instead of using a dictionary?

@bgusach

This comment has been minimized.

Copy link
Owner Author

bgusach commented Sep 3, 2019

How is it possible to use a pandas DataFrame for replacement instead of using a dictionary?

I'm not familiar with pandas DataFrames, but I guess you can convert that data structure into a dictionary in a meaningful way and then use this function.

@bgusach

This comment has been minimized.

Copy link
Owner Author

bgusach commented Oct 2, 2019

Hi @thorfi and @HatScripts, I updated the gist with a solution inspired in yours.

@kungfoomanchu

This comment has been minimized.

Copy link

kungfoomanchu commented Nov 23, 2019

This is a great script. I have two novice questions:

  1. Is there an easy change that would allow me to feed in a list to be replaced instead of a string?
  2. Is there a way for the list of replacements to contain regular expressions? Something like this:
    replacements = { /\bcheese\b/g, 'cheesey', /\bbig\b/g, 'bigger' }
@HatScripts

This comment has been minimized.

Copy link

HatScripts commented Nov 25, 2019

@kungfoomanchu

  1. Is there an easy change that would allow me to feed in a list to be replaced instead of a string?

Yes, you can simply use a list comprehension to call multireplace for each string in your list.

strings = ["who is he?", "who did he see?", "who will he become?"]
replacements = {"he" : "she", "who": "what"}
strings = [multireplace(string, replacements) for string in strings]
print(strings)  # ['what is she?', 'what did she see?', 'what will she become?']

  1. Is there a way for the list of replacements to contain regular expressions? Something like this:
    replacements = { /\bcheese\b/g, 'cheesey', /\bbig\b/g, 'bigger' }

If you mean you want all of your matches to be surrounded by word boundaries (\b), you would just need to change this line within the multireplace function:

pattern = re.compile("|".join(rep_escaped), re_mode)

to this:

pattern = re.compile(r"\b(" + "|".join(rep_escaped) + r")\b", re_mode)

If instead you mean you want to be able to pass in any regular expressions as your needles, it gets a bit more complicated, and the multireplace function would need to be essentially rewritten I believe.

If your replacements are simple enough, you could just do something like this:

import re

def multi_replace_regex(string, replacements, ignore_case=False):
    for pattern, repl in replacements.items():
        string = re.sub(pattern, repl, string, flags=re.I if ignore_case else 0)
    return string

replacements = {
    "bee?"     : "b",
    "t(oo?|wo)": "2",
    "not"      : "!",
    "question" : "?",
}
string = "to be or not to be, that is the question"
string = multi_replace_regex(string, replacements)
print(string)  # 2 b or ! 2 b, that is the ?

Important to note is that this will give unexpected results with many/complicated overlapping patterns and replacements, because unlike OP's multireplace function, this does not do the replacements in a single pass. Thus, the order of the dictionary may be important to you. If so, I would recommend using an OrderedDict, unless you are using Python 3.7 or later, wherein it is guaranteed that the built-in dict class will remember insertion order.

@kungfoomanchu

This comment has been minimized.

Copy link

kungfoomanchu commented Dec 5, 2019

@HatScripts This is fantastic, thank you very much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.