Created
February 7, 2018 21:09
-
-
Save pete-rai/27c5056fd38d484553000ed92f895895 to your computer and use it in GitHub Desktop.
A text cleansing function that is useful for preparing strings prior to lexical analysis.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
<?php | |
function cleanse ($text) | |
{ | |
$text = iconv ('UTF-8', 'ASCII//TRANSLIT//IGNORE', $text); // accented character to 'normal' | |
$text = preg_replace ('/[\r\n\s\t]+/xms', ' ' , $text); // normalise whitespace to one space | |
$text = preg_replace ('/[^\w\s]+/xms' , '' , $text); // remove all punctuation | |
return strtolower (trim ($text)); // lowercase and trimmed | |
} | |
?> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment