Skip to content

Instantly share code, notes, and snippets.

@gboeer
Created September 21, 2011 07:33
Show Gist options
  • Save gboeer/1231475 to your computer and use it in GitHub Desktop.
Save gboeer/1231475 to your computer and use it in GitHub Desktop.
C++ String Tokenizer
#include <string>
#include <vector>
//! Tokenize the given string str with given delimiter. If no delimiter is given whitespace is used.
void Tokenize(const std::string& str, std::vector<std::string>& tokens, const std::string& delimiters = " ")
{
tokens.clear();
// Skip delimiters at beginning.
std::string::size_type lastPos = str.find_first_not_of(delimiters, 0);
// Find first "non-delimiter".
std::string::size_type pos = str.find_first_of(delimiters, lastPos);
while (std::string::npos != pos || std::string::npos != lastPos)
{
// Found a token, add it to the vector.
tokens.push_back(str.substr(lastPos, pos - lastPos));
// Skip delimiters. Note the "not_of"
lastPos = str.find_first_not_of(delimiters, pos);
// Find next "non-delimiter"
pos = str.find_first_of(delimiters, lastPos);
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment