string input = "The quick brown fox jumps over the lazy dog."; ITokenizer tokenizer = new WhitespaceTokenizer(); // using WhitespaceTokenizer IEnumerabletokens = tokenizer.Tokenize(input); foreach (string token in tokens) { Console.WriteLine(token); }
The quick brown fox jumps over the lazy dog.
string input = "I am a sentence. I am a second sentence."; ITokenizer tokenizer = new SentenceTokenizer(); // using SentenceTokenizer IEnumerabletokens = tokenizer.Tokenize(input); foreach (string token in tokens) { Console.WriteLine(token); }
I am a sentence. I am a second sentence.Both examples show how ITokenizer can be implemented with different tokenizers to suit various needs. The package library used in these examples is not specified, as ITokenizer could potentially be used with any package that provides tokenizing functionality.