Tools Home : Plain Text Tools : Tokenize Text

Click here to show HTML tools HTML Tools

Click here to expand XML tools XML tools

Click here to expand plain text tools Plain Text Tools

Click here to expand other tools Other tools

 Beta tools
 Add Tools Demo
 Manual
 About

Tokenize Text with Specified Separators
?
Summary

This tool splits a text (.txt) document, located either at a user-specified web address or uploaded from the user’s files, at specified points (tokens). Tokens may be words, lines, sentences or paragraphs, as well as specific characters or pattern. The results can be listed with or without the token, and with the split made before or after the token.

Note: This tool works best when the input is plain text. If an HTML or XML text is submitted, the tool will strip all tags and process it as plain text.

Please click the ? buttons at the bottom right of each set of options for more information on that set.

For further information on this tool, please see the TADA Wiki's Tokenize entry here. A glossary of terms is also available here.

Walkthrough

To process the text found within http://tada.mcmaster.ca/wikita/pub/Main/ToolTestingTexts/GulliversTravels.txt, split the resulting text into sentences before the token, preserve the token, and display the results as HTML:
  1. Source text
    1. Enter ‘http://tada.mcmaster.ca/wikita/pub/Main/ToolTestingTexts/GulliversTravels.txt’ into the ‘URL’ field.
  2. Token types
    1. Click the radio button next to 'Sentences'.
  3. Results
    1. Set the ‘Display options’ drop menu to ‘keep with previous token’.
    2. Set the ‘Display as’ drop menu to ‘HTML’.
    3. Click the ‘Submit’ button to process the text.
*
» Source text
  Example: http://taporware.ualberta.ca/sampleDocs/plainText.txt


?
Summary

This section determines the source of the document you wish the tool to process.

Fields

Source URL
To use content from a web page, enter a full web address (URL) in the field provided. Copy and paste from your browser’s address bar for best results. If the web address directs to an HTML or XML document instead of plain text, the tool will strip all tags and process it as plain text.

Local file
To upload a plain text (.txt) file from your computer, choose ‘Local file,’ click ‘Browse,’ and select the file you wish to use from your directory.

*
» Token types





?
Summary

This section determines what to process as a token.

Fields

Words
Splits text on individual words.

Lines
Splits text line by line.

Sentences
Splits text sentence by sentence.

Paragraphs
Splits text paragraph by paragraph.

Characters
Splits text along a user-specified character (separated by spaces). To include a space use ^s.

Pattern
Splits text along a use-specified pattern, either in Unix format or as a regular expression.

Unix Format
Splits the text by tokens based on a Unix format regular expression.

Regular Exp
Splits the text by tokens found with a regular expression. Do not use \d, \D, \w, \W etc. Instead, please use [0-9], [^0-9], [a-zA-Z] or [^a-zA-Z]. Using \n, \r, or \t is fine.
» Results
?
Summary

This section determines how the tool’s final results will be displayed.

Fields

Display options
This drop menu allows users to treat the token one of four ways in the final results: strip out the separator, keep the separator as a token, keep the separator with the previous token, or keep the separator with the following token.

Display as
This drop-down lists enables users to choose from several output formats: HTML, XML text in HTML, XML tree, and Tab delimited text.

Open results in new window
Check this box to display the results in a new window or browser tab. This option is selected by default. In some cases pop-up blockers may disallow windows from being created, in which case this option may be de-selected.
`*' indicates a required field
`' do not use \d, \D, \w, \W etc. use [0-9], [^0-9], [a-zA-Z] or [^a-zA-Z] instead. Using \n, \r, \t is fine.

 

 

TAPoRware Project, McMaster University,