Tools Home : XML Tools : Tokenize XML Document

Click here to show HTML tools HTML Tools

Click here to expand XML tools XML tools

Click here to expand plain text tools Plain Text Tools

Click here to expand other tools Other tools

 Beta tools
 Add Tools Demo

Tokenize XML Document

This tool splits an XML (.xml) document, located either at a user-specified web address or uploaded from the user’s files, at specified points (tokens). Tokens may be words, lines, sentences or paragraphs, as well as specific characters, patterns or XML elements and attributes. The results can be listed with or without the token, and with the split made before or after the token.

Please click the ? buttons at the bottom right of each set of options for more information on that set.

For further information on this tool, please see the TADA Wiki's Tokenize entry here. A glossary of terms is also available here.


To process the text found within; extract text between <para> and </para>, split the resulting text into sentences before the token, preserve the token, and display the results as HTML:
  1. Source text
    1. Enter ‘’ into the ‘URL’ field.
  2. Subtext limited to
    1. Enter ‘para’ in the ‘Elements’ field, leaving ‘Attribute name’ and ‘Attribute value’ blank.
  3. Token types
    1. Click the radio button next to 'Sentences'.
  4. Results
    1. Set the ‘Display options’ drop menu to ‘keep with previous token’.
    2. Set the ‘Display as’ drop menu to ‘HTML’.
    3. Click the ‘Submit’ button to process the text.
» Source text


This section determines the source of the document you wish the tool to process. XML can be obtained either from a web address or by uploading a file.


Source URL
To use content from a web page, enter a full web address (URL) ending in .html in the field provided. Copy and paste from your browser’s address bar for best results.

Local file
To upload an XML (.xml) file from your computer, choose ‘Local file,’ click ‘Browse,’ and select the file you wish to use from your directory.
» Subtext limited to

This section determines which elements to extract text from. Users can also specify the attribute name or name/value pair modifying the element.


Determines which XML elements to extract text from. Multiple tags must be separated by commas. This field cannot be left blank.

Attribute name
Use this field to specify an attribute name that modifies the element listed above. Only instances the element with this attribute will have text extracted from them.

Attribute value
Use this field to specify an attribute value modifying the element and attribute name listed above. The attribute value must be paired with an attribute name. Only instances the element with this attribute name/value pair modifying it will have text extracted from them.
» Token types

(use ^s for whitespace)


This section determines what to consider a token within the source document. Tokens are fragments of a text that can be defined as words, lines, sentences, paragraphs, characters or specific patterns.


Splits text on individual words.

Splits text line by line.

Splits text sentence by sentence.

Splits text paragraph by paragraph.

Splits text along a user-specified character (separated by spaces). To include a space use ^s.

Separate on tags
Splits text on a use-specified tag. Users may keep the tag (element) with the token by checking the box to the right of the text field. If this option is chosen, the results cannot be displayed as an XML tree. Note: The text can only be split by one element at the time.

keep element tags with token
Check this box to display the element with the token.

Splits text along a user-specified pattern, either in Unix format or as a regular expression.

Unix Format
Splits the text by tokens based on a Unix format regular expression

Regular Exp.
Splits the text by tokens found with a regular expression. Do not use \d, \D, \w, \W etc. Instead, please use [0-9], [^0-9], [a-zA-Z] or [^a-zA-Z]. Using \n, \r, or \t is fine.
» Results

This section allows users to choose how the results will be formatted, and whether to display it in a new browser window.


Display options
This drop menu allows users to treat the token one of four ways in the final results: strip out the separator, keep the separator as a token, keep the separator with the previous token, or keep the separator with the following token.

Display as
This drop-down lists enables users to choose from several output formats: HTML, XML text in HTML, XML tree, and Tab delimited text.

Open results in new window
Checking this box will display the results in a new window. This option is selected by default. Some pop-up blockers may prevent a new window from being opened; if so, un-check the box to open the results in the same window instead.
`*' indicates a required field
`' can not be displayed in "XML tree" format
`' do not use \d, \D, \w, \W etc. use [0-9], [^0-9], [a-zA-Z] or [^a-zA-Z] instead. Using \n, \r, \t is fine.



TAPoRware Project, McMaster University,