The first module that touches the input documents or queries is the parser. You will need to modify our implementation if your documents need special handling and you can simply describe their elements by a regular expression (as shown below).

The parser in Egothor tries to reduce the total number of active words, attempting to keep valid tokens of text as one word. For instance, it is better for an IR system to store a date like "10-Jul-2002" as one word (term, token) rather than decomposing it to its three constituent elements: "10", "Jul" and "2002". Using the second method entails the implementation of a complicated algorithm that recognizes that the three independent words are in reality just the constituent parts of the date as a whole. Remember, when parsing, less is definitely more!

The table below outlines which text elements are recognized (the actual list can be found in org.egothor.parser.plain.Simple):

Where: L is a letter, D is a digit, S is a string (as defined above), W is a word (as defined above), H is a host name (as defined above), () groups elements, + means "at least one", * means "may appear several times", ? means "it may appear", | means "or".

As you can see, the grammar does not validate, for instance, the date format. It can be done by filters which go in action after the initial parser. The filters can parse already parsed elements to smaller pieces. They can discard or copy (clone) elements, and do many other interesting things.