|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object org.egothor.core.Filter org.egothor.parser.filter.WordNGrammer
public class WordNGrammer
This class produces N-grams of words.
From a sentence with W words w1, w2, w3,... wW it produce W - N + 1 N-grams.
The N is determined by the Constants.WORDNGRAMS_LENGHT
constant.
Filter treats all types of input tokens equally.
Output tokens:
ParagraphPunctFilter
shall be applied to the token sequence before this filter. If it is not, than
all the document is taken as one sentence and N-grams are produced "on document level".
This filter should be the last filter to apply.
The filter has an inner context thus it cannot be shared in a filtering chain.
Field Summary |
---|
Fields inherited from class org.egothor.core.Filter |
---|
prev |
Constructor Summary | |
---|---|
WordNGrammer(Sequence<Token> prev)
Constructor for the WordNGrammer object |
Method Summary | |
---|---|
Token |
next()
Return the next token. |
Methods inherited from class org.egothor.core.Filter |
---|
action, getPrevTokenizer, setPrevTokenizer |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public WordNGrammer(Sequence<Token> prev)
prev
- tokenizer used by the filterMethod Detail |
---|
public Token next()
next
in interface Sequence<Token>
next
in class Filter
Filter.next()
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |