|
||||||||||
PREV NEXT | FRAMES NO FRAMES |
Packages that use Token | |
---|---|
org.egothor.cache | Provides classes that help implementing a cache for the egothor project. |
org.egothor.core | This package concentrates the core data objects and interfaces. |
org.egothor.core.memory | This package contains an implementation of barrels in memory. |
org.egothor.core.query | This package contains objects that represent a structure of a query in our inner and binary form, readers of a barrel and a result queue. |
org.egothor.dir | This package defines objects of distributed IR layer. |
org.egothor.duplicity.algorithm | This package contains top-level classes that implement the duplicity checking algorithm. |
org.egothor.duplicity.visualization | This package contains classes implementing the visualization of the duplicities found in a document by the duplicity checking algorithm. |
org.egothor.html | This package contains a specific implementation of core objects for HTML with "home" and "content" support. |
org.egothor.parser | |
org.egothor.parser.filter | This package defines objects that filter tokens. |
org.egothor.parser.plain | This package defines JavaCC parser for a plain text. |
org.egothor.query | This package contains objects that represent a structure of a query in our inner form. |
org.egothor.query.parser | This package defines JavaCC parser of user's query. |
org.egothor.query.runner | This package defines a machinery that navigates the rider during query execution in the Vector model. |
org.egothor.query.runner.enhanced | This package defines a machinery that navigates the rider during query execution in any model. |
org.egothor.text | This package contains support of some textual routines/processes. |
Uses of Token in org.egothor.cache |
---|
Methods in org.egothor.cache that return Token | |
---|---|
Token[] |
CachedQuery.getTokens()
Gets the tokens that are forming the query |
Constructors in org.egothor.cache with parameters of type Token | |
---|---|
CachedQuery(Query query,
Token[] tokens,
int offset,
int length,
long max_hits2scan,
double pg_rerank)
Constructor for the CachedQuery object. |
Uses of Token in org.egothor.core |
---|
Fields in org.egothor.core with type parameters of type Token | |
---|---|
Sequence<Token> |
Filter.prev
The Tokenizer used by Filter. |
Methods in org.egothor.core that return Token | |
---|---|
Token |
Filter.action(Token t)
Used for changing tokens of the input tokenizer. |
Token |
Token.newText(java.lang.String name,
java.lang.String text)
Clones this objects with a new name and text. |
Token |
Filter.next()
The next token of input tokenizer is modified by Filter.action(org.egothor.core.Token) and the product is also the product of
this method. |
Token[] |
QueryResponse.queryTokens()
Return an array of all tokens recognized in a query. |
Methods in org.egothor.core that return types with arguments of type Token | |
---|---|
Sequence<Token> |
Filter.getPrevTokenizer()
Return the tokenizer this filter reads from. |
Sequence<Token> |
DocumentData.words(boolean readlinx,
boolean readilinx,
boolean lowercase,
boolean phonetics,
HTMLField.Diacritics diacritics,
boolean paragraphs,
boolean paragraphsKeepPunctuation,
java.lang.String encoding)
|
Methods in org.egothor.core with parameters of type Token | |
---|---|
Token |
Filter.action(Token t)
Used for changing tokens of the input tokenizer. |
Method parameters in org.egothor.core with type arguments of type Token | |
---|---|
void |
Filter.setPrevTokenizer(Sequence<Token> prev)
Set the tokenizer this filter reads from. |
Constructors in org.egothor.core with parameters of type Token | |
---|---|
QueryResponse(int offset,
long wouldBe,
long positives,
Sequence<Hit> e,
Token[] queryTokens,
int enum_len,
Query adaptedQuery)
Constructor for the QueryResponse object. |
Constructor parameters in org.egothor.core with type arguments of type Token | |
---|---|
Filter(Sequence<Token> prev)
Constructor for the Filter object. |
Uses of Token in org.egothor.core.memory |
---|
Methods in org.egothor.core.memory that return types with arguments of type Token | |
---|---|
Sequence<Token> |
FTField.filteredWords()
Return an enumeration of the terms in the field filtered by filters. |
Sequence<Token> |
FTField.words()
Return an enumeration of the terms in the field. |
Uses of Token in org.egothor.core.query |
---|
Method parameters in org.egothor.core.query with type arguments of type Token | |
---|---|
void |
Query.addTerms(java.util.HashSet<Token> to)
Add the given HashSet of terms to this Query. |
Uses of Token in org.egothor.dir |
---|
Methods in org.egothor.dir with parameters of type Token | |
---|---|
abstract DocumentData |
Group.expandDocMetadata(DocumentData ofBarrel,
Token[] interest)
Retrives the document data, but restrict the data block to the part relevant to some tokens of our interest. |
abstract DocumentData |
Tanker.expandDocMetadata(DocumentData ofBarrel,
Token[] interest)
|
DocumentData |
TankerImpl.expandDocMetadata(DocumentData ofBarrel,
Token[] interest)
|
DocumentData |
TankerImplSecure.expandDocMetadata(DocumentData ofBarrel,
Token[] interest)
Deprecated. |
DocumentData |
TankerImplSecure.expandDocMetadataSecure(DocumentData ofBarrel,
Token[] interest)
Standard exapnding of doc metadata, but using multithreaded safe way. |
Method parameters in org.egothor.dir with type arguments of type Token | |
---|---|
CWI |
Group.getCWI(java.util.HashSet<Token> terms)
Return the subset of CWI for the given subset of terms. |
CWI |
TankerImplSecure.getCWI(java.util.HashSet<Token> terms)
|
Uses of Token in org.egothor.duplicity.algorithm |
---|
Method parameters in org.egothor.duplicity.algorithm with type arguments of type Token | |
---|---|
void |
PermutatedMinsFiller.computeDocumentMins(DocumentPermutatedMins result,
Sequence<Token> terms,
long documentUID,
int documentDBRevision)
Computes the permutated mins values for given sequence of tokens of a document and fills it into the result under the identificator documentID. |
Uses of Token in org.egothor.duplicity.visualization |
---|
Methods in org.egothor.duplicity.visualization that return types with arguments of type Token | |
---|---|
static java.util.List<java.util.List<Token>> |
DocumentDuplicities.getDocumentUnits(Sequence<Token> words)
Takes the sequence of document words and depending on the Constants.CHECK_DUPLICITY_LEVEL splits it
to the appropriate text units - documents, paragraphs or sentences. |
Method parameters in org.egothor.duplicity.visualization with type arguments of type Token | |
---|---|
static java.util.List<java.util.List<Token>> |
DocumentDuplicities.getDocumentUnits(Sequence<Token> words)
Takes the sequence of document words and depending on the Constants.CHECK_DUPLICITY_LEVEL splits it
to the appropriate text units - documents, paragraphs or sentences. |
Uses of Token in org.egothor.html |
---|
Methods in org.egothor.html that return types with arguments of type Token | |
---|---|
Sequence<Token> |
HTMLField.words()
|
Method parameters in org.egothor.html with type arguments of type Token | |
---|---|
void |
HTMLField.setAppendix(Sequence<Token> appendix)
|
Uses of Token in org.egothor.parser |
---|
Methods in org.egothor.parser that return Token | |
---|---|
Token |
Strings2Tokens.next()
|
Uses of Token in org.egothor.parser.filter |
---|
Methods in org.egothor.parser.filter that return Token | |
---|---|
Token |
DupWithoutDiacritics.action(Token t)
If the name/type of the token is <WORD> then
transform the text of the token to lower case. |
Token |
LowerCase.action(Token t)
If the name/type of the token is not <EMAIL/PUNCT/NUM> then
transform the text of the token to lower case. |
Token |
Stemmer.action(Token t)
A simple stemming algorithm which works as follows: |
Token |
DupWithoutDiacritics.next()
Return the next token. |
Token |
Grammer.next()
Return the next token. |
Token |
ParagraphFilter.next()
Return the next token. |
Token |
ParagraphPunctFilter.next()
Return the next token. |
Token |
Phonetics.next()
Return the next token. |
Token |
PunctFilter.next()
Return the next token. |
Token |
RemoveDiacritics.next()
If the name/type of the token is <WORD> then
transform the text of the token to lower case. |
Token |
StopFilter.next()
Return the next token. |
Token |
WordNGrammer.next()
Return the next token. |
Methods in org.egothor.parser.filter with parameters of type Token | |
---|---|
Token |
DupWithoutDiacritics.action(Token t)
If the name/type of the token is <WORD> then
transform the text of the token to lower case. |
Token |
LowerCase.action(Token t)
If the name/type of the token is not <EMAIL/PUNCT/NUM> then
transform the text of the token to lower case. |
Token |
Stemmer.action(Token t)
A simple stemming algorithm which works as follows: |
static boolean |
ParagraphPunctFilter.isParagraphDelimiter(Token t)
Test whether a token is a paragraph delimiter. |
static boolean |
ParagraphPunctFilter.isPunctuation(Token t)
Test whether a token is a punctuation (mark). |
boolean |
PunctFilter.isPunctuation(Token t)
Test whether a token is a punctuation (mark) or it can be ignored. |
abstract boolean |
StopFilter.isStoppedToken(Token t)
Test whether a token should be processed or ignored. |
Constructor parameters in org.egothor.parser.filter with type arguments of type Token | |
---|---|
DupWithoutDiacritics(Sequence<Token> prev)
Constructor for the Diacritics object. |
|
Grammer(Sequence<Token> arg0)
Constructor for the Grammer object. |
|
LowerCase(Sequence<Token> prev)
Constructor for the LowerCase object. |
|
LowerCase(Sequence<Token> prev,
java.util.Locale locale)
Construct a LowerCase object using the given localization setting. |
|
ParagraphFilter(Sequence<Token> prev)
Constructor for the ParagraphPunctFilter object |
|
ParagraphPunctFilter(Sequence<Token> prev)
Constructor for the ParagraphPunctFilter object |
|
Phonetics(Sequence<Token> arg0)
Constructor for the Phonetics object. |
|
PunctFilter(Sequence<Token> arg0)
Constructor for the PunctFilter object |
|
RemoveDiacritics(Sequence<Token> prev)
Constructor for the Diacritics object. |
|
Stemmer(Sequence<Token> prev,
Trie stemmer)
Construct a Stem object using the given stemmer table. |
|
StopFilter(Sequence<Token> arg0)
Constructor for the StopFilter object. |
|
WordNGrammer(Sequence<Token> prev)
Constructor for the WordNGrammer object |
Uses of Token in org.egothor.parser.plain |
---|
Methods in org.egothor.parser.plain that return Token | |
---|---|
Token |
Plain.next()
Return the next Token. |
Token |
Simple.next()
Returns the next token in the stream, or null at EOS. |
Uses of Token in org.egothor.query |
---|
Methods in org.egothor.query with parameters of type Token | |
---|---|
static int |
Configuration.defaultBoost(Token tok)
Description of the Method |
static boolean |
Configuration.isControlToken(Token tok)
Is this a control token which is not excluded when it has a low idf? |
Method parameters in org.egothor.query with type arguments of type Token | |
---|---|
void |
QAnd.addTerms(java.util.HashSet<Token> to)
Add the given HashSet of terms to this Query. |
void |
QGroup.addTerms(java.util.HashSet<Token> to)
Add the given HashSet of terms. |
void |
QNot.addTerms(java.util.HashSet<Token> to)
Add the given HashSet of terms to this Query. |
void |
QOr.addTerms(java.util.HashSet<Token> to)
Add the given HashSet of terms to this Query. |
void |
QPhrase.addTerms(java.util.HashSet<Token> to)
Add the given HashSet of terms to this Query. |
void |
QProx.addTerms(java.util.HashSet<Token> to)
Add the given HashSet of terms to this Query. |
void |
QTerm.addTerms(java.util.HashSet<Token> to)
Add the given HashSet of terms to this Query. |
Constructors in org.egothor.query with parameters of type Token | |
---|---|
QTerm(Token token,
java.lang.String field)
Constructor for the QTerm object. |
|
QTerm(Token token,
java.lang.String field,
double idf,
boolean req,
boolean proh,
int boost,
int lowerBound,
int upperBound)
Constructor for the QTerm object. |
Uses of Token in org.egothor.query.parser |
---|
Methods in org.egothor.query.parser that return Token | |
---|---|
Token |
Parser.next()
Returns the next token in the stream, or null at EOS. |
Uses of Token in org.egothor.query.runner |
---|
Constructors in org.egothor.query.runner with parameters of type Token | |
---|---|
TermRunner(double idf,
Rider r,
java.lang.String field,
Token token,
int boost,
boolean req,
boolean proh)
Constructor for the TermRunner object |
|
TermRunner(IListWasher washer,
double idf,
Rider r,
java.lang.String field,
Token token,
int boost,
boolean req,
boolean proh)
Constructor for the TermRunner object. |
Uses of Token in org.egothor.query.runner.enhanced |
---|
Constructors in org.egothor.query.runner.enhanced with parameters of type Token | |
---|---|
TermRunner(int model,
double idf,
Rider r,
java.lang.String field,
Token token,
int boost,
boolean req,
boolean proh)
Constructor for the TermRunner object |
|
TermRunner(int model,
IListWasher washer,
double idf,
Rider r,
java.lang.String field,
Token token,
int boost,
boolean req,
boolean proh)
Constructor for the TermRunner object. |
Uses of Token in org.egothor.text |
---|
Methods in org.egothor.text that return Token | |
---|---|
Token |
Generator.next()
|
Methods in org.egothor.text with parameters of type Token | |
---|---|
abstract boolean |
Snipper.matches(Token w)
Description of the Method |
boolean |
SnipperOfStrings.matches(Token w)
Description of the Method |
boolean |
SnipperOfTokens.matches(Token w)
Description of the Method |
Method parameters in org.egothor.text with type arguments of type Token | |
---|---|
java.lang.String |
Snipper.filter(Sequence<Token> tokens)
Description of the Method |
Constructors in org.egothor.text with parameters of type Token | |
---|---|
SnipperOfTokens(Token[] word,
boolean htmlAware,
java.lang.String startHit,
java.lang.String endHit)
Constructor for the Snipper object |
|
||||||||||
PREV NEXT | FRAMES NO FRAMES |