Egothor comes with its own web crawler, code named Capek.

Figure 2.1.  Karel Capek, Czech playwright

Karel Capek (1890-1938) was a Czech playwright early in the last century who wrote, most notably, Rossum's Universal Robots, which introduced the word robot into the English language.

The crawler uses "rules" contained in the etc/rules file in your Egothor directory. You'll have to edit this file to establish some boundaries for Capek to operate within. Below is what the rules file that ships with Egothor looks like:

# http://a.a/b/v/d.html?a=a
# {A }   {B}{    C    } {D}
# {          E            }

# this is default
must                    
# {A} allow http
protocol ^http$           
# {B} our local domain
domain	.shark$           
# {E}
whole	http://www.       
# {C}
uripath	^/                
# {D}
query	=                 

# what is prohibited
not                     
# antihackers
domain	^127.             
domain	^localhost
# antiapache dirlists
# {D}
query	^C=D&O=A$	  
query	^C=D&O=D$
query	^C=S&O=A$
query	^C=S&O=D$
query	^C=N&O=A$
query	^C=N&O=D$
query	^C=M&O=A$
query	^C=M&O=D$
    

	The file is divided into two parts: must and not. The must block defines what must be in the URL. The not block, conversely, defines what cannot be in the URL
	URL's that follow the HTTP protocol are allowed. Tip ^ means begin string and $ means end string.
	The host's name ends with ".shark"...
	...and the path starts with a slash.
	We also allow a query string which contains "=" (thus ...?abc is not allowed, but ...?id=3455 is).
	Moreover, we disallow a URL if its hostname starts with "127." or "localhost".
	The query string must not contain any of the eight strings which are generated by apache httpd server for directory listings (?C=D&O=A etc.)

# http://a.a/b/v/d.html?a=a
# {A }   {B}{    C    } {D}
# {          E            }

# this is default
must
# {A} only http is acceptable
protocol ^http$
# {B} only this site will be scanned
domain	^www.egothor.org$
# {C} this is not required, but why not...
uripath	^/
# {D} we will accept URLs with query strings
query	=

# what is prohibited
not
# antiapache dirlists
# {D}
query	^C=D&O=A$
query	^C=D&O=D$
query	^C=S&O=A$
query	^C=S&O=D$
query	^C=N&O=A$
query	^C=N&O=D$
query	^C=M&O=A$
query	^C=M&O=D$
			

Since we know exactly where the robot will be going, the prohibited list only excludes Apache httpd directory listing stuff. In practice, you'll probably want to crawl more than just one website, so your prohibited list will probably need to resemble the default list rather than this one.

Starting Capek. 

To start Capek working for you (and no it won't write any famous works of literature, at least not yet ;-) ), just issue the command:

java org.egothor.robot.Capek [URLs of websites to crawl]

Additionally, there are a number of options that you can specify when starting up Capek. They are specified by appending the -D flag to the java command above.

Stopping Capek. 

Stopping Capek can occur in one of two ways:

Capek Options. 

Capek can be run in daemon mode (meaning that it will run forever without an explicit stop call) by specifying it as an optional parameter on the command line.

stat
Threads 10
Max idle time 60000
Last job at 1068448877389
Started at 1068448864654
Tasks 12
Rate (t/s) 0.9397760200485551
Manager cache miss-rate 0.0
Corpus in ./corpus/
Cache url->id miss-rate 1.0
Cache NodeOfSH miss-rate 0.07692308
Link DB in ./linkdb/db.txt
Records 0 links 0
Scheduler in ./scheduler/
Ready servers 1
Root level 2 items planned 2

list
*capek#1 1068448881680 15 http://www0.it-dept.centralbank:80/manual/dso.html
*capek#1 1068448880579 14 http://www0.it-dept.centralbank:80/manual/programs/
*capek#1 1068448879529 13 http://www0.it-dept.centralbank:80/manual/mod/index.html
*capek#1 1068448878459 12 http://www0.it-dept.centralbank:80/manual/mod/index-bytype.html
*capek#1 1068448877389 11 http://www0.it-dept.centralbank:80/manual/stopping.html
 capek#1 1068448876329 36 10 http://www0.it-dept.centralbank:80/manual/invoking.html
 capek#1 1068448875259 64 9 http://www0.it-dept.centralbank:80/manual/install.html
 capek#1 1068448874229 19 8 http://www0.it-dept.centralbank:80/manual/LICENSE
 capek#1 1068448873189 37 7 http://www0.it-dept.centralbank:80/manual/upgrading_to_1_3.html
 capek#1 1068448872059 112 6 http://www0.it-dept.centralbank:80/manual/new_features_1_3.html
 capek#1 1068448871009 49 5 http://www0.it-dept.centralbank:80/manual/mod/
 capek#1 1068448869909 83 4 http://www0.it-dept.centralbank:80/manual/mod/directives.html
 capek#1 1068448868809 59 3 http://www0.it-dept.centralbank:80/manual/sitemap.html
 capek#1 1068448867609 140 2 http://www0.it-dept.centralbank:80/manual/mod/mod_ssl/
 capek#1 1068448865949 657 1 http://www0.it-dept.centralbank:80/manual/index.html
 capek#1 1068448864772 172 0 http://www0.it-dept.centralbank:80/

threads
 0 pop
 1 save: http://www0.it-dept.centralbank:80/manual/dso.html
 2 http://www0.it-dept.centralbank:80/manual/netware.html
 3 wait
 4 pop
 5 pop
 6 pop
 7 pop
 8 pop
 9 pop

Once you've used Capek to crawl the pages you want, you'll want to index them using the Michelangelo indexer.

Figure 2.2.  Michelangelo Buonarroti, Italian Renaissance artist

Capek saves the pages it gathers (compressed using gzip) while crawling in a file marked /corpus/. Capek also creates two other directories while it's at it: /linkdb/ and /scheduler/. The linkdb directory will contain just one file, db.txt, which contains the structure of hypertext links found in the documents. The scheduler directory contains a data structure which is used by a scheduler in the robot.

Starting Michelangelo. 

Issue the command:

	Note
	If you omit the parameter, the directories will be created in the current directory.

Michelangelo will create an index in, of all places, an /index/ directory inside the directory specified on the command line, or the current one if one isn't specified.

Your new index is ready for searching!

Now let's say that you want to compile an index with linksrank. This just a bit more involved than just using Michelangelo, but should not prove to be too difficult for even a novice Egothor user. You'll have to make use of a series of applications to form the eventual index. They should be called in this order:

java org.egothor.apps.Compile linkdb/db.txt linkdb/db.xxx
java org.egothor.oracle.LinksFileReader linkdb/db.xxx ranks/
java org.egothor.apps.Michelangelo
java org.egothor.apps.Oracul index/ -linksrank ranks/

Figure 2.3. Processing of links

Here is what happens with each (also see Figure 2.3):

	Note
	Michelangelo and Oracul cannot run concurrently as they both depend on and alter the index.