MaxMo Search Development

2006-06-02T18:28:00.000-07:00

As promised, there is now support for Microsoft Excel and Microsoft PowerPoint documents. And this list will continue to grow.

How are non-HTML files indexed?
In order to best index non-HTML files, MaxMo Search will first call on a conversion tool which will output an HTML version of the document. This allows for original formatting to influence the indexing process, ensuring more accurate results. In the absense of an HTML converter, MaxMo Search can also use text conversion tools as well.

New File Types Supported

2006-06-01T19:18:00.000-07:00

The crawler will now process Microsoft Word and WordPerfect documents, allowing them to be fully indexed and accessible through the search software. This complements our existing support for Adobe PDFs and RTF (Rich Text Format). Soon to be added: Microsoft Excel and Microsoft Powerpoint.

Testing the Limits

2006-02-01T23:38:00.000-08:00

We recently ran a test of the search software across thirty domains, to test the abilities and limits of the latest interface version. Although not all pages across the domains were crawled, the search software crawled and indexed approximently 4,500 pages. The initial crawl speed was extremely good downloading about nine pages a second. This speed remained for the first two thousand pages, but began to go downhill with later pages (crawling still retained its speed, but the index rate was down to about one page every one to two seconds). The main decrease in speed was related to the indexers attempt to avoid duplicate content, and as the number of pages in the index increases, this becomes a big task.

Statistics for the data digged up by the crawl:
Pages: 4,550
Lexicon: 37,344 words
Inverted Index: 876,135 entries
Links: 122,631

This part of the data took up 135MB in the MySQL database the crawler/indexer work with. After converting it to the MaxMo Search data format, which is optimized for fast searching, the data only took up 56MB (interesting that the faster database format created for MaxMo Search, which far exceeds the speed of MySQL, is also smaller). Search speeds were usually 0.1-0.3 seconds, but for super common words (ie, appeared on two thousdand pages), the search time exceeded one second.

Results of this test

Indexing must be improved to better detect duplicate pages, or at least no longer worry about duplicate pages (maybe worry about them in later processing). Also, not really a problem in the test, but a long needed improvement is better ability to track http redirects. Although they are properly handled now, the redirect page can end up being crawled multiple times.

Ranking must be easier to tweak, and also calculate in IDF (Inverse Document Frequency) scores to reward rare (and most likely more descriptive, or mispelled) words.

Results need to be sped up for super common words, by ignoring results after, say, the 1,000th page.

Crawling and indexing system will be revamped to put less stress on servers being crawled, and to allow for more continous crawling.

In addition, the overall system should also eventually work with simply the MaxMo Search data format. After testing has shown it to be consistently and signifigantly faster then MySQL (simply due to its very narrowly designed parameters), it seems only logical to take MySQL completely out of the picture. In addition, doing so will help remove the timely step of converting the MySQL data into MaxMo Search data.

Version 4

2006-01-30T00:58:00.000-08:00

A new version of the interface has changed the layout of files to allow for much faster processing. In the new version of the interface, databases for seperate sites are kept distinct, no longer sharing certain tables (ie, the lexicon and such). This improvement allows for easier creation and management of sites and their configuration data, with faster database access (no longer having to include query conditions to seperate sites).

MaxMo Technologies can be searched using the new interface. If you look at the URL, you will notice the URL management system has changed. Features for managing sites with the search software have been removed, as it is a waste of resources. The search software should only have to handle queries, not serve pages as well. Apache is better equipped for site management.

Note: Interface refers to the way all the information is stored and organized, as well as accessed (ie, URL format, database layout, files, etc). Essentially, all the programming other then the crawling and parsing algorithms. The version of the actual crawler/indexer and the interface are tracked independently.

New Blog

2006-01-30T00:49:00.000-08:00

This new blog will trace the development of the MaxMo Search product line, showing search statistics and linking to new samples we create. Note that this is a side product, and therefore will develop slowly.