If so, what server power did you require to get v.fast search results(less than 3secs), whether 7000 docs were found or 100?
(Production data rather than archived data that is static - so the indexer task is also having to add new docs continuously and serve results)
The documents in the database contain a subset of fields from the original document (4 fields + Body). This is to reduce the database / index size and include documents from other databases in the search.
Maybe its the Client power rather than the server power, as the delay could be building the results view on the client.
I notice that limiting the number of results significantly reduces the delay.
Subject: Full Text Index Performance - 15GB data - 1.3M docs
Tommy,
The time to return the results to the client is related to the number of hits. There are notes.ini parameters which limit the number of hits. Unless you’re using an agent to programmatically get a large collection, you presumably only need 100 or so. Any ftsearch that gets more than that is just a bad search and needs to be fine tuned.
I know I’m being a bit extreme, but even if you limit the results to 250, which is fairly normal for a limit, that’s well below your 7000.
Subject: RE: Full Text Index Performance - 15GB data - 1.3M docs
Hi Raphael,
The 7000docs was an example, in theory the person could return all docs in database if the text is in all docs, as a search should do.
Obviously this would cause a delay, but I would expect a product designed for enterprise use could handle the amount of messages a group of 10people process within a 2year period.
I ran another search that returned 3631 in 15secs
When limiting to 100 hits it still took 7secs.
Unfortunately I would be fine with this as I am not a user. But if you run 100 searches in a day it would become tedious, as we know that at times the results will be delayed further by server processed or by other apps running on the client.
Should I only expect slow searches within Domino, or can this be overcome by using bigger hardware which is not a problem ?
I find it strange that they have Domain Search built in, but search results cannot be returned on more than 1 small team’s document collection.
Subject: RE: Full Text Index Performance - 15GB data - 1.3M docs
Paul,
I see two issues here:
Slow ftsearch performance. I have not run tests against your db size, not to mention configuration, to be able to say that 2 seconds or 20 seconds is expected. If you have any smaller db’s that are ftindexed and you can do a few quick tests there, you can get a sense of whether the delay is caused by the size of the ftindex in this db or the size of the returned set or just general server performance issues. So do a handful of searches and jot down the time and the number of hits. Do this in a few db’s and jot down the number of docs in the db’s and the size of their ftindexes. This will give you a kind of chart that can indicate what the most relevant factors are.
Even slower performance for large sets of returned docs. You are getting the most relevant n documents returned, when you limit to n. I honestly cannot imagine any scenario where having n greater than, say, 100 or 200 makes sense, although I’m sure there is such a scenario out there somewhere.
Programmatically, of course you need to be able to get all results so that an agent can delete, modify, whatever. But for users? Who scrolls down to the 500th hit in a search?
Subject: Full Text Index Performance - may relate to Domain Index implementation
1,5M document is a lot of documents but depending on your query, it could be very fast. One of my client was indexing 1.3m documents with 30000 updates a week. It was pretty fast.
But in your scenario, you have 40g of data…It’s a big difference. Indexing large db is a cause of bad server performance.
Subject: RE: Full Text Index Performance - may relate to Domain Index implementation
Cheers for your response.
I have edited the original post, as I am using a product called CSearch now, which reduces the size of the original data by taking a subset of fields from the original doc.
So the CSearch DB is now 15GB, currently 450000, docs but once the new DB is added 1.3M docs.
When returning 7000 docs in a search it is slow. Freezes for ages when returning 80,000 docs which I could expect.
I need to find out where the delay is. Is it the search on the server, or returning the results to a view for the user interface? This may determine if I should increase RAM / CPU on the server or on the Client PC?
Unfortunately, my users want results within 3secs max.
If I can understand where I need to increase the performance, I could see if this satisfies the users. At the moment, the search results do not appear fast enough if returning more than 500 docs.
To be fair on the system, once I know where the limitations are on performance, I can propose search methods such as reducing the max results returned. But at this time some people may require the large number of results.