We figured out that FTSearches does not always return all possible documents. It occurs when you use the wildcard operators * and/or ? in a fielded query. But only when the word with the wildcard results in more than 1000 possible combinations. Like the SQL statement:
SELECT Table.Field FROM Table WHERE Table.Field Like ‘Phrase’ GROUP BY Table.Field;
If this would return more than 1000 the problem exists. We are not sure if the the result is grouped. From our testing we would say yes. So the combinations 1001 to n are not found with FTSearch.
This leads to really strange effects. Let’s say you have two documents where the field TF contains the values
Doc A: TF = “Data0000 TESTDOC”
Doc B: TF = “Data1000 TESTDOC”
then you have also 2000 documents where the TF value is
Doc 1: TF = “Data0000”
Doc 2: TF = “Data0001”
Doc 3: TF = “Data0002”
…
Doc 2000: TF = “Data1999”
If you search with the query
query = "FIELD TF CONTAINS (Data* AND TESTDOC)
you should get 2 documents returned but you just get 1. You just find doc A. The reason is that the query part “Data*” has more than 1000 combinations and the combination “Data1000” of Doc B is after the limit of 1000.
We were doing hundreds of tests. This effects all R6.x versions.
Following the message from IBM:
Full Text and View searches fail to find some documents when the search criteria uses the FIELD operator and Wildcard (* or ?).
The solution is to update our underlying GTR engine. This is risky to do in an MR as it may introduce regressions.
Historically upgrading the GTR engine has exposed us to regressions.
We recommend an interim solution of views that key off required fields, until a new GTR engine is put in a later feature release.
We hope that we can solve this problem in Domino Notes version 7.x