Fuzzy lookup transform row scores 'inconsistent' with individual column scores

I am trying to interpret some of the results I observe when trying to match similar records using a fuzzy lookup transform, but it's not entirely clear how the overall row similarity score is calculated. In particular, sometimes rows with lower individual column similarity scores will achieve a higher similarity and confidence score than a matching row with higher individual column scores.

The transform is configured with 6 text fields set to fuzzy mapping and a minimum similarity of 0, and 3 additional numeric fields with an exact mapping. It is set to return a maximum of 2 matches per lookup and to do an exhaustive search of the reference table.

For example, from the following matching pair of records Match 1 is picked over Match 2 even though it's individual scores are lower.

Match 1 Match 2
-- --
_similarity_author 1.0 1.0
_similarity_title 0.85344648 1.0
_similarity_headline 0.0125 0.0125
_similarity_summary 0.0125 0.0125
_similarity_picture 1.0 1.0
_similarity_caption 1.0 1.0

_similarity 7.8429267E-2 7.3196657E-2
_confidence 0.55728668 0.44271332

In another case both matching records have *identical* scores for every mapped column and yet their similarity and confidence scores are different.

Clearly there are other factors involved in calculating the overall row score. Anybody know what these are?

Fernando Tubio

Can't even begin to describe it in my own words. This article describes the Fuzzy Math real well. Don't know if you've seen it.

http://msdn.microsoft.com/msdnmag/issues/05/09/SQLServer2005/

|||

Thank you Martin.

I've read the article and it explains the lookup process well. Unfortunately it doesn't answer my question. Specifically, having found two matches, why does the matching algorithm discard what appears to be a better match, at least judging from individual column similarity scores.

I am trying to understand the mechanism to determine if there is anything I can tweak in order to force the algorithm to make a better choice.

Full-Text search indexing HTML content

I've been quite busy these days looking up which was the best way to create
a full-text search catalog for a field composed entirely of HTML content, a
nd I found out that the best way to handle it, so it could ignore HTML tags
as the search was performed, was to make the column an Image-type field, and
associate a file type in a separate column.
I developed a small application that read each of the text field entries, co
nverted it to a byte[] variable using the UnicodeEncoding class (and it's Ge
tString and GetBytes methods) and saved the resulting byte array as a binary
file in the new image field. I even tested how the field data would fare in
a physical file, by using a FileStream to write the Byte[] data on several
different records.
So far, so good. It all seemed to be working. The problem is that after I bu
ild up the full-text catalog (using the wizard, and specifying the file type
field related to the "image" field where the HTML file is stored), I get no
resulting records whenever I perform a query using either CONTAINS or FREET
EXT. Such queries worked just fine when the field was a text field instead o
f an image field, so I'm guessing something went wrong either on the data co
nversion, or on the catalog building process.
Anyone has any ideas?On Wed, 23 Nov 2005 16:45:54 -0600, bacusgod wrote:
(snip)
>Anyone has any ideas?
Hi bacusgod,
You might want to try reposting this in the group where the full text
experts hang out: microsoft.public.sqlserver.fulltext
Best, Hugo
--
(Remove _NO_ and _SPAM_ to get my e-mail address)