Showing posts with label null. Show all posts
Showing posts with label null. Show all posts

Friday, March 23, 2012

Fuzzy Lookups and Groupings Algorithm

Hello,
I'm trying to clean my data using fuzzy lookup algorithm though SSIS, but i get null values everywhere. This is what i did:

I applied the fuzzy lookup in a table (tblValues). As source table i have the tblValues, and as reference table in Fuzzy Lookup i have the tblValues as well, resulting null values in all fields/columns.

Do i have to create my own reference table? If yes, how do i do that and what values will i have in this table?I didn't understand how the reference table must be in order the algorithm to work. Any suggestions?

Thank you in advance!

I'm not sure I understand what your objective is. If you are trying to remove duplicates in tblValues, you're better off using a fuzzy grouping task and not a fuzzy lookup task. See the following article for how to use both if you haven't read it yet.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnsql90/html/FzDTSSQL05.asp

|||

Thanks for the article i have already read it and i used it as reference to my project. My objective is to fill the null values with the most possible value.

I'm not sure if fuzzy does that, but in that articles says that "Fuzzy Lookup matches input records that are "dirty" (because of misspellings, truncations, missing or inserted tokens, null fields, unexpected abbreviations, and other irregularities) with clean records in a reference table. ", but i didn't understand how the reference table must be.

Wednesday, March 21, 2012

Fuzzy grouping matching nulls to empty strings/spaces

Will the fuzzy grouping task match a null value to an empty string (or spaces)? I've got 5 columns I'm matching on, and one of them may be null for certain rows but an empty string for others. Given the 4 other columns may match, will this difference stop similar columns being grouped together?

(Someone's modified my grouped data since it was deduped, which takes a while, and I'm hoping for a quick answer on this).

Thanks in advance.

Ben

Not sure, but building a quick package and adding a data viewer should give you the answer.

I would hope that a NULL wouldn't match to spaces...

Couldn't you run an update query to "fix" the data?|||

Computer says "no"

<cough>

No duplicate matching between null and empty string/spaces, as you suspected. Thanks for the idea of the quick test - seems obvious in retrospect, if only it had been beforehand Smile Unfortunately I can't fix the data, other than rerunning the grouping task, but good to know the source of the problem - thanks Phil!