Showing posts with label number. Show all posts
Showing posts with label number. Show all posts

Friday, March 23, 2012

Fuzzy lookup match issue

Hello,

I have a peculiar problem in my project. My project design is like this

The number in (...) are count of records.

File feed (1000)

|

|

Fuzzy Lookup

against Table2

|

|

Split Fz Lookup results

(_Similarity >= 0.60 && _Confidence >= 0.85)

| |

| |

| Write matches to Table1 (250)

|

Fuzzy Group

Remaining rows (750)

|

|

Split Fz Group results

| |

| |

Write Canonicals Write Dupes

to Table2 to Table1

(300) (450)

This is basically a customer de-dupification project.

The Table2 has the canonicals and Table1 has the dupes (of the canonicals).

I already have some data in these tables and the new data is matched against the existing data

in these tables and classified as new customers and duplicate customers.

In the above process one could notice that the rows identified as dupes of already exsting canonicals

by the Fuzzy Lookup task are written into the dupes table (Table1) and will not be processed further down

the line in the project.

But in my case I see that those matches identified by Fuzzy lookup are further being included in the

Fuzzy Grouping also.

When I run this in debug mode in BIDS, it shows the correct numbers as I have depicted in the

illustration above. But, after execution, when I query the tables it shows that all 1000 rows

went through Fuzzy Grouping.

Any thoughts?

Btw, is there anyway to upload attachments to the postings here?

I also tried introducing a Derived Column between the 'Split Fz Lookup Results' and 'Write matches to Table1' to write some string into one of the table columns. It did not.

Monday, March 19, 2012

Function with unlimited parameters

Is there any way to write procedure with ulimited number of parameters?

Like in COALESCE function. You can pass one or more parameters.

The short answer would be no, it's not.
(there is a hard limit on # parameters, but if you ever get there, you're in deep trouble most likely)

Why would you need it? Don't you know beforehand what the procedure will do?
There is often a higher cost in reaching for the ultimate in generic, instead of specializing, which will do 'less' but with lower overhead and less test/development/maintenance time.

Keep it simple, and it will work forever =:o)

/Kenneth

|||

Nope, you can't even write a function like that where you can skip parameters. I have a date function that I need to be able to pass "unlimited" values to but I have only 16 works right now. Even worse you would have to default every parameter too.

dbo.function ('value1','value2',null,null,null,null,null,null,null,null,null,null,null,null,null,...)

As an alternative (yet more costly method) you could pass a comma delimited list as a string parameter and split it up into your N values. If that is a reasonable possibility, then you can look here for how to do this: http://www.sommarskog.se/arrays-in-sql.html. XML is another possibility. Both of these require work on your end to compile the string of course, so that might not be a great way to go either.

If that doesn't make sense, someone here can write you a query based on the data you have.

|||I believe you can create an extended stored procedure that can take an unlimited number of parameters.|||Maybe You know how?

Some example?|||

Extended stored procedures are marked for deprecation so don't plan on writing new code since you will have to convert again in couple of releases or so. What is the problem you are trying to solve? Why do you need to pass optional parameters? You could also use a temporary table for example to pass the parameter values as rows. So it depends on what functionality you want and why.

|||My question is purely teoretical. There is no some kind of problem I'am trying to solve with this solution.

Monday, March 12, 2012

Function to return week number

Does anyone now how I can create a SQL function to return a w
number for any date with the following guidlines?
-Ws begin on a Thursday and end on a Wednesday.
-1st January is always in w 1.
-W 1 for this year would go from 30/12/2004 to 05/01/2005
I have been going round in circles trying to crack this one. I would
be very grateful if anyone has any ideas.Mark
Use DateFirst and DatePart
an Example in T-SQL:
Set DateFirst 4
Declare @.D DateTime Set @.D = '20050101'
Select DatePart(wk, @.d)
To make a function:
-- **********************************
Create Functiondbo.WNumber
(@.D DateTime,
@.FDOW TinyInt) -- The day of w yo want to be first Mon = 1; Sunday= 7
Returns TinyInt
As
Begin
Declare @.WkNo TinyInt
Set @.WkNo = (DatePart(dy, @.d ) +
@.FDOW + 4) / 7
Return @.WkNo
End
-- ---
Use it like this:
Select dbo.WNumber('20050106',4)
"Mark Powell" wrote:

> Does anyone now how I can create a SQL function to return a w
> number for any date with the following guidlines?
> -Ws begin on a Thursday and end on a Wednesday.
> -1st January is always in w 1.
> -W 1 for this year would go from 30/12/2004 to 05/01/2005
> I have been going round in circles trying to crack this one. I would
> be very grateful if anyone has any ideas.
>|||The you tried the built-in DATEPART after setting appropriate SET DATEFIRST?
Tibor Karaszi, SQL Server MVP
http://www.karaszi.com/sqlserver/default.asp
http://www.solidqualitylearning.com/
http://www.sqlug.se/
"Mark Powell" <mark@.muddyboots.com> wrote in message
news:f10f176.0503300059.5537cf67@.posting.google.com...
> Does anyone now how I can create a SQL function to return a w
> number for any date with the following guidlines?
> -Ws begin on a Thursday and end on a Wednesday.
> -1st January is always in w 1.
> -W 1 for this year would go from 30/12/2004 to 05/01/2005
> I have been going round in circles trying to crack this one. I would
> be very grateful if anyone has any ideas.|||Thanks - that's much better than anything I could come up with. The
only problem with this function is that 30/12/04 and 31/12/04 are
returning w 53 and I want them to return w 1.|||That IS counterintuitive, that the last days of 2004 should be considered as
being in the first w of 2005, But if you want it that way, modify
function so that if value calculates to 53, it returns 1 instead
Alter Functiondbo.WNumber
(@.D DateTime,
@.FDOW TinyInt) -- The day of w yo want to be first Mon = 1; Sunday= 7
Returns TinyInt
As
Begin
Declare @.WkNo TinyInt
Set @.WkNo = (DatePart(dy, @.d ) +
@.FDOW + 4) / 7
Return Case @.WkNo When 53
Then 1 Else @.WkNo End
End
"mark@.muddyboots.com" wrote:

> Thanks - that's much better than anything I could come up with. The
> only problem with this function is that 30/12/04 and 31/12/04 are
> returning w 53 and I want them to return w 1.
>|||It doesn't make sense to me either, exept that it keeps the same w
number across the year end. It is how our client wants it, so it's the
way it needs to be done.
The next problem I have found is that the w number does not return
the correct value if I use a date from 2004 or 2006 (i.e. not 2005). I
imagine the +4 in your procedure should be a variable based on the
year, but I am not sure what it does?
Thanks for your help.
Mark|||Mark,
This was much harder than I thought at first, but this is it... Give it a
shot...
ALTER FUNCTION dbo.WNumber
(@.D DateTime, @.FDOW TinyInt)
Returns TinyInt
As
Begin
Declare @.FDOY Smallint, @.Shft Smallint, @.Yr SmallInt
Set @.Yr = Year(@.D)
Set @.FDOY = DatePart(dw, Str(@.Yr,4,0) + '0101') +
(@.@.DateFirst%7) - 1
Set @.Shft = (@.FDOY - @.FDOW + 7) % 7
Declare @.WkNo TinyInt
Set @.WkNo = (DatePart(dy, @.d) + @.Shft + 6) / 7
-- Now adjust for last partial w
Return Case When @.WkNo < 53 Then @.WkNo
When @.WkNo > 53 Or @.Shft < 5 Then 1
When @.Shft = 6 Then 53
-- Leap Year Consideration
When DatePart(dy, Str(@.Yr,4,0) + '1231') = 365
Then 1
Else 53 End
End
-- ****************************************
***************
And here is the code to test it...
Declare @.Y SmallInt Set @.Y = 2000
Set NoCOunt On
Declare @.D DateTime
Declare @.FDOW TinyInt Set @.FDOW = 4
Declare @.DP TinyInt,@.DF TinyInt,
@.DY SmallInt, @.I TInyInt
Declare @.DTs Table(DT TinyInt)
Set @.I = 0
While @.I < 7 Begin
Set @.I = @.I + 1
Insert @.DTs(DT) Values(@.I)
End
Print 'Year Day Date WNo Day Date WNo '
While @.Y < 2010 Begin
--Set @.D = STR(@.Y, 4,0) + '0101'
Set @.DF = @.@.DateFirst
--Set @.DP = datepart(wday, @.D)
--Set @.DY = datepart(dy, @.D)
Select @.Y,
Left(DateName(dw, Str(@.Y, 4,0) + '01' + Replace(Str(DT, 2,0),' ', '0')),2)
+
' ' + Str(@.Y, 4,0) +'01' + Left(Replace(Str(DT, 2,0),' ', '0'),8) +
' ' +
Cast(dbo.WNumber(Str(@.Y, 4,0) +'01' + Left(Replace(Str(DT, 2,0),'
', '0'),2), 4) as Char(2)),
Left(DateName(dw, Str(@.Y, 4,0) + '12' + Replace(Str(DT+24, 2,0),' ',
'0')),2) +
' ' + Str(@.Y, 4,0) +'12' + Left(Replace(Str(DT+24, 2,0),' ', '0'),8)
+ ' ' +
Cast(dbo.WNumber(Str(@.Y, 4,0) +'12' + Left(Replace(Str(DT+24,
2,0),' ', '0'),2), 4)as Char(2))
From @.DTs
Set @.Y = @.Y + 1
End
"mark@.muddyboots.com" wrote:

> It doesn't make sense to me either, exept that it keeps the same w
> number across the year end. It is how our client wants it, so it's the
> way it needs to be done.
> The next problem I have found is that the w number does not return
> the correct value if I use a date from 2004 or 2006 (i.e. not 2005). I
> imagine the +4 in your procedure should be a variable based on the
> year, but I am not sure what it does?
> Thanks for your help.
> Mark
>|||Sorry , error in test script...
Use the following to test UDF In Prev Post
-- ****************************************
********
Set NoCount On
Declare @.D DateTime
Declare @.FDOW TinyInt Set @.FDOW = 2
Declare @.Y SmallInt Set @.Y = 2000
-- --
Declare @.I TInyInt Set @.I = 0
Declare @.DTs Table(DT TinyInt)
While @.I < 7 Begin
Set @.I = @.I + 1
Insert @.DTs(DT) Values(@.I)
End
-- ---
Print 'Year Day Date WNo Day Date WNo '
While @.Y < 2010 Begin
Select @.Y,
Left(DateName(dw, Str(@.Y, 4,0) + '01' +
Replace(Str(DT, 2,0),' ', '0')),2) +
' ' + Str(@.Y, 4,0) +'01' +
Left(Replace(Str(DT, 2,0),' ', '0'),8) + ' ' +
Cast(dbo.WNumber(Str(@.Y, 4,0) +'01' +
Left(Replace(Str(DT, 2,0),' ', '0'),2), @.FDOW) as Char(2)),
Left(DateName(dw, Str(@.Y, 4,0) + '12' +
Replace(Str(DT+24, 2,0),' ', '0')),2) +
' ' + Str(@.Y, 4,0) +'12' +
Left(Replace(Str(DT+24, 2,0),' ', '0'),8) + ' ' +
Cast(dbo.WNumber(Str(@.Y, 4,0) +'12' +
Left(Replace(Str(DT+24, 2,0),' ', '0'),2), @.FDOW)as Char(2))
From @.DTs
Set @.Y = @.Y + 1
End
-- ****************************************
********
"mark@.muddyboots.com" wrote:

> It doesn't make sense to me either, exept that it keeps the same w
> number across the year end. It is how our client wants it, so it's the
> way it needs to be done.
> The next problem I have found is that the w number does not return
> the correct value if I use a date from 2004 or 2006 (i.e. not 2005). I
> imagine the +4 in your procedure should be a variable based on the
> year, but I am not sure what it does?
> Thanks for your help.
> Mark
>