Archive

Archive for the ‘BI’ Category

T-SQL Delete Join

May 22nd, 2009 NothingMan 1 comment

Here’s are two quick ways to Delete data in one table based on another table.  This works perfectly in an ETL job where you have a staging table that indicate whether to Insert, Update, or Delete the data in the other table.

The first one uses an inner join, the second uses a simple EXISTS clause.

DELETE b

  FROM [dbo].[TABLE_1] b

 INNER JOIN [dbo].[TABLE_2] w

    ON b.[ROW_ID] = w.[ROW_ID]

 WHERE w.[CTRL_CD] = 'D'

 

 

DELETE b

  FROM [dbo].[TABLE_1] b

 WHERE EXISTS (

            SELECT 1

          FROM [dbo].[TABLE_2] w

         WHERE b.[ROW_ID] = w.[ROW_ID]

           AND w.[CTRL_CD] = 'D')

 

VN:F [1.9.6_1107]

Rating: 4.8/5 (5 votes cast)
VN:F [1.9.6_1107]
Rating: +1 (from 1 vote)
Categories: BI, Code, Tips Tags: , , ,

Google Geocoding in SSIS

May 20th, 2009 NothingMan 4 comments

I have an old YouTube tutorial on how to use Google Geocoding within SSIS, but I lost all the downloadable scripts.  I recently had a few people asking me if I could send them the scripts, so I deiceded to rewrite them.

Again, you can check out the YouTub video (it’s rough) for more details, just go to http://www.youtube.com/watch?v=HTSHzR-wSgc with this DISCLAIMER:  This code was just thrown together and is quite ugly and my microphone was terrible, I usually sound much better than that Google Geocoding in SSIS  :-)

In a nutshell, I sniped about 18,000 golfcourses off the web with a PHP script and got basic course information for all of them, including the address information.  I wanted to use Google Maps to display these courses, so I thought I’d use Google Geocoding.  Caution, they don’t like it when you hit them 1000 times per second, so I would recommend using a Sleep command in your code like I did so you can throttle it without getting kicked off. 

So, I have an OLE Source task that reads from my database.  I created a Derived Column task to create placeholders for latitude, longitude, and level of accuracy.  I created a Script Component and selected street, city, state, and zip as input columns and added lat, long, and accuracy as read/write columns. 

Open the script editor, add a reference to System.Web, then create a new class called Google GeoCode, or whatever the heck you want to call it and paste in this code:

Option Strict Off

''' Might need this, might not

Imports System

Imports System.Data

Imports System.Math

Imports System.Net

Imports System.Web

Imports System.IO

Namespace GoogleGeoCoder

    Public Interface ISpatialCoordinate

        Property latitude() As Double

        Property longitude() As Double

    End Interface

    ''' <summary>   

    ''' Coordiate structure. Holds Latitude and Longitude.   

    ''' </summary>   

    Public Structure Coordinate

        Implements ISpatialCoordinate

 

        Private _latitude As Double

        Private _longitude As Double

 

        Public Sub New(ByVal lattitude As Double, ByVal longitude As Double)

            _latitude = lattitude

            _longitude = longitude

        End Sub

 

        Public Property latitude() As Double Implements ISpatialCoordinate.latitude

            Get

                Return _latitude

            End Get

            Set(ByVal value As Double)

                _latitude = value

            End Set

        End Property

 

        Public Property longitude() As Double Implements ISpatialCoordinate.longitude

            Get

                Return _longitude

            End Get

            Set(ByVal value As Double)

                _longitude = value

            End Set

        End Property

 

    End Structure

    Public Class GeoCode

        Const _googleUri As String = "http://maps.google.com/maps/geo?q="

        Const _googleKey As String = "yourgooglekeyhere"  ' get your own       

        Const _outputType As String = "csv"

        Private Shared Function GetGeoCodeUri(ByVal address As String) As Uri

            address = HttpUtility.UrlEncode(address)

            Return New Uri(String.Format("{0}{1}&output={2}&key={3}", _googleUri, address, _outputType, _googleKey))

        End Function

        ''' <summary>       

        ''' Gets a Coordinate from a address.       

        ''' </summary>       

        ''' <param name="address">An address.       

        ''' <remarks>       

        ''' <example>1600 Amphitheatre Parkway Mountain View, CA 94043</example>       

        ''' </remarks>       

        ''' </param>       

        ''' <returns>A spatial coordinate that contains the latitude and longitude of the address.</returns>       

        Public Shared Function GetCoordinates(ByVal address As String) As Coordinate

            Dim client As WebClient = New WebClient()

            Dim uri As Uri = GetGeoCodeUri(address)

            Dim geoCodeInfo As String()

            'The first number is the status code,            

            'the second is the accuracy,            

            'the third is the latitude,            

            'the fourth one is the longitude.           

            Try

                geoCodeInfo = client.DownloadString(uri).Split(",")

                Return New Coordinate(Convert.ToDouble(geoCodeInfo(2)), Convert.ToDouble(geoCodeInfo(3)))

            Catch ex As Exception

                Return New Coordinate(0.0, 0.0)

            End Try

        End Function

    End Class

End Namespace

 

Save it and then go back to the main script and add Import that class (see code).  Notice, the way I’m getting the level of accuracy with my ugly nested IF statements isn’t the prettiest way to do this, use your imagination.  Copy in this code into the Main Script:

 

' Microsoft SQL Server Integration Services user script component

' This is your new script component in Microsoft Visual Basic .NET

' ScriptMain is the entrypoint class for script components

 

Imports System

Imports System.Data

Imports System.Math

Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper

Imports Microsoft.SqlServer.Dts.Runtime.Wrapper

Imports ScriptComponent_4a8732cababc49a999159a91c737082a.GoogleGeoCoder

 

Public Class ScriptMain

    Inherits UserComponent

 

    Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)

 

        Dim coordinate As Coordinate = GeoCode.GetCoordinates(Row.street & ", " & Row.city & ", " & Row.state & ", " & Row.zip)

 

        If ((coordinate.latitude <> 0) And (coordinate.longitude <> 0)) Then

            Row.lat = Convert.ToDecimal(coordinate.latitude)

            Row.long = Convert.ToDecimal(coordinate.longitude)

            Row.acc = 1

        Else

            coordinate = GeoCode.GetCoordinates(Row.city & ", " & Row.state & ", " & Row.zip)

            If ((coordinate.latitude <> 0) And (coordinate.longitude <> 0)) Then

                Row.lat = Convert.ToDecimal(coordinate.latitude)

                Row.long = Convert.ToDecimal(coordinate.longitude)

                Row.acc = 2

            Else

                coordinate = GeoCode.GetCoordinates(Row.zip)

                If ((coordinate.latitude <> 0) And (coordinate.longitude <> 0)) Then

                    Row.lat = Convert.ToDecimal(coordinate.latitude)

                    Row.long = Convert.ToDecimal(coordinate.longitude)

                    Row.acc = 3

                Else

                    coordinate = GeoCode.GetCoordinates(Row.city & ", " & Row.state)

                    If ((coordinate.latitude <> 0) And (coordinate.longitude <> 0)) Then

                        Row.lat = Convert.ToDecimal(coordinate.latitude)

                        Row.long = Convert.ToDecimal(coordinate.longitude)

                        Row.acc = 4

                    Else

                        coordinate = GeoCode.GetCoordinates(Row.state)

                        If ((coordinate.latitude <> 0) And (coordinate.longitude <> 0)) Then

                            Row.lat = Convert.ToDecimal(coordinate.latitude)

                            Row.long = Convert.ToDecimal(coordinate.longitude)

                            Row.acc = 5

                        End If

                    End If

                End If

            End If

        End If

    End Sub

 

End Class

 

VN:F [1.9.6_1107]

Rating: 5.0/5 (3 votes cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)
Categories: BI, Code, Misc, g33k Tags: , ,

RowNum() Equivalent in SQL 2000+

May 18th, 2009 NothingMan No comments

Oracle has a pseudo-column called ROWNUM that allows you to display the row number for each row.  There are infinite reasons why you might want to use the ROWNUM, but in SQL it’s not nearly as easy to determine.

 

In Oracle, it’s simply

SELECT ROWNUM from table_name

 

In SQL 2005+, there is actually a ROW_NUMBER() function, but you have to give it some more information to use it.

ROW_NUMBER ( )     OVER ( [ <partition_by_clause> ] <order_by_clause> )

 

For example:
use [Database]
select row_number() over (order by data_srce_id desc) rownum

      ,data_srce_id
from data_srce 

 

rownum

data_srce_id

1

8

2

7

3

6

4

5

5

4

6

3

7

2

8

1

9

-1



 

However, in SQL 2000 it’s a little more difficult.  The only way I’ve found is using the IDENTITY() function, which requires an INTO statement, so you pretty much have to load the results into a #temporary table and then select from the #temporary table like this:

use [Database]

SELECT IDENTITY(int, 1,1) id

      ,t1.[column1]           col1

      ,t1.[column2]           col2

      ,t1.[column3]           col3

      ,t2.[column2]           col4

  INTO #Temp

  FROM [dbo].[Table] t1

 INNER JOIN [dbo].[Table2] t2

    ON t.[column] = t2.[column]

 

select *

  from #Temp

VN:F [1.9.6_1107]

Rating: 0.0/5 (0 votes cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)
Categories: BI, Code, Tips Tags: , , , ,

Sending Mail with MSSQL

May 15th, 2009 NothingMan 4 comments

Here’s a quick and dirty way to send email via MSSQL.  For some reason, we’re not allowed to send email using the Email Task in SSIS, so I chose this option.  Unfortunately, even if you have access to execute sp_SQLSMTPMail, you might not have access to run xp_cmdshell.

 

I ended up having to create my SSIS package with a SQL Task that runs this proc, then I had to run it with a SQL Agent Job with a proxy account that had access to xp_cmdshell.  Fun stuff.

 

 

 

 

declare @developerperson varchar(80)

declare @message_body varchar(1000)

 

set @message_body =

‘Dear ‘ + @developerperson + ‘Use \n for line breaks and escape your ”s’

 

 

EXECUTE dbo.sp_SQLSMTPMail

      @vcTo = ‘You@you.com’,

      @vcCC= ‘Them@them.com’,

      @vcSubject = ‘Super Cool SQL Email’,

      @vcBody = @message_body,

      @vcFrom = ‘Me@me.com’,

      @vcSMTPServer = ‘SMTPName’,

      @vcSenderName = ‘Some One’,

      @vcServerName = @@SERVERNAME –no need to specify because this is the default anyway

Or, lets say you want to email query results as an attachment:

EXECUTE dbo.sp_SQLSMTPMail

  @vcTo=‘you@you.com’

, @vcCC=‘them@them.com’

, @vcSubject=‘Here is your wonderful report that is terribly formatted as a txt file attachment’

, @vcquery= ‘select col1, col2, col3, col4 from database.schema.table’

 

 

 

 

 

 

 

 

Here’s the actual Proc definition: Read more…

VN:F [1.9.6_1107]

Rating: 0.0/5 (0 votes cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)
Categories: BI, Code, Tips Tags: , , , , , , ,

Delete TOP n Records in T-SQL

May 13th, 2009 NothingMan No comments

If you’re trying to delete the first n records in a table, here’s how to do it in SQL 2005: Read more…

VN:F [1.9.6_1107]

Rating: 5.0/5 (1 vote cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)
Categories: BI, Code, Tips Tags: , ,

SQL WAITFOR Command with WHILE Loop

May 8th, 2009 NothingMan 2 comments

I recently needed a way to execute a script overnight to count the number of records that were being updated in a given hour.  I needed it to run at a specific time and to execute in 10 minute incriments and stop .  I was hoping to log the results to a file, but this does just as well.  When it’s done executing, it spits it all to the output window so I can see it in the morning when I get back.

 

This script below will start at 11pm and end at midnight and it will execute every 10 minutes and display the current time and the count.  You could just as easily use a counter and have it execute a certain number of  times.

 

–don’t even start until 11pm

WAITFOR TIME ’23:00:00.000′

declare @endtime datetime

set @endtime = cast(’2009-05-09 00:00:00.000′ as datetime)

–stop at midnight

while getdate() < @endtime

begin

      –every 10 minutes, execute

      WAITFOR DELAY ’00:10:00′;

      select getdate(), count(*)

      from tablename

      where datetimecolumn > ’5/7/2009′

      and datetimecolumn < ’5/8/2009′

end

 

 

Kudos to Jerry for helping me with this.

VN:F [1.9.6_1107]

Rating: 5.0/5 (1 vote cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)
Categories: BI, Code, Tips Tags: , , , , ,

Instr() Equivalent in SQL Server

May 8th, 2009 NothingMan 6 comments

While delving into the wonderful (*ahem) world of SQL server, I was attempting to parse through some text fields and needed to find an alternative to the Oracle instr() function.

The syntax for the INSTR Oracle function is:


instr( string1, string2 [, start_position [, nth_appearance ] ] )

string1 is the string to search.

string2 is the substring to search for in string1.

start_position is the position in string1 where the search will start. This argument is optional. If omitted, it defaults to 1. The first position in the string is 1. If the start_position is negative, the function counts back start_position number of characters from the end of string1 and then searches towards the beginning of string1.

nth_appearance is the nth appearance of string2. This is optional. If omitted, it defaults to 1.

The syntax for CHARINDEX in T-SQL is


CHARINDEX ( expression1 ,expression2 [ , start_location ] )

expression1 Is a character expression that contains the sequence to be found. expression1 is limited to 8000 characters.

expression2 Is a character expression to be searched.

start_location Is an integer or bigint expression at which the search starts. If start_location is not specified, is a negative number, or is 0, the search starts at the beginning of expression2.

VN:F [1.9.6_1107]

Rating: 4.0/5 (2 votes cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)
Categories: BI, Code, Tips Tags: , , , , ,

Getting Table/Column Metadata in SQL Server

May 7th, 2009 NothingMan No comments

First of all, when I’m trying to find metada in Oracle, I am usually using Toad, and have no problem finding what I need because it’s such a great too. However, even in SQL-Plus, it’s still really easy to find what I need by querying the data dictionary:


select *
from all_columns
where owner = user
and table_name = 'tablename'

… Or even easier, if it’s in the schema I’m already in:

select *
from user_columns
where table_name = 'tablename'

It seems that it’s still pretty easy in SQL Server to do this, but off the bat, it’s not exactly as intuitive. Apparently you can go through sysobjects, but even Microsoft suggests against it, considering they reserve the right to change any and all system tables from version to version.

There are alternatives though; views in INFORMATION_SCHEMA:
For instance, to view tables:

select *
from INFORMATION_SCHEMA.TABLES

… And for columns:

select *
from INFORMATION_SCHEMA.Columns
where table_schema = 'schemaname'
and table_name = 'tablename'

Here’s some more great information about INFORMATION SCHEMA: Here

VN:F [1.9.6_1107]

Rating: 2.0/5 (1 vote cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)
Categories: BI, Tips Tags: , , , ,

Script to Delete a SQL Agent Job

May 6th, 2009 NothingMan No comments

I have been in under the impression that you could not drop SQL Agent Jobs without being able to look at the properties and being able to get the JOB ID, since you need the jobid to pass to the sp_delete_job function.  Because of that, whenever I wanted to change a Job, I always had to incude a manual step to log onto the server, right click and delete the job.

However, I found a simple script to automate this, so now I can just include this in my script that will recreate the job.  Simple… should have known.
USE [msdb]
GO
DECLARE
@jobid sysname;
IF
EXISTS (SELECT job_id FROM msdb.dbo.sysjobs_view WHERE name = N'EXACT_JOB_NAME')
BEGIN
SELECT
@jobid = job_id FROM msdb.dbo.sysjobs_view WHERE name = N'EXACT_JOB_NAME'
EXEC  msdb.dbo.sp_delete_job @job_id=@jobid
END 

VN:F [1.9.6_1107]

Rating: 4.0/5 (2 votes cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)
Categories: BI, Code, Tips Tags: , , , ,

Sending Email via Script Task in SSIS

May 5th, 2009 NothingMan No comments

The other day I was trying to figure out how to send an email via Script Task in SSIS. For some reason, the Send Email Task wasn’t working for me (still can’t figure out why). Anyway, here’s what I ended up with.

Works like a charm…


' Microsoft SQL Server Integration Services Script Task
' Write scripts using Microsoft Visual Basic
' The ScriptMain class is the entry point of the Script Task.
Imports System
Imports Microsoft.SqlServer.Dts.Runtime
Imports System.Net.Mail
Imports System.Net

Public Class ScriptMain
    Public Sub Main()
      Dim myHtmlMessage As MailMessage
      Dim mySmtpClient As SmtpClient

      myHtmlMessage = New MailMessage(“email@somewhere.com”, “email@somewhereelse.com”, “Subject”, “Body”)
      mySmtpClient = New SmtpClient(“SMTP”)
      mySmtpClient.Credentials = CredentialCache.DefaultNetworkCredentials
      mySmtpClient.Send(myHtmlMessage)
      Dts.TaskResult = Dts.Results.Success
    End Sub
End Class

VN:F [1.9.6_1107]

Rating: 3.3/5 (3 votes cast)
VN:F [1.9.6_1107]
Rating: 0 (from 2 votes)
Categories: BI, Code Tags: , , , , ,