Archive

Posts Tagged ‘ssis’

Generate ROWNUM pseudo-column in SSIS

May 27th, 2009 NothingMan No comments

I posted earlier on how to generate a rownumber in SQL 2000, but that would not work as a source query in SSIS for some reason.  Without having to create a stored procedure or temp tables based on other temp tables, I wanted a simple way to generate this rownum in SSIS.  I was considering something with variable expressions, but then I found a way to do it with a Script Component.

Drag a Script Component Task into your DataFlow and connect it between your source and destination tasks.  Open it up and click “Inputs and Outputs”.  Double Click “Output 0″ and highlight “Output Columns” and click Add Column.  For this code to work, name that new column “rownum“.

Then open the script tab and click “Design Script” and paste this code over what is already there:

  

Imports System

Imports System.Data

Imports System.Math

Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper

Imports Microsoft.SqlServer.Dts.Runtime.Wrapper

 

Public Class ScriptMain

    Inherits UserComponent

    Dim counter As Integer = 0

 

    Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)

 

        Row.rownum= counter

        counter = counter + 1

    End Sub 

End Class

 

VN:F [1.9.6_1107]

Rating: 5.0/5 (1 vote cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)
Categories: BI, Code, Tips Tags: , , ,

Google Geocoding in SSIS

May 20th, 2009 NothingMan 4 comments

I have an old YouTube tutorial on how to use Google Geocoding within SSIS, but I lost all the downloadable scripts.  I recently had a few people asking me if I could send them the scripts, so I deiceded to rewrite them.

Again, you can check out the YouTub video (it’s rough) for more details, just go to http://www.youtube.com/watch?v=HTSHzR-wSgc with this DISCLAIMER:  This code was just thrown together and is quite ugly and my microphone was terrible, I usually sound much better than that Google Geocoding in SSIS  :-)

In a nutshell, I sniped about 18,000 golfcourses off the web with a PHP script and got basic course information for all of them, including the address information.  I wanted to use Google Maps to display these courses, so I thought I’d use Google Geocoding.  Caution, they don’t like it when you hit them 1000 times per second, so I would recommend using a Sleep command in your code like I did so you can throttle it without getting kicked off. 

So, I have an OLE Source task that reads from my database.  I created a Derived Column task to create placeholders for latitude, longitude, and level of accuracy.  I created a Script Component and selected street, city, state, and zip as input columns and added lat, long, and accuracy as read/write columns. 

Open the script editor, add a reference to System.Web, then create a new class called Google GeoCode, or whatever the heck you want to call it and paste in this code:

Option Strict Off

''' Might need this, might not

Imports System

Imports System.Data

Imports System.Math

Imports System.Net

Imports System.Web

Imports System.IO

Namespace GoogleGeoCoder

    Public Interface ISpatialCoordinate

        Property latitude() As Double

        Property longitude() As Double

    End Interface

    ''' <summary>   

    ''' Coordiate structure. Holds Latitude and Longitude.   

    ''' </summary>   

    Public Structure Coordinate

        Implements ISpatialCoordinate

 

        Private _latitude As Double

        Private _longitude As Double

 

        Public Sub New(ByVal lattitude As Double, ByVal longitude As Double)

            _latitude = lattitude

            _longitude = longitude

        End Sub

 

        Public Property latitude() As Double Implements ISpatialCoordinate.latitude

            Get

                Return _latitude

            End Get

            Set(ByVal value As Double)

                _latitude = value

            End Set

        End Property

 

        Public Property longitude() As Double Implements ISpatialCoordinate.longitude

            Get

                Return _longitude

            End Get

            Set(ByVal value As Double)

                _longitude = value

            End Set

        End Property

 

    End Structure

    Public Class GeoCode

        Const _googleUri As String = "http://maps.google.com/maps/geo?q="

        Const _googleKey As String = "yourgooglekeyhere"  ' get your own       

        Const _outputType As String = "csv"

        Private Shared Function GetGeoCodeUri(ByVal address As String) As Uri

            address = HttpUtility.UrlEncode(address)

            Return New Uri(String.Format("{0}{1}&output={2}&key={3}", _googleUri, address, _outputType, _googleKey))

        End Function

        ''' <summary>       

        ''' Gets a Coordinate from a address.       

        ''' </summary>       

        ''' <param name="address">An address.       

        ''' <remarks>       

        ''' <example>1600 Amphitheatre Parkway Mountain View, CA 94043</example>       

        ''' </remarks>       

        ''' </param>       

        ''' <returns>A spatial coordinate that contains the latitude and longitude of the address.</returns>       

        Public Shared Function GetCoordinates(ByVal address As String) As Coordinate

            Dim client As WebClient = New WebClient()

            Dim uri As Uri = GetGeoCodeUri(address)

            Dim geoCodeInfo As String()

            'The first number is the status code,            

            'the second is the accuracy,            

            'the third is the latitude,            

            'the fourth one is the longitude.           

            Try

                geoCodeInfo = client.DownloadString(uri).Split(",")

                Return New Coordinate(Convert.ToDouble(geoCodeInfo(2)), Convert.ToDouble(geoCodeInfo(3)))

            Catch ex As Exception

                Return New Coordinate(0.0, 0.0)

            End Try

        End Function

    End Class

End Namespace

 

Save it and then go back to the main script and add Import that class (see code).  Notice, the way I’m getting the level of accuracy with my ugly nested IF statements isn’t the prettiest way to do this, use your imagination.  Copy in this code into the Main Script:

 

' Microsoft SQL Server Integration Services user script component

' This is your new script component in Microsoft Visual Basic .NET

' ScriptMain is the entrypoint class for script components

 

Imports System

Imports System.Data

Imports System.Math

Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper

Imports Microsoft.SqlServer.Dts.Runtime.Wrapper

Imports ScriptComponent_4a8732cababc49a999159a91c737082a.GoogleGeoCoder

 

Public Class ScriptMain

    Inherits UserComponent

 

    Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)

 

        Dim coordinate As Coordinate = GeoCode.GetCoordinates(Row.street & ", " & Row.city & ", " & Row.state & ", " & Row.zip)

 

        If ((coordinate.latitude <> 0) And (coordinate.longitude <> 0)) Then

            Row.lat = Convert.ToDecimal(coordinate.latitude)

            Row.long = Convert.ToDecimal(coordinate.longitude)

            Row.acc = 1

        Else

            coordinate = GeoCode.GetCoordinates(Row.city & ", " & Row.state & ", " & Row.zip)

            If ((coordinate.latitude <> 0) And (coordinate.longitude <> 0)) Then

                Row.lat = Convert.ToDecimal(coordinate.latitude)

                Row.long = Convert.ToDecimal(coordinate.longitude)

                Row.acc = 2

            Else

                coordinate = GeoCode.GetCoordinates(Row.zip)

                If ((coordinate.latitude <> 0) And (coordinate.longitude <> 0)) Then

                    Row.lat = Convert.ToDecimal(coordinate.latitude)

                    Row.long = Convert.ToDecimal(coordinate.longitude)

                    Row.acc = 3

                Else

                    coordinate = GeoCode.GetCoordinates(Row.city & ", " & Row.state)

                    If ((coordinate.latitude <> 0) And (coordinate.longitude <> 0)) Then

                        Row.lat = Convert.ToDecimal(coordinate.latitude)

                        Row.long = Convert.ToDecimal(coordinate.longitude)

                        Row.acc = 4

                    Else

                        coordinate = GeoCode.GetCoordinates(Row.state)

                        If ((coordinate.latitude <> 0) And (coordinate.longitude <> 0)) Then

                            Row.lat = Convert.ToDecimal(coordinate.latitude)

                            Row.long = Convert.ToDecimal(coordinate.longitude)

                            Row.acc = 5

                        End If

                    End If

                End If

            End If

        End If

    End Sub

 

End Class

 

VN:F [1.9.6_1107]

Rating: 5.0/5 (3 votes cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)
Categories: BI, Code, Misc, g33k Tags: , ,

Sending Mail with MSSQL

May 15th, 2009 NothingMan 4 comments

Here’s a quick and dirty way to send email via MSSQL.  For some reason, we’re not allowed to send email using the Email Task in SSIS, so I chose this option.  Unfortunately, even if you have access to execute sp_SQLSMTPMail, you might not have access to run xp_cmdshell.

 

I ended up having to create my SSIS package with a SQL Task that runs this proc, then I had to run it with a SQL Agent Job with a proxy account that had access to xp_cmdshell.  Fun stuff.

 

 

 

 

declare @developerperson varchar(80)

declare @message_body varchar(1000)

 

set @message_body =

‘Dear ‘ + @developerperson + ‘Use \n for line breaks and escape your ”s’

 

 

EXECUTE dbo.sp_SQLSMTPMail

      @vcTo = ‘You@you.com’,

      @vcCC= ‘Them@them.com’,

      @vcSubject = ‘Super Cool SQL Email’,

      @vcBody = @message_body,

      @vcFrom = ‘Me@me.com’,

      @vcSMTPServer = ‘SMTPName’,

      @vcSenderName = ‘Some One’,

      @vcServerName = @@SERVERNAME –no need to specify because this is the default anyway

Or, lets say you want to email query results as an attachment:

EXECUTE dbo.sp_SQLSMTPMail

  @vcTo=‘you@you.com’

, @vcCC=‘them@them.com’

, @vcSubject=‘Here is your wonderful report that is terribly formatted as a txt file attachment’

, @vcquery= ‘select col1, col2, col3, col4 from database.schema.table’

 

 

 

 

 

 

 

 

Here’s the actual Proc definition: Read more…

VN:F [1.9.6_1107]

Rating: 0.0/5 (0 votes cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)
Categories: BI, Code, Tips Tags: , , , , , , ,

Putting VB.Net (2008) to sleep

May 6th, 2009 NothingMan No comments

While working on my geo-coding experiment, I realized that the Google web service stops allowing requests after a certain amount of submittals in a certain amount of time. I’d get about 30 successful results and then about 10-15 unsuccessful results.

The easiest way I could solve this problem was to send n requests and then sleep for a few seconds, and then continue.

So, this is how you put the program to sleep:

Add this code:
Declare Sub Sleep Lib "kernel32" (ByVal dwMilliseconds As Long)

And call it like this:
Sleep(2000)

VN:F [1.9.6_1107]

Rating: 3.0/5 (1 vote cast)
VN:F [1.9.6_1107]
Rating: +1 (from 1 vote)
Categories: Code, Tips Tags: , , , , ,

Sending Email via Script Task in SSIS

May 5th, 2009 NothingMan No comments

The other day I was trying to figure out how to send an email via Script Task in SSIS. For some reason, the Send Email Task wasn’t working for me (still can’t figure out why). Anyway, here’s what I ended up with.

Works like a charm…


' Microsoft SQL Server Integration Services Script Task
' Write scripts using Microsoft Visual Basic
' The ScriptMain class is the entry point of the Script Task.
Imports System
Imports Microsoft.SqlServer.Dts.Runtime
Imports System.Net.Mail
Imports System.Net

Public Class ScriptMain
    Public Sub Main()
      Dim myHtmlMessage As MailMessage
      Dim mySmtpClient As SmtpClient

      myHtmlMessage = New MailMessage(“email@somewhere.com”, “email@somewhereelse.com”, “Subject”, “Body”)
      mySmtpClient = New SmtpClient(“SMTP”)
      mySmtpClient.Credentials = CredentialCache.DefaultNetworkCredentials
      mySmtpClient.Send(myHtmlMessage)
      Dts.TaskResult = Dts.Results.Success
    End Sub
End Class

VN:F [1.9.6_1107]

Rating: 3.3/5 (3 votes cast)
VN:F [1.9.6_1107]
Rating: 0 (from 2 votes)
Categories: BI, Code Tags: , , , , ,