Traversing a folder looking for files to import into a database table is a common practice using integration services. What if you have little control over the import folder or files because of security restrictions? What if there are many files stored in the folder and number of files grows daily? One way to deal with this is using a Foreach Loop Container, call to a database table or log and determine if the file had previously been imported. This may be a sufficient solution if the number of files is small, however if the number of files is larger this could have a big impact on database performance and/or package performance. A better approach may be to use a Script Task. In this blog post I will show how we can use a Script Task with some simple C# code to capture a list of files, remove processed files, and use a Foreach Loop Container to traverse the new files.
File Import Table
Before we get into creating the Integration Services package, we will need a FileImport table to keep a list of all files that have already been imported.
CREATE TABLE dbo.FileImport( FileImportID INT IDENTITY(1, 1) NOT NULL , FileName NVARCHAR(1000) NOT NULL , ImportDate DATETIME NULL , RecordCount INT NULL , ErrorMessage NVARCHAR(MAX) NULL ) ON [PRIMARY]; |
New File Import SSIS package
The New File Import SSIS package will query the FileImport table to capture a list of previously imported files as well as capturing a list of available files in the import folder. Using some code we will eliminate files that have already been imported and then loop over this new set of files that have not been imported.
Let’s start by creating a new SSIS Package and named it NewFileImport. We will use an OLE DB connection to query the FileImport table and a few package variables to store processing data. The following package variables will be created for the associated purposes.
Variable Definitions
- The FileName variable will be used store each of the new files as we traverse the new files using a Foreach Loop Container.
- The ImportedFiles variable will be used to store the previously imported files that is received from FileImport Table
- The ImportedPath variable is the path where the files to be imported are stored.
- The NewFiles variable is a list of files that have not been imported into the system yet.
Package Tasks
The package will use a SQL Task, Script Task and Foreach Loop Container.
SQL – Get a list of all processed files
This Task will capture a list of all the files that have been previously imported files and will store them into the ImportedFiles variable. The ResultSet should be set to “Full result set” and a simple query to the FileImport table “SELECT FileName FROM dbo.FileImport;” should be saved in the SQLStatement.
The Result Set should be saved to the User::ImportedFiles variable.
SCR – Get a list of files to Import
This task will create a data set and array list object that will store all of the file names in the FileImport Table and all the file names in the Import Path. A loop will be used to traverse the files names in FileImport object. On each pass through the loop the corresponding file name in the Import Path object will be removed. When the loop is complete a list of only new file names will be remaining in the Import Path object. This Import Path object will be stored in the NewFiles variable and used in the Foreach Loop Container. The following C# script implements this logic.
public void Main(){ OleDbDataAdapter da = new OleDbDataAdapter(); DataTable dt = new DataTable(); object ImportedFiles = Dts.Variables["ImportedFiles"].Value; string ImportedPath = Dts.Variables["ImportedPath"].Value.ToString(); // Load the processed files from the record set into a DataTable variable. da.Fill(dt, ImportedFiles); // Load the files to be imported from the directory to be imported into an // ArrayList variable. ArrayList ImportDataFiles = new ArrayList(Directory.GetFiles(ImportedPath)); // Traverse the DataTable and remove any matching files from the ArrayList. foreach (DataRow dr in dt.Rows) { ImportDataFiles.Remove(dr[0].ToString()); } // Return the ArrayList which will only have new files. Dts.Variables["NewFiles"].Value = ImportDataFiles; Dts.TaskResult = (int)ScriptResults.Success; } |
The ImportedFiels, ImportPath and NewFiles are used in the script, so we will need to declare their use in the script.
FELC – Loop through all files
This Foreach Loop Container will loop through all of the files that are stored in the NewFiles variable. The enumerator type that will be used is the “Foreach From Variable Enumerator”. The selected Variable should be User::NewFiles.
The Variable Mapping should store the result of each loop into the User::FileName variable.
On each iteration of the Foreach Loop Container, the full path to the file, from the NewFiles variable will be stored into the FileName variable. Having the path to the file will help in the file import process inside the Data Flow Task. The details of importing file data can be found in this Simple SSIS: Importing Data from Flat Text Files blog post.
