Talend: Incremental Load using Talend ETL tool

Consider you want to perform an incremental load between Source and Target table. Let assume you have both source and target table in MS SQL database. Perform the below steps for incremental loading:

1. Create a reference table for incremental load in target database (e.g., dbo.Incrementalload with columns TableName and Incrementaldate1).  Your table will look like this:

---------------------------------------------------------------
TableName   |  Incrementaldate1 | Incrementaldate2
----------------------------------------------------------------
Event            | 2000-01-01           | NULL
----------------------------------------------------------------
User             | 2010-01-01            | NULL
----------------------------------------------------------------


2. Design your Talend job as like below:


3. Create a context variable 'Incrementaldate1'

4. In tMSSQLInput_2: Get incremental date to assign to context variable.


 5. tJavaRow_1: Assign incremental date to context variable.



6. tMSSQLInput_3: Retrive data from source.



In WHERE condition call the context variable.

7. tMSSQLOutput_2: Load data to target table.

8. tMSSQLRow_2 : Update the Incrementaldate1 in IncrementalLoad table with Talend Getdate()



So in next load, only the data with modified date greater than incremental date will be retrieved and loaded.