This page is a place-holder for collecting best practices and sharing examples related to data migration using Talend Open Studio.
It is still in DRAFT state, which means that the information contained here does not necessarily represent a recommended approach for data migration.
Creating and managing jobs
This page will not (unless it's requested) provide a step-by-step approach about how to create a job, insert Talend components and create connections between them (hint: right click on a module and select Row). Talend has many resources to help get started, including the User Guide , the Reference Guide and the Talend community.
Note: Because it's basically a Java code generator, there's as many ways to migrate data in Talend as there are ways to write the code by hand to achieve the same. Shown below is just one of these approaches.
Approach A : Using CollectionSpace JAXB libraries (CS v1.3)
The main benefit of this approach is that the xml formatting is handled by the existing libraries, which saves manually duplicating and maintaining all the tag names. The main drawback is that some Java coding is required to map the fields.
1. Job Overview
The following job contains an simple example of creating two CollectionSpace Person records in the appropriate XML format. The job (which is attached here) is shown below.
2. Input from the existing system
The 'existing system' in this case is simplified to two csv input files containing the following data....
tFileInputDelimited_1 contains a reference to file containing records consisting of (person_id, surname)
tFileInputDelimited_2 contains a reference to file containing records consisting of (person_id, occupation)
Note: tFileInputDelimited inputs are used just to create a simple example. In practice it's more likely another input component (ie. connectors to other systems) would be used which would interface directly to a database.
The file schemas are created by hand in the component properties. The schema for tFileInputDelimited_1 (id, surname) is shown below. The schema for tFileInputDelimited_2 contians 'occupation' instead of 'surname'.
3. Grouping for repeatable fields
As there is a 1:N relationship between person and occupations, it is represented by a repeatable field in CollectionSpace. In the CollectionSpace JAXB libraries this is represented by a List. The next step will aggregate a persons occupations into a list, grouped by the persons id.
The data schema for tAggregateRow_2 is setup as follows. A new occupation_list output is manually created with type 'List'.
The tAggregateRow_2 module should be set up as shown below. Note the "Group by" and "Operations" settings.
4. Mapping the data migration
The tMap component is used to visually map fields from input components to outputs. Below is mapping from the two input rows to the (manually created) 'persons' output row. A join has been made on the persons 'id' of the inputs to enable mapping to a single output row (ie. record). Note that this new output row will be automatically represented in Talend as a new 'personsStruct' type with the column names mapped to variables of the same names in this class (this is relevant to the code routine, shown later).
5. Convert to CollectionSpace format
This approach uses the CollectionSpace JAXB libraries to create valid XML from Persons information. The record variables are copied to the CollectionSpace JAXB object variables from which are generated the correct XML.
Create new Talend routine
A new Talend 'routine' (ie. Java class) is created and the following external Java libraries added (right click on the routine and select "Edit routine libraries"). The libraries can be downloaded from these locations:
- org.collectionspace.services.person.jaxb-1.1-SNAPSHOT.jar (copied from Services code in release 1.3, path "/services/JaxRsServiceProvider/target/cspace-services/WEB-INF/lib/")
Below is the code contained in the routine (named 'personTest'). This class contains the 'get'PersonXml' method referenced in the call from tJavaRow component, above. Note: The type named "personsStruct" only exists if the output in the mapping is named "persons". The import path for that structure will depend on the workspace/job names. As can be seen below, adding additional single fields is trivial, where as adding lists requires finding the type of that list and populating it from the Talend list.
Import the routine
The routine needs adding to the Routine Dependencies of the job (close the job and right click on it to edit these).
Call the routine
To call the conversion routine a tJavaRow component is used. This component passes the row data to a routine, receives the xml and forwards it to the next component. The schema for TJavaRow is modified to add just one output row as shown here :
The "code" part of the component only contains the following line:
6. Import to CollectionSpace
For simplicity this job contains no actual import to CollectionSpace. Possibilities for import include:
- importing and calling the Services Client APIs from within a Talend Routine
- generating Nuxeo formatted xml for import with Nuxeo Shell
- generating appropriately formatted files for import via CollectionSpaces "import" functionality (not yet implemented)
However, in this job we just create a single file for each Person record in the appropriate CollectionSpace XML format. This uses the tFileOutputDelimeted component. To configure this component provide a path and file name, set both delimiters to "", and under the "Advanced" settings, select "split into several files", set "rows in each output file" to 1 and "encoding" to UTF-8.
Below is the output of the job.
Output file 1:
Ouput file 2: