Wednesday, July 10, 2013

Quick view of Funnel Stage

Funnel Stage:

Funnel stage is processing stage is Datastage, which is used to combine more than one file into single file. It can support n #.of Input links and one output link.But prerequisite for it is that each and every file in source should have same Metadata. Here metadata means Data types and Column names too.

Funnel Stage provides 3 funnel types in combining the files.Below same has been described.
v     Sequence Funnel.
v     Continuous Funnel.
v     Sort Funnel.

Below is the Stage Page of Funnel Stage.





Continuous Funnel:
            In this type of funneling, all the input rows sent to output as they arrived for processing.

Below example explains in detail.
Below are 3 input files data.

File 1:
EID
ENAME
DEPT NO
MGR_ID
102
Joe
10
105
103
Latha
10
102
101
Leela
20
101

File 2:
EID
ENAME
DEPT NO
MGR_ID
110
Vidhya
30
109
134
Rama
20
101
112
Neethu
10
111

File 3:
EID
ENAME
DEPT NO
MGR_ID
456
Yogesh
10
324
345
Jeevan
20
101
909
Varu
10
101


Below can be the output of Continuous Funnle(you can’t guarantee the Order).
Output:

EID
ENAME
DEPT NO
MGR_ID
102
Joe
10
105
103
Latha
10
102
101
Leela
20
101
456
Yogesh
10
324
134
Rama
20
101
112
Neethu
10
111
110
Vidhya
30
109
345
Jeevan
20
101
909
Varu
10
101


Sequence Funnel:
Sequence Funnel copies all records from the first input data set to the output data  set, then all the records from the second input data set, and so on.

Output would be below for above input.

EID
ENAME
DEPT NO
MGR_ID
102
Joe
10
105
103
Latha
10
102
101
Leela
20
101
110
Vidhya
30
109
134
Rama
20
101
112
Neethu
10
111
456
Yogesh
10
324
345
Jeevan
20
101
909
Varu
10
101


Sort Funnel:
            Sort Funnel combines the input records in the order defined by the Value of one or more key columns and the order of the output records is determined by these sorting keys.         
Typically all input data sets for a sort funnel operation are hash-partitioned before they’re sorted (choosing the auto partitioning method will ensure that this is done). Hash partitioning guarantees that all records with the same key column values are located in the same partition and so are processed on the same node. If sorting and partitioning are carried out on separate stages before the Funnel stage, this partitioning must be
Preserved.
The sort funnel operation allows you to set one primary key and multiple
Secondary keys. The Funnel stage first examines the primary key in each input record. For multiple records with the same primary key value, it then examines secondary keys to determine the order of records it will output.

Output would be below for above 3 input files.

101
Leela
20
101
102
Joe
10
105
103
Latha
10
102
110
Vidhya
30
109
112
Neethu
10
111
134
Rama
20
101
345
Jeevan
20
101
456
Yogesh
10
324
909
Varu
10
101


0 comments:

 

Datastage Doctrina Copyright © 2011 -- Template created by O Pregador -- Powered by Blogger

Receive all updates, tips and tricks via Facebook. Just Click the Like Button Below