Quick view of Funnel Stage

Funnel Stage:

Funnel stage is processing stage is Datastage, which is used to combine more than one file into single file. It can support n #.of Input links and one output link.But prerequisite for it is that each and every file in source should have same Metadata. Here metadata means Data types and Column names too.

Funnel Stage provides 3 funnel types in combining the files.Below same has been described.

v     Sequence Funnel.

v     Continuous Funnel.

v     Sort Funnel.

Below is the Stage Page of Funnel Stage.

Continuous Funnel:

            In this type of funneling, all the input rows sent to output as they arrived for processing.

Below example explains in detail.

Below are 3 input files data.

File 1:

EID	ENAME	DEPT NO	MGR_ID
102	Joe	10	105
103	Latha	10	102
101	Leela	20	101

File 2:

EID	ENAME	DEPT NO	MGR_ID
110	Vidhya	30	109
134	Rama	20	101
112	Neethu	10	111

File 3:

EID	ENAME	DEPT NO	MGR_ID
456	Yogesh	10	324
345	Jeevan	20	101
909	Varu	10	101

Below can be the output of Continuous Funnle(you can’t guarantee the Order).

Output:

EID	ENAME	DEPT NO	MGR_ID
102	Joe	10	105
103	Latha	10	102
101	Leela	20	101
456	Yogesh	10	324
134	Rama	20	101
112	Neethu	10	111
110	Vidhya	30	109
345	Jeevan	20	101
909	Varu	10	101

Sequence Funnel:

Sequence Funnel copies all records from the first input data set to the output data  set, then all the records from the second input data set, and so on.

Output would be below for above input.

EID	ENAME	DEPT NO	MGR_ID
102	Joe	10	105
103	Latha	10	102
101	Leela	20	101
110	Vidhya	30	109
134	Rama	20	101
112	Neethu	10	111
456	Yogesh	10	324
345	Jeevan	20	101
909	Varu	10	101

Sort Funnel:

            Sort Funnel combines the input records in the order defined by the Value of one or more key columns and the order of the output records is determined by these sorting keys.         

Typically all input data sets for a sort funnel operation are hash-partitioned before they’re sorted (choosing the auto partitioning method will ensure that this is done). Hash partitioning guarantees that all records with the same key column values are located in the same partition and so are processed on the same node. If sorting and partitioning are carried out on separate stages before the Funnel stage, this partitioning must be

Preserved.

The sort funnel operation allows you to set one primary key and multiple

Secondary keys. The Funnel stage first examines the primary key in each input record. For multiple records with the same primary key value, it then examines secondary keys to determine the order of records it will output.

Output would be below for above 3 input files.

101	Leela	20	101
102	Joe	10	105
103	Latha	10	102
110	Vidhya	30	109
112	Neethu	10	111
134	Rama	20	101
345	Jeevan	20	101
456	Yogesh	10	324
909	Varu	10	101