Wednesday, August 19, 2015

IBM - InfoSphere DataStage Xml Pack(XML Input)

IBM Info sphere Datastage provides three different stages to read,write and transform xml data.Below are the 3 stages available to perform the above mentioned operations.
  • XML Input
  • XML Output
  • XML Transformer
Below we are going to look about XML Input stage.
XML Input Stage :
Xml Input stage can read and validate input xml against provided xsd in schemalocation attribute value.In case of non availability of xsd it will not be able to read your xml data at all.So make sure you have a valid path and file or else simply not to have it at all.
You can read xml data through External source stage or sequential file stage and then source it to XML input stage to make it tabular data. In order to do that you need to provide the valid name space declarations available in your input xmls in transformations/stage tab of xml input stage. Under input tab specify what is your type of input whether it is file or column, based on the option you decide to choose under input tab use appropriate commands in external source stage. For either of the options provide appropriate values like column name or file path accordingly.
Under Output tab of Xml input stage you need to specify the columns you want to extract and their corresponding XPATH.In order to make this thing simple import your xsd of input file using XML Table Definition under Table Definitions of Import Menu. Load the schema you imported using Imported operation to output tabs schema which will also load all the information required with XPATH too.

Below is the sample job :


External Source stage:

As mentioned above select appropriate command in here based on option you are going to choose in xml stage.


Output Tab of External Source stage:



XML Input Stage :

If you already reading data in ESS stage then choose XML document else if you are reading filename with path then choose other.

Input Tab:


Output Tab:


Monday, October 13, 2014

Difference Between Normal Lookup and Sparse Lookup

Normal Lookup :- 
  • Normal Lookup data needs to be in memory
  • Normal might provide poor performance if the reference data is huge as it has to put all the data in memory.
  • Normal Lookup can have more than one reference link.
  • Normal lookup can be used with any database
Sparse Lookup :- 
  • Sparse Lookup directly hits the database.
  • If the input stream data is less and reference data is more like 1:100 or more in such cases sparse lookup is better.
  • Sparse Lookup,we can only have one reference link.
  • Sparse lookup,we can only use for Oracle and DB2.
  • Sparse lookup sends individual sql statements for every incoming row.(Imagine if the reference data is  huge).
This Lookup type option can be found in Oracle or DB2 stages.Default is Normal.


Saturday, December 21, 2013

Things to be considered while desinging a DS job

  1. When you need to run the same sequence of jobs again and again, better create a sequencer with all the jobs that you need to run. Running this sequencer will run all the jobs. You can provide the sequence as per your requirement. 
  2.  If you are using a copy or a filter stage either immediately after or immediately before a transformer stage, you are reducing the efficiency by using more stages because a transformer does the job of both copy stage as well as a filter stage 
  3. Use Sort stages instead of Remove duplicate stages. Sort stage has got more grouping options and sort indicator options. 
  4. Turn off Runtime Column propagation wherever it’s not required. 
  5.  Make use of Modify, Filter, and Aggregation, Col. Generator etc stages instead of Transformer stage only if the anticipated volumes are high and performance becomes a problem. Otherwise use Transformer. It is very easy to code a transformer than a modify stage. 
  6. Avoid propagation of unnecessary metadata between the stages. Use Modify stage and drop the metadata. Modify stage will drop the metadata only when explicitly specified using DROP clause. 
  7. Add reject files wherever you need reprocessing of rejected records or you think considerable data loss may happen. Try to keep reject file at least at Sequential file stages and writing to Database stages. 
  8. Make use of Order By clause when a DB stage is being used in join. The intention is to make use of Database power for sorting instead of Data Stage resources. Keep the join partitioning as Auto. Indicate don’t sort option between DB stage and join stage using sort stage when using order by clause. 
  9. While doing Outer joins, you can make use of Dummy variables for just Null checking instead of fetching an explicit column from table. 
  10. Data Partitioning is very important part of Parallel job design. It’s always advisable to have the data partitioning as ‘Auto’ unless you are comfortable with partitioning, since all Data Stage stages are designed to perform in the required way with Auto partitioning. 
  11.  Do remember that Modify drops the Metadata only when it is explicitly asked to do so using KEEP/DROP clauses. 
  12.  Range Look-up: Range Look-up is equivalent to the operator between. Lookup against a range of values was difficult to implement in previous Data Stage versions. By having this functionality in the lookup stage, comparing a source column to a range of two lookup columns or a lookup column to a range of two source columns can be easily implemented. 
  13.  Use a Copy stage to dump out data to intermediate peek stages or sequential debug files. Copy stages get removed during compile time so they do not increase overhead 
  14. Where you are using a Copy stage with a single input and a single output, you should ensure that you set the Force property in the stage editor TRUE. This prevents DataStage from deciding that the Copy operation is superfluous and optimizing it out of the job

SQL Query Order of Operations:

SQL Query Order of Operations:

Tags: SQL
Lately, I have been looking into SQL query optimization. We recently installed SeeFusion on our server and I can see where my long running tasks are causing the server to slow down. Turns out, not surprisingly, that the slow pages are very query-intense. Granted, a lot of these pages were pages years ago before I knew what nice code looked like, but the good news it, lots of room for optimization and clean up.
To start out, I thought it would be good to look up the order in which SQL directives get executed as this will change the way I can optimize:
1.                FROM clause
2.                WHERE clause
3.                GROUP BY clause
4.                HAVING clause
5.                SELECT clause
6.                ORDER BY clause
This order holds some very interesting pros/cons:

FROM Clause

Since this clause executes first, it is our first opportunity to narrow down possible record set sizes. This is why I put as many of my ON rules (for joins) as possible in this area as opposed to in the WHERE clause:
·                                 FROM
·                                 contact c
·                                 INNER JOIN
·                                 display_status d
·                                 ON
·                                 (
·                                 c.display_status_id = d.id
·                                 AND
·                                 d.is_active = 1
·                                 AND
·                                 d.is_viewable = 1
·                                 )
This way, by the time we get to the WHERE clause, we will have already excluded rows where is_active and is_viewable do not equal 1.

WHERE Clause

With the WHERE clause coming second, it becomes obvious why so many people get confused as to why their SELECT columns are not referencable in the WHERE clause. If you create a column in the SELECT directive:
·                                 SELECT
·                                 ( 'foo' ) AS bar
It will not be available in the WHERE clause because the SELECT clause has not even been executed at the time the WHERE clause is being run.

ORDER BY Clause

It might confuse people that their calculated SELECT columns (see above) are not available in the WHERE clause, but they ARE available in the ORDER BY clause, but this makes perfect sense. Because the SELECT clause executed right before hand, everything from the SELECT should be available at the time of ORDER BY execution.
I am sure there are other implications based on the SQL clause order of operations, but these are the most obvious to me and can help people really figure out where to tweak their code.
Courtesy: Ben Nadel

Thursday, July 11, 2013

Is Modify Stage can be a alternate for Transformer



The Modify stage is a processing stage. It can have a single input link and
a single output link. It can also be used to modify schema, Keep or Drop the input columns.If you are using transformer just to drop or change schema of columns then it is better we use modify stage.

Below is the syntax of each of the operation possible in modify stage and below mentioned syntax can be used in specification box of modify stage..

'keep field1, field2, ... fieldn; '
'drop field1, field2, ... fieldn;' 

 ‘newField1=oldField1; newField2=oldField2;...newFieldn=oldFieldn;’

a_1 = a
Date field conversions:
dateField = date_from_days_since[date](int32Field)
dateField = date_from_julian_day(uint32Field)
dateField = date_from_string[date_format | date_uformat] (stringField)
dateField = date_from_timestamp(tsField
int8Field = month_day_from_date(dateField)
int8Field = weekday_from_date[originDay](dateField)
int16Field = year_day_from_date(dateField)
int32Field = days_since_from_date[source_date]
uint32Field = julian_day_from_date(dateField)
int8Field = month_from_date(dateField)
dateField = next_weekday_from_date[day](dateField)
dateField= previous_weekday_from_date[day]
stringField = string_from_date [date_format | ufornat] (dateField)
tsField = timestamp_from_date[time](dateField)
int16Field = year_from_date(dateField)
int8Field=year_week_from_date(dateField)

Decimal Field Conversions:
decimal from decimal decimalField = decimal_from_decimal[r_type](decimalField)
dfloat from dfloat dfloatField = mantissa_from_dfloat(dfloatField)
int32 from decimal int32Field = int32_from_decimal[r_type, fix_zero](decimalField)
int64 from decimal int64Field = int64_from_decimal[r_type, fix_zero](decimalField)
string from decimal stringField = string_from_decimal[fix_zero][suppress_zero](decimalField)
uint64 from decimal uint64Field = uint64_from_decimal[r_type, fix_zero](decimalField)

string Conversions:

hue:string[10] = string_trim['Z', end, begin](color)
stringField=substring(string, starting_position, length)
decimalField = decimal_from_string(stringField) Converts strings to decimals.
stringField = string_from_decimal[fix_zero] [suppress_zero] (decimalField) Converts decimals to strings
dateField = date_from_string [date_format | date_uformat] (stringField)
stringField = string_from_date [date_format | date_uformat] (dateField)
int32Field=string_length(stringField)stringField=lowercase_string (stringField)
stringField=uppercase_string (stringField)
stringField = string_from_time [time_format | time_uformat ] (timeField)
int8Field = hours_from_time(timeField) hours from time
int32Field = microseconds_from_time(timeField) microseconds from time
int8Field = minutes_from_time(timeField) minutes from time
dfloatField = seconds_from_time(timeField) seconds from time
dfloatField = midnight_seconds_from_time(timeField) seconds-from-midnight from time
stringField = string_from_time [time_format | time_uformat] (timeField)


Wednesday, July 10, 2013

Quick view of Funnel Stage

Funnel Stage:

Funnel stage is processing stage is Datastage, which is used to combine more than one file into single file. It can support n #.of Input links and one output link.But prerequisite for it is that each and every file in source should have same Metadata. Here metadata means Data types and Column names too.

Funnel Stage provides 3 funnel types in combining the files.Below same has been described.
v     Sequence Funnel.
v     Continuous Funnel.
v     Sort Funnel.

Below is the Stage Page of Funnel Stage.





Continuous Funnel:
            In this type of funneling, all the input rows sent to output as they arrived for processing.

Below example explains in detail.
Below are 3 input files data.

File 1:
EID
ENAME
DEPT NO
MGR_ID
102
Joe
10
105
103
Latha
10
102
101
Leela
20
101

File 2:
EID
ENAME
DEPT NO
MGR_ID
110
Vidhya
30
109
134
Rama
20
101
112
Neethu
10
111

File 3:
EID
ENAME
DEPT NO
MGR_ID
456
Yogesh
10
324
345
Jeevan
20
101
909
Varu
10
101


Below can be the output of Continuous Funnle(you can’t guarantee the Order).
Output:

EID
ENAME
DEPT NO
MGR_ID
102
Joe
10
105
103
Latha
10
102
101
Leela
20
101
456
Yogesh
10
324
134
Rama
20
101
112
Neethu
10
111
110
Vidhya
30
109
345
Jeevan
20
101
909
Varu
10
101


Sequence Funnel:
Sequence Funnel copies all records from the first input data set to the output data  set, then all the records from the second input data set, and so on.

Output would be below for above input.

EID
ENAME
DEPT NO
MGR_ID
102
Joe
10
105
103
Latha
10
102
101
Leela
20
101
110
Vidhya
30
109
134
Rama
20
101
112
Neethu
10
111
456
Yogesh
10
324
345
Jeevan
20
101
909
Varu
10
101


Sort Funnel:
            Sort Funnel combines the input records in the order defined by the Value of one or more key columns and the order of the output records is determined by these sorting keys.         
Typically all input data sets for a sort funnel operation are hash-partitioned before they’re sorted (choosing the auto partitioning method will ensure that this is done). Hash partitioning guarantees that all records with the same key column values are located in the same partition and so are processed on the same node. If sorting and partitioning are carried out on separate stages before the Funnel stage, this partitioning must be
Preserved.
The sort funnel operation allows you to set one primary key and multiple
Secondary keys. The Funnel stage first examines the primary key in each input record. For multiple records with the same primary key value, it then examines secondary keys to determine the order of records it will output.

Output would be below for above 3 input files.

101
Leela
20
101
102
Joe
10
105
103
Latha
10
102
110
Vidhya
30
109
112
Neethu
10
111
134
Rama
20
101
345
Jeevan
20
101
456
Yogesh
10
324
909
Varu
10
101


 

Datastage Doctrina Copyright © 2011 -- Template created by O Pregador -- Powered by Blogger

Receive all updates, tips and tricks via Facebook. Just Click the Like Button Below

▼