A Quick look at Routines in Datastage

Datasatge has 2 types of routines ,Below are the 2 types. 1.Before/After Subroutine. 2.Transformer routines/Functions. Before/After Subroutines :  These are built-in routines.which can be called in before or after subroutines. Below is the list of the same. 1.DSSendMail :Used to Send mail using Local send mail program. 2.DSWaitForFile : This routine is called to suspend a job until a named job either exists, or does not exist. 3.DSReport :Used to Generate Job Execution Report. 4.ExecDos: This routine executes a command via an MS-DOS shell. The command executed is specified...

Useful Options of sed command

          How to use sed, a special editor for modifying files automatically. If you want to write a program to make changes in a file, sed is the tool to use. Sed is the ultimate stream editor. If that sounds strange, picture a stream flowing through a pipe. sed is a marvelous utility. The essential command: s for substitution:             Sed has several commands, but most people only use substitute: s. The substitute command changes all occurrences of the regular expression into a new value. A...

Datastage Parallelism Explained

               The simultaneous use of more than one CPU or processor core to execute a program or multiple computational threads is called parallel processing or Parallelism. Ideally, parallel processing makes programs run faster because there are more engines (CPUs or Cores) running it. as you all know Datastage supports 2 types of parallelism. 1.Pipeline parallelism. 2.Partition parallelism. Pipeline Parallelism :      As and when a row/set of rows is/are processed...

Importance of Configuration File in Datastage.

            The Datastage configuration file is a master control file (a text file which sits on the server side) for a job which describes the parallel system resources and architecture. The configuration file provides hardware configuration for supporting such architectures as SMP (Single machine with multiple CPU, shared memory and disk), Grid, Cluster or MPP (multiple CPU, multiple nodes and dedicated memory per node). DataStage understands the architecture of the system through this file.            This is...

How to Run Datastage Job from Unix Command Line .?

                    Most Data warehousing project requires that your job runs in batches at specified time slots. In such cases the Datastage jobs are usually scheduled by using an external scheduling tool like ESP Scheduler, Control M, Autosys, etc… This is made possible by writing scripts that will run your jobs through the command line.  I would consider the command line & very powerful interface to Datastage which lets us do more than just run the normal job. There guides present in the Datastage documentation will be very helpful...

Dataset Handling in Unix with Orchadmin

                DataStage stores data in persistent internal (specific to DataStage) format in the form of Data sets. Orchestrate Data set aid in the parallel processing of data and are much faster performance wise. They help in achieving end-to-end parallelism by writing data in partitioned  form and maintaining the sort order. Orchestrate Data set consists of one or more data  files stored on multiple processing nodes. A parallel data set is represented on disk by:  • A  single descriptor  file  -...

Know your DS Server architechture is SMP or MPP. ?

                   To know whether you are working on SMP or MPP ,Just have look at your configuration file .If "FastName"has same value for all the nodes in your configuration file then you are running on SMP. If there are different "FastName's" then its MPP. Sample config File for SMP: main_program: APT configuration file: /ibmiscode/Configurations/ETL/my_config.apt {    node "Node1" {      fastname "FastName1"      pools ""      resource disk "/is_resource1/isent" {pools ""}   ...