Sharing Knlowledge With The World…

Month: April 2014

Talend Netezza Interaction

NET_IBM_RGB

Hello Readers,

Today we will discuss about how  Talend – Netezza interactwith each other , along with a simple demonstration that will use the available Netezza components.

you will find all Netezza related components in Palette available.

STEPS : –

1) Installing NZJDBC,jar :-  Initially when you drag any of Netezza component from the Palette , and if your library does not contain required .jar to support those components then it will throw an error as shown below.

This means your TOS do not have supported .jar

Talend Netezza Interaction

nzjdbc-error-mark

AUTOMATIC SETUP

Immediately a popup will come which will automatically install the required drivers. However if it fails to install the required .jar then you have to manually place that jar file.

MANUAL SETUP

(a)  Download nzjdbc.jar from the internet.

(b) Copy downloaded .jar file into this location  (C:\TOS\TOS_DI-Win32-r101800-V5.3.0\lib\java)

(c) Save your Current Job and Re-Start TOS (Talend Open Studio).

 

 EXAMPLE 

Now you are set to go ,just drag and drop your required components from the palette and connect them as per the business requirement.

This below job will show a simple Talend Netezaa Interaction and will create an excel file from a Netezza Database Table.

 

Talend Netezza Interaction

Talend Netezza Interaction

 

A) Drag tNetezzaConnection from the palette and fill  all required entries like   Host , DB-Name and  Password

B) Now to Read data from a table drag tNetezzaInput and specify attributes such as table Name and write your SQL like

(select emp_id , ename from emp limit 100)    *It will only fetch 100 rows from the employee table

C) At final to populate the row/data to some output( Excel file ) drop this component  to the designer area.

D) Connect All components as shown in figure.

E) Save and Run The Job.

 

Talend Netezza Interaction Successful …

 

 

Check out our other posts on Talend Interview Questions

 Author : – Ankit Kansal & Nayan Naik

 

 

Continue Reading

No-SQL — A Basic Overview

NO-SQL Systems
What are they?
The first question that came into my mind when I initially started working with these systems.
I heard all kinds of technologies Hadoop,MongoDB,Couch,Hive,Pig,Cassandra , all kinds of technical jargons kicked in ,Map-Reduce,JSON etc etc…so here i try n give our readers a basic overview of what No-SQL Systems are :With the boom of data in this modern-day , not all large scale data management and analysis is BEST solved using RDBMS,though we cant say that No-SQL systems are here to replace Traditional databases in any way.Solutions to the use of a type of database depends largely on the application or the problem faced.No-SQL systems are evolving at a fast pace ,and its a treat to see the developments.As of now there are no declarative query language in NoSQL statements hence more programming is needed.
Lets start with discussing the features of a NoSQL system:

  • No defined schema: unlike the traditional DB ,there are no defined tables with columns,in NoSQL systems the schema is flexible.
  • Cheap: Most NoSQL systems are open source , and very easy to setup.
  • Scalability: These systems are highly scalable.
  • Avalibility:Highly reliable in terms of avalibilty.
  • Performance : high performance.

There are various NoSQL databases available as of no, I would broadly categorize them into four categories:

  • Mapreduce framework
  • Key-Value stores.
  • Document stores.
  • Graph database systems.

Map-Reduce framework
This framework is mostly implemented in an OLAP(Online Analytical Processing) system where the complex analysis covers a large section of the data.Was originally invented by Google ,and now there is an open source Hadoop that implements MapReduce framework.
There is no data model,data is stored in files both as input and file.In Hadoop ,the implemented file system is called HDFS(Hadoop file distriduted system)
User provides a set of specific functions to process data using HDFS:
map()
reduce()
reader() – this function is used to read records from a file
writer() – this funtion is used to write records to a file.

Once the user provides the above functions the system provides data processing and scalability

Key-Value stores
This framework is more specific to OLTP(Online Transaction processing) systems,the Key -Value stores allows users to store schema less data in form of a key-value relationship in contrast to tradition SQL which had to scan through hierarchies of tables or structured schema to get complex data sets.

Document stores

Graph database systems

 

Continue Reading

Get Latest File from FTP using shell scripting in Unix

Hello readers,I couldn’t find anything much helpful regarding this topic , hence decided to do what we do best, blog about it !!!!!Our FTP server receives files on a daily basis with the format FILENAME_YYYYMMDD.txt , hence we needed a shell script to get the file with the latest timestamp from the server.

Here’s the shell script we developed:

************code_begins********************************

#!/bin/ksh
#file_format : filename_YYYYMMDD

ftp -inv sa2-sftp01.zs.local << END_FTP_SCRIPT
quote USER USERNAME
quote PASS PASSWORD
bin
cd DIRECTORY_PATH
ls . temp.txt
bye
#cat temp.txt
END_FTP_SCRIPT
pwd
fil_name = `awk -F_ ‘{print $1 ” ” $2}’ temp.txt | sort -n -k 2 | tail -1`
#awk -v {print $9} $filename
echo $fil_name
#ftp -inv sa2-sftp01.zs.local<< EOF
#quote USER USERNAME
#quote PASS PASSWORD
#bin
#cd DIRECTORY_PATH
#get($filename)
#bye
#quit

************code_ends********************************(Changes to code to be made don’t publish)

Continue Reading
PageLines