Sharing Knlowledge With The World…

Category: MONGO DB

Business Intelligence Benefits – Top 10 Reasons your business should go for BI

Business Intelligence Benefits

Business Intelligence BenefitsBusiness Intelligence is on the verge of huge change. For a long time it has been limited to only the larger organizations . With the advent of open-source technologies , BI has become much more accessible . In today’s post we would be discussing some key  Business Intelligence benefits to create awareness of this wonderful concept and as to why one should leverage BI capabilities in their enterprise solutions.

1) Stay ahead of competition

Arm your team with compelling BI reporting and examination so they can stay in front of the pack, and exploit patterns and new open doors as they arrive. Visually intelligent dashboards gives your group access to sales financial and operational parameters of your product/organization as well as your competitor.

2) Foresee and react to patterns

Examine organization and competitor information to set competitive objectives, and track achievements against key performance indicators for the business . Analyze trends and mitigate business risks. Apply hypothetical variables to your information to foresee possible outcomes , historical informational data caters to creation of a predictive model.

3) Empower Decision Making with on-demand data .

Business Intelligence  lets your team, from field reps to top level administration, access dynamic data and empower them with business perspective by offering deep insights into data , enabling better decision making.

4) Visual dashboards and custom-made reports at your fingertips.

Visually stunning and detailed information could be conveyed to the significant individuals via reporting dashdboards with custom reports , with ability to drill down and dissect the information to gain insights. Furthermore BI reports has provisions to customize reports to fit each users needs through simple drag drop functionality.

5) Reduce wastage , efficient manage resources.

BI helps to get rid of  nonaligned data systems and eliminate resource wastage. Eliminates wasting time and resources spent on correcting data errors and manually compiling spreadsheet based reports , so your resources can add maximum value to the organization growth by focusing more on the business growth.

6) Track and monitor Customer behavior

Business Intelligence helps you to identify what customers are buying, what are trends of the buying customers. You can monitor customer behavior and use data insights to drive loyalty and maximize profit by identifying up-sell and cross-sell opportunities. BI also provides opportunities to identify conversions for a potential lead to become a customer

DataWareho

7) Ability to mine data , discover your true manufacturing costs

Business Intelligence software has the ability to create automated alerts to notify managers of incidents that require action. This brings more efficiency in managing procurement , managing inventory , delivery which reduces cost and mitigates risks.

8) Discovering New Questions and Their Answers

Business intelligence is data driven in nature , trending patterns helps us identify and ask new questions leading to a data-driven discovery. Else it could work the other way round where-in you may have certain questions which could be answered using historical data in the warehouse.

9)  Mobile Device Accesibility

BI enables easy integration with various  mobile devices like I-pads,laptops , mobiles. Hence reps/directors can have access to reports on-site on -fields or in office.

10) Efficient Data Management

BI enables integration of enterprise wide  disparate data sources into a single unified data-warehouse,  to ensure consistency and accuracy. This data is cleansed quality data , which makes data management very easy.

These points here just scratch the surface when it comes to Business Intelligence benefits , if you feel strongly feel about other benefits you want to point out please feel free to comment below and let us know.

Check this very awesome video out which outlines BI concepts in a very interesting way.

Continue Reading

Journaling in MongoDB

Ankit Kansal & Nayan Naik
JOURNALING is the concept in MongoDB which provides you with the feature of write ahead logging(WAL) mechanism. It helps in achieving write operations durability and crash recovery to your system. 
WRITE AHEAD LOGGING
Write ahead logging technique in mongoDB provides durability against miss-happenings like power failures. WAL enabled system writes data to a log file before they writes data to actual data files for the database. If a system using WAL got some failures then after restarting the system cross verify it against the log and after applying some operations comes into consistent state.
If a mongo server i.e. mongod instance is running without journaling enabled then there may be much possibility that your system is not in a consistent state. In that scenario you must perform repair mechanism or resync your system if replication(making duplicacy of  data) is enabled.
 In 64 bit systems journaling is by default enabled while in 32 bit system you explicitly have to enable it using –journal command.
JOURNAL FILES
By default in 64 bit, system creates a folder named journal in the specified –db folder as a default however you can change the destination of your journaling file logs. By default journaling file size is of 1GB and when the max size is reached a new file is created. you can change this limit explicitly by changing some parameters.

WRITING OPERATIONS IN MONGODB
When you starts your mongoDB instance the operating system maps your data files that is db files presented onto disk to the shared view(it is a view which is present between your mongod instance and db files). Basically it maps the memory address of the db file so from there you have access to your data presented in the data file.
when the OS maps the address and you performs any CRUD operations then operating system simply flush all the changes to your DB files with the help of Shared view. This is what happens when you do not have journaling enabled on your system. In this time if any of the system crash down than all the data which is not yet flushed to the DB files will not be recovered and thus, your DB will be in inconsistent state.

When you have journaling enabled then Private view comes into picture Private view is the view which do not have direct access to your DB files. So when you perform some operations then mongod write these changes to your private view and then private view maps the data onto journal files. As more and more data comes in Journal file is appended till it’s max limit is reached. Once the journal file is having data so from now if system crashes then your data will be easily recovered. Now the jounal files maps will be mapped to the shared view for flushing to the db files but before flushing the data onto the db files the shared view is again mapped with shared view. And finally the data is flushed to the DB files.

thanks a lot for reading…..






 
Continue Reading

functions in mongoDB

printHello
Ankit Kansal & Nayan Naik 
MongoDB provides you with the functionality of creating functions for session specific as well as at database level.
SESSION SPECIFIC:-  Session specific functions are those functions which you create at that specific time only and they can only be accessed till you are logged in to that client. Once you disconnect from the session the function existence will be removed.
example:-
Lets start with a basic example
Steps
  1. write a key word function.
  2. write the name of the function which you want to create.
  3. define the number of parameters.
  4. write the logic for which function is to be created.
  5. use return statement if you want to return any data member from your function.
NOW LETS CREATE A FUNCTION WHICH TAKES TWO PARAMETERS

*compiler itself will not check whether you are passing correct data type or not if you pass one number and one character then it will concatenate them and return.

The functions described are session specific so once you close down your session and then want to use again then you don’t them available for you.
To use the function again and again you have to save the function inside the DB itself in the name collection system.js

*To view the definition of a function you just have to write the name of the function and the shell itself return you the whole definition of the function.


CREATING FUNCTION OVER DB SIDE
MongoDB also enables you to save the function which remains at db level in the name collection db.system.js
syntax:-

db.system.js.save(
                   { _id: “nameOfFunction”,
                     value : function(args..) {Body of the function }
                   }
                 )

This command will help you to save a function at the DB level in the system.js collection. By default this function acquire a WRITE AND READ lock on your mongod instance hence you won’t be able to read and write on the DB for that particular time.

 example:-

To call a function the are few ways one way is to use 
db.eval() method as shown in figure below but there are some restrictions in this method as i already told you that when you call a function which resides at DB level it acquires a full mongod lock but there is a way so that you can avoid it but using db.eval() method you can not achieve this functionality.

**If you wish to execute a function that does not block a mongod instance then you have to use eval method with the db.runCommand() method.

example:


From the latest version you can directly call the DB methods just you have to call the >db.loadServerScripts() 
method and then by simply typing the method name you can run all the system.js collection methods.
>printValue(args) –and method will execute.







Thanks for reading…

Continue Reading

Adding Mongo Components to Talend Open Studio

talendConnection
Ankit Kansal & Nayan Naik

TALEND MONGO INTEGRATION

Today had an opportunity to implement data-integration using Talend where our source and target was MongoDB,Talend has been playing a major role in implementing Analytics over Big Database Systems and the functionalities they have provided are a treat!!!! Strangely Talend does not have its own MongoDB component by default though it can be downloaded from the Talend Exchange (Thanks to the component developed by Adrien MOGENET , you made our job real easy today !!).
Components provided for MongoDB:
1)tMongoDBInput
2)tMongoDBOutput
3)tMongoDBConnection

We are going to elaborate the steps we followed to add these components to your TOS and use them.

1)Download the components from http://www.talendforge.org/exchange/ and search for Mongo.

2)The components will be zipped format , Unzip the same.

3)Copy these Unzipped Components to the installation Path of TOS.
    C:TalendTOS_DI-Win32-r84309-V5.1.1pluginsorg.talend.designer.components.localprovider_5.1.1.r84309components

4)Copy the mongo-1.3.jar file in the component folder into the C:TalendTOS_DI-Win32-r84309-V5.1.1libjava
In many systems you might not be able to see this file then go with ADMINISTRATOR priviliges.

5)–optional for few systems——>>> Inside index.xml add
<jarsToRelativePath key=”mongo-1.3.jar” value=”org.talend.designer.components.localprovider_5.1.1.r84309/components/tMongoDBConnection/mongo-1.3.jar”/>
<jarsToRelativePath key=”mongo-1.3.jar” value=”org.talend.designer.components.localprovider_5.1.1.r84309/components/tMongoDBInput/mongo-1.3.jar”/>
<jarsToRelativePath key=”mongo-1.3.jar” value=”org.talend.designer.components.localprovider_5.1.1.r84309/components/tMongoDBOutput/mongo-1.3.jar”/>

save index.xml

6) Restart TOS

Drag and Drop the components from  the palette under no-sql  tag
 Describe the schema for input 
Describe the schema for Output
By this way you can transfer the data to and from the MongoDB.
* But keep in mind these components does not support Nested MongoDB Documents at any chance….
Thanks for Reading…..

Talend Interview Questions

Author : Ankit Kansal & Nayan Naik

Continue Reading

Loops in MongoDB

LOOPS IN MONGODB

Ankit Kansal & Nayan Naik
MongoDB itself has provided some basic looping mechanism through which you can achieve iteration functionality.
let’s discuss one by one.
FOR LOOP:- For loop is a basic loop available in mongodb and its syntax is just same as we have in other technologies.
For(var iTemp=0;iTemp<100;i++){
–list of statements;
.;
..;
…;
..n;
}
Within the curly braces{..} you can write your own statements and each of them will be executed till the loop condition satisfies.
lets see one examples using FOR LOOP
******************************************************************************
BASIC INSERTS AND DELETES
******************************************************************************
For(var iTemp=0;iTemp<100;iTemp++){
db.things.insert({idies:iTemp}); –to insert a document in a collection things.
db.things.remove({idies:iTemp}); –to remove a document from a collection things.
print (iTemp); –to print a local variable value iTemp.
}
*******************************************************************************

IF(CONDITION):-
MongoDB also provides you with the functionality of checking a condition and then performs some operation.
here is one example using FOR LOOP INTEGRATED WITH IF CONDITION 
******************************************************************************
REMOVING DOCUMENTS CONTAINING EVEN IDIES FROM COLLECTION
******************************************************************************
For(var iTemp=0;iTemp<100;iTemp++){
if(iTemp%2==0) –checking whether a condition is true or not
{
db.things.remove({idies:iTemp}); –to remove a document from a collection things.
print (iTemp); –to print a local variable value iTemp.
}
}
*This loop will removes all the documents which have even idies.
*******************************************************************************
CURSOR ATTRIBUTES
When you fetch a result and store it in a variable then you have certain attributes by which you can iterate over the retrieved data and perform several actions.
  1. cursor.hasNext()–> It returns Boolean either the true or false i.e. whether after the current document  any other document is present or not.
  2. cursor.next()–> It will return you the present value and move the cursor pointer to the next available document.

Lets see a basic example…

    *******************************************************************************
    WHILE LOOP WITH CURSOR ATTRIBUTES
    *******************************************************************************
    var curTemp = db.things.find(); –declaring a variable curTemp and initializing it with query result.
    var temp;
    while(curTemp.hasNext()){
    temp=curTemp.next(); –returning a json document and assigning it to temp variable
    printjson (temp);  — it will print json document one by one
    print (temp.idies); –it will print the values of idies field as accessed by . operator.
    }
    *Once the cursor is completely traversed then it will not return any value because the other attribute curTemp.hasNext() return false.
    error:-uncaught exception: error hasNext: false
    *******************************************************************************
    USING FOREACH:- Foreach provides you better functionality, using foreach you do not have to use cursor attributes such as curosr.hasNext() and cursor.next(). It implicitly performs those operations it automatically iterates on the data/documents available.

    syntax:-

    db.collection_name.find().forEach( function(obj)
    {
    .
    ..

    }
    );

    let’s go through an example:-The problem defines that i want to insert a column that will contain employee name,department_no and empno in a concatenated format.

    *******************************************************************************
    USING FOREACH
    *******************************************************************************

    db.employee.find().forEach( function(obj)
    {
    empno_str = (obj.empno).toString()+”-“;
    deptno_str = (obj.deptno).toString();
    name_str = obj.name+”-“;
    final = empno_str+name_str+deptno_str;
    obj.new_field_name = final;
    print (final);
    db.employee.save(obj);
    }
    );

    *using obj. you can access any field of the selected document and in this operator save function is updating the selected field as _id field is already present.more info UPDATE USING SAVE
    *******************************************************************************
    Continue Reading

    Data Retrieval MONGO – DB

    subdocument

    DATA RETRIEVAL IN MONGODB

    Ankit Kansal & Nayan Naik
    As we now that MongoDB is a NO-SQL database so we just can’t write  SQL queries to retrieve data from the MonogoDB. MongoDB itself has provided some basic methods by which you can easily retrieve the data. There are various operators provided through which you can easily retrieve the desired data.
    We will provide you MongoDB methods as well as SQL Queries on the same data set. For Practice session we have taken the favorite emp table from scott schema and populate the same emp table in MongoDB schema.
    1) Which query/method returns you all set of data from the Table/Collection(emp) ?
    ALL DATA——-
    SQL:-    select * from emp
    Mongo:- db.emp.find() —-for better representation you can write db.emp.find().pretty()  
    *To retrieve a single result from collection—-db.emp.findOne()
    2) Which query/method will return you data from specific columns lets say EMPNO,ENAME,SAL ?
    SELECT FIELD TO BE DISPLAYED
    SQL:-      select EMPNO,ENAME,SAL from emp;
    Mongo:-  db.emp.find({},{_id:0,EMPNO:1,ENAME:1,SAL:1});
    *please take care of field names they must be in same case as in collection unlike oracle it will not change the field cases implicitly.
    **Each document implicitly displays you _id field so if you don’t want it to be displaed then use _id:0.
     
    3) Which query/method will help in filtering data ?
    FILTERING DATA
     SQL:-     select * from emp where EMPNO = 7788;
    Mongo:-  db.emp.find({EMPNO:7788},{});
    *For String Data Type ENAME:”NAME” “” quotes are mandatory.4)Which method will return you the count of rows/documents currently present ? 
    COUNT
    SQL:-     select count(*) from emp;
    Mongo:- db.emp.count()——-or——db.emp.find().count() –return total count of documents.

    5)How to perform sorting on a given data set?
    SORT 
    SQL:-     select * from emp order by empno asc/desc    –by default it’s ascending
    Mongo:-  db.emp.find().sort({EMPNO:1/-1});     –(-1) for desending

    6)Querying with two or more condition and all must be true like AND condition?
    AND CONDITION 
    SQL:-     select * from emp where EMPNO=7369 and DEPTNO=20;
    Mongo:-  db.emp.find({EMPNO:7369,DEPTNO:20}).pretty()
     
    7)Querying with two or more condition and ANY of them will be true?
    OR CONDITION 
    SQL:-     select * from emp where EMPNO=7369 or DEPTNO=20;
    Mongo:-  db.employee.find({“$or”:[{EMPNO:1111},{DEPTNO:30}]}) –This will return you any of the matched result.

    8)Querying with two or more condition and ANY of them will be true with aggregation framework?
    OR WITH GROUP/AGGREGATION
    SQL:-     select column_name from emp where condition_1=? or condition_2=? group by column_name;
    Mongo:-  db.employee.aggregate({$match:{$or:[{empno:1111},{deptno:30}]}},{$group:{_id:”$deptno”}}) –Within array after $or clause you can have any number of condition with in each separate group distinguished by ,

    9)Explain how $in operator works with example?
    IN OPERATOR / $in operator
    SQL:-     select * from emp where empno in (1113,1114);
    Mongo:-  db.employee.find({empno:{$in:[1113,1114]}})


    10)Explain how $nin operator works with example?
    NIN OPERATOR / $in operator
    SQL:-     select * from emp where empno NOT in (1113,1114);
    Mongo:-  db.employee.find({empno:{$nin:[1113,1114]}})

    11)Explain how $lt/$gt operator works with example?
    $lt/$gt OPERATOR
    SQL:-     select * from emp where SAL > 1500 and SAL < 3000;
    Mongo:-  db.emp.find({EMPNO:{$gt:1500},EMPNO:{$lt:3000}});

    12)Explain how $gte/$lte operator works with example?
    Ans:- Same as above but it includes the given values also

    13)Explain how $ne operator works with example?
    Ans:- All outputs except the the match/where condition.

    14)Explain the usage of $exists operator?

    * Relational database management system works on fixed schema structure where  as MongoDB is schema less document based Database. $exists operator is to find out those documents which contains the specified field on the basis of two conditions either true or false. If true then only those documents will retrieved which contains the specified filed within the document and if false then only those records will be retrieved which do not contain that  field.
    example:- This query will return those documents which do not contain comm field.
    db.emp.find({COMM:{$exists:true}})
    15) Accessing data of sub documents in MongoDB?
    As we know that in mongodb a document can be contained with in another document and to access a sub document . operator must be used.
    example:-  
    Sub Documents
    16) Find out those employees which belongs to department higher than department 10 using max() or $max operator in MongoDB?
    max function provides you a functionality of specifying an upper bound to your query result.
    To use max function or the $max operator in mongodb first you have to build the index on that particular column and then you can query the collection to retrieve your data.
    example describes your query returns only that data which is lower to the department 30

    max function

    Min function totally works in reverse direction/order.





    Continue Reading

    Data Aggregation FrameWork

    aggr_project

    DATA AGGREGATION FRAMEWORK

    Ankit Kansal & Nayan Naik

    Data Aggregation is a powerful technique provided by MongoDB for performing aggregation analysis. This technique is mostly similar with what we have in oracle as GROUP BY clause.
    There are some of guidelines provided by MongoDB developer zone which you must follows to obtain the desired output.
    Data flow inside an aggregation function moves from left to right.
    Syntax:- 
                   db.collection_name.aggregate(–set of operators required) 
     Operators Lists:-
    1. $project
    2. $match 
    3. $skip
    4. $limit
    5. $group
    6. $unwind
    7. $sort
    8. $sum      

    Let’s understand the working of Aggregation framework with the use of Operators listed above.

    $project:-

                   $project is a PROJECTION operator and is used with aggregation framework and mainly its work is to project the selected fields from a  given document.

    $project
    In this query $project selects name filed from things collection by writing name:1 and as we know that _id field is by default selected so to avoid displaying _id field we have written _id:0.
    NOTE:- Aggregation framework works upon the principle of pipe lines that is once you have select the fields and you have moved to next part then only those selected fields will be available to perform operations for you and other fields from the collection will be vanished.more….

    example:-
     

     In the above figure the query is selecting/projecting only the names from the collection and later on when i am using $match/WHERE operator to put a condition on the data the output which received is empty. This shows working as a pipe line in aggregation framework.

    $match:-

                  $match operator is used to filter the documents based upon some defined conditions.This operator is mostly similar with WHERE clause of SQL.
    $match
     In the previous example i have used $match operator with $project operator. As described earlier $match operator is used for filtering the data based upon some required condition and then $project operator is used which helps in selecting the desired fields only.
                   
                     
    LETS TAKE SOME SCENARIOS WHICH WILL COVER MOST OF THE OPERATORS:-

    Note:-
    When you are using _id operator which tells about the column on which aggregation is to be performed, there you have to use “$” symbol before the field name and it tells that data should be taken of the field, and keep in mind that when you are using double group by aliasing of columns must be done.


    Scenario 1:-

    For this scenario i have a collection name employee which is somehow resembles emp table of oracle.


    $group operator is similar as group by clause in oracle. As, we already know that $project is used for selecting the columns from the collection. _id operator is used to specify the column on which group by operation is to be performed (such as group by deptno). count is a variable name that is used to display the values generated by $sum operator for each group. $sum is used to add a value specified in the clause within a group for each member. In my example i have used 1 so, in this case 1 will addded for each member within a group. If i have taken 2 instead of 1 then for each member in a group 2 is added.Here 1 behaves as count(empno) function in ORACLE.

    Scenario 2:-
    In this scenario same employee collection is used and $sort operator is used for sorting the result. Now, 
    If you want to perform some operations on the data generated within aggregation function then it should be performed to the next level in which it is produced. Such as count data is produced with in the group operator so to perform some functions onto it you must perform onto the next level in which it is produced. In this scenario i want to sort the result on the basis of count generated. That’s why $sort operator was applied onto the next level.
    Scenario 3:-
    Same employee collection is used in this case 
    1) Firstly $skip operator is used to skip the top rows generated. 
    2) Secondly $limit operator is used to limit the number of rows to be processed further.

    Scenario 4:-

    This Scenario helps you to understand how a double group by works in MongoDB. Like in Oracle where we do as group by deptno,jobid. Now to achieve a similar kind of functionality in MobgoDB it should be done as-
    Note:- In this case i have sorted the data on the basis of deptno which is a sub-document of _id field. So in this case we have to use “_id.deptno” field name to sort the final data. Nested documents can be accessed using . operator and “” quotes were also necessary.(-) negative symbol signifies that data should be sorted in descending order only.
    Scenario 5:-
    This scenario considered the usage of $unwind operator. $unwind operator is only used when you have array as a data type and $unwind operator is used to separate the values present in the array. For each element in the array a separate document is created with the same object_Id.
    collection formed:-


    Output Received After the query:-
    Scenario 6:-
    In this scenario i ll show you the usage of $substr operator to be used in mongoDB. Like in other technologies, here also $substr is used for data extraction from a particular selected field, but currently it is only supported by String Fileds.
    $substr
    So, by writing this type of query one can extract data from the given selected field and later on group by operation can also be performed onto it. The first integer inside the syntax specifies the location to start from the string and the next integer tells how many characters you want from the string to be extracted. By writing this query and later on by applying $group operator you can identify and count the number of people who have same aliases.




    Scenario 7:-

    In this scenario we will see the usage of $subtract operator. This operator provides functionality of deleting one field from the other.
    $subtract

    This will return you the difference of two numbers. Second number is subtracted from the first one.
        
    Similarly, you can work upon $sum operator.

    $strcasecmp:-This operator is used to compare the two given string in the document if the length of the first string is greater then that of second string then the result is positive, and if the length of the second string is greater then that of first string then the result is in negative. If both the string are same then return output is 0.    
    Continue Reading
    PageLines