Informatica PowerCenter Notes

hi everyone welcome to informatica power center complete tutorial session in this session we are going to learn complete informatica power center with practical sessions so please watch the session and practice it without skipping it if you haven&#39;t subscribed our channel please subscribe the channel and click on the bell icon so that you will be getting notified whenever i&#39;m uploading the videos if you have any doubts any clarifications while watching the session please add your doubts and clarification in the comments i will respond to it without wasting your time let us move on the agenda this is the agenda for today&#39;s session we are going to learn about what is business intelligence what is data values and what is data lake so what what is the difference between these two and what is etl extract transform unload and what is informatica so once we have seen the basic then we will go with architecture of informatica and informatica different client components and we are going to learn about informatica different load types flat file table to table all the different types of load and the different transformations this is very very important so this different transformations once we have completed we are going to see all the practical sessions then the most importantly slowly changing dimensions and the performance tuning also [Music] so what is business intelligence business intelligence is nothing but each and every company at the end of the day they want to track their business how the business is going on okay at the end of the day they want to track their business you can take any any uh organizations like banking insurance automobile industries health care right so any industries even you can take any software companies also any manufacturing companies any small small companies also at the end of the day they want to track their business how the bad business is going up see this business intelligence tool is meant for business people like a manager senior manager director vp&#39;s and see even sometimes ceo so what is their job so each and every day they will analyze the business okay how the business is going on so which product is moving fast which product is not moving fast if i&#39;m going to increase some product price so what could be the impact on the customer&#39;s feedback or so what how the customers is liking the product uh so which product they are liking which product they are not liking okay which product is having fast movement which product is not moving fast say sales and profit all this revenue and whatever we are doing it on the customer how many customers are there all over the globe so just we can analyze it everything like every customer every organization at the end of the day they want to analyze the business if you take online food delivery app okay how this app will be how they will analyze it right see at the end of the day okay how many customers today have they have ordered how many delivery partners are delivered by on time okay how many products 90 percentage 99 percentage of the delivery has been uh delivered within the time we have given okay out of which how many of them priced some concerns something like that right see a lot of analysis they will do they will improve their business okay this is what the dispute we are getting it how can we improve it so what is the sales and everything they will go for it on the end of the day they will make their reports right settlement reports lot of they need to set amount settle the amount to the vendors and the merchants different merchants will be the right hotels and everything they need to settle that after settlement they will analyze it okay how many of them are going for cash how many of them are going for online okay like cash on delivery and then credit card debit card paytm so different kind of analysis they will do then they will go for okay these customers are these many customers are the regional wise customers if i have all of the india business then which region we are having huge customers okay transaction sales profit and everything they will analyze it right then only they can they can improve their business in next day next quarter next year right they can project it so this is what the business intelligence see if you are the java developer you are developing application for the end users i&#39;m going to use that then you are developing it but business intelligent developer you are the etl developer informatica developer are like a tableau or power bi developer we are working for the business people business people you know what is done by artificial intelligence right so artificial agents we are we are trying to make the mission to think and then take addition business intelligence is nothing but we are giving the data to the customer that is what business people then they will take addition accordingly if you look at here in 1980s we used to store the data in the database in the database we used to store the data in the database for business transactions at the same time we will pull the report from the database itself on the same database itself what will happen if you consider any banking any banking whatever we do atm transaction whatever we are going to the bank and then we are doing the transaction or online transaction or gateway transactions whatever the transaction we do all the transaction will happen will store in the database right so it will store in the database at the same time banking manager that particular regional manager branch manager they want to know in our branch what is the today sales what is this today&#39;s deposit so based on their business they will analyze it right what is the today&#39;s deposit and with travel how much is the loans personal loan journal loans different categories they will analyze it what is the amount we have taken today and then what is a current amount we have see all this analysis we will do so that&#39;s what report okay that is what report this is a transaction see if i&#39;m going to do all this in one single database right if the data volume is very high this particular system will get impacted impacted means you are pulling the data for reports at the same time you are storing the data on the same database then what will happen this particular database will get impacted impacted in the sense you will feel very slowness some network slowness right same way you also will feel some slowness so that&#39;s what they have come up with some solution called okay don&#39;t take any reports from this database we will create one another database okay we will create separate database for reporting purpose for reporting purpose alone for analysis we will create it nowadays we have the data lake and all right see the same concept okay data lake and it&#39;s the same concept for the business analytics we will have the data on the separate database okay whatever the data we are collecting here at the end of the day we will move the data in 1997 at the end of the day only we will move so one day once we will move the data from here to here so this database we will call it as oltp system oltp so what is my olpp oltp is nothing but your online transaction processing online transaction processing your business will be there from different places you&#39;ll be having the different places different applications your data might be coming from sap data might be coming from a sales force and oracle sql server db to different different applications right different different places also see all this data whatever the databases we have say for example here on database here on source applications here another source here under source these aren&#39;t for business day-to-day business your payroll information will be there in the sap right general results general ledger crm all this related data will be available in the oltp this is for transaction say for an example if you take irctc whatever the ticket we are booking right so all the ticket booking and everything will be happening in this ol tp only okay then well what is ol t oil ap oil ap is nothing but this is what our ol ap oil ap is nothing but your online analytical processing whenever we want to do analysis i&#39;ll go for the online analytical processing you do all the analytical okay so analytical process in this olap system don&#39;t do on oltp see this is what the oil ap do analysis on the olap system olap means it is a data barriers what is warehouse meaning vargos is nothing but it&#39;s a place where we will store all the historical things right same way here we used to have the data values initially we used to have the data variables nowadays we are moving towards on the data lake see data lake is nothing but more than the volume of data warehouse this data vargos will handle only structured data structured data in the sense it will have the data only in the table format like how we have the data and the excel sheet right the same way you&#39;ll have only stable format we will have the data but in data lake we may have the data see not only one single table we have a lot of tables okay thousands of tables will be there not thousands even 10 000 more than that also but data will be stored in tables like a structured one this columns and rows but here we have the structured semi-structured unstructured data we can have it on the data lake all our facebook comments on all our data right ola ubers wiki zomato see all this youtube and everything like to store the data here now we are going with this data lake only that&#39;s what the spark will do all this with the processing okay see we we need to store the date this is what our business business we are not going to touch this for analysis if i want to analyze my business you should not touch this one you have to analyze all the analysis here only data here only okay so i have to move the data from here to here right i have to copy the data from here to here so every day i have to copy see this is what you will have and this offer oltp to olap data lake as i told data lake will have structured okay it will have all the structured data so that&#39;s what the data lake will have you have the azure sql right you have aws also bigquery uh sorry gcp we had a big query it will have structured semi-structured okay so i will say short form semi-structure and then unstructured data also that&#39;s what it will have all this not only is construction it will have all the data okay that&#39;s what your data lake can handle it all the type of data okay so this is called your oa lapis oil ap is nothing but your data lake or data minerals we have to move the data from here to here right we are the heterogeneous sources this is called heterogeneous sources as i told you might be having the applications from uh we may have applications from sap right we have we have data from sap we have the data from salesforce okay salesforce and we might be getting the data from a file file system flat file and we might be getting the data from your oracle your oracle and your sql server see microsoft sql okay ms sql see all this microsoft sql server or you might be having the data from db2 see other data sources also any different data sources we can get the data this is called heterogeneous source system we will take the data from here okay we&#39;ll extract the data from here and we have to load the data into this data of arrows now we are seeing only structured data we are not going for any unstructured data in our informatica so we are having only structured data that&#39;s what we are going for the tool called informatica power center so this tool as i told yesterday this tool will extract the data from source it will extract the data from source transform the data load the data into the data barriers on a periodic time okay scheduled timing this process we will call it as etl process extraction transformation loading it&#39;s like a postman okay extract transform load the data into data values okay load the data into data values this is the etl process okay elt process what is eld process normally in data lake we will go for elt process extract the data from here load the data into data lake after loading it we will go for the processing that&#39;s what extraction ingestion and then processing it that&#39;s the elt process so we do have different processing type one is the processing we have batch processing okay batch processing what is batch processing right batch processing means see we will be having a scheduled timing like one hour once half an hour once one day once weekly monthly jobs will be running periodic time whatever the transaction happened here periodically it will move the data from here to here for the analysis purpose for an example now the time is 8 30 one hour once your job is running etl process it should have executed by eight o&#39;clock right eight o&#39;clock but now date that time is 8 30 if you go and check the data in our analytics purpose up to eight o&#39;clock only data has been refreshed so when the now whatever the transaction is happening right right now the timing it will get reflected at the nine o&#39;clock only okay nine o&#39;clock only so why we have scheduled this etl process by one hour once that&#39;s what etl see you you are doing some cancellation and all you are doing some some some transaction right in the online process on that sometimes you they will say like it will get reflected after four hours after one hour or uh something like it will get reflected after 12 hours or 24 hours or three days something right see it&#39;s all about etl process okay it will move the data from multiple applications it will make an entry it will get some entry and then it will go to the other one that&#39;s what it will go for this is called batch processing we do have live processing we have something like you have iot devices right iot internet of things iot devices or you may have having the data from sensors or facebook comments and like this data sources you have the kafka it will read the data it will have the all the logs it will pitch and then we have a lot of tools are there by using that we will have the spark engine then we will load it into the data lake we have other tools also this is called live processing live processing means we will be seeing the near real time data near real time means say 8 35 you are seeing 835 data okay that&#39;s called live crushing it will take live data from here so it&#39;s all your iot devices and everything okay it will take the data from here this is called iotr on the devices live processing it will take the data and then here the spark engine or kafka will be pulling the data and then here it will load the data here you will be processing it ingestion this is called ingest will say like it&#39;s a extract ingest and then you will be processing it here we will go for the process okay this is called your data lake processing engine that&#39;s what your aws your azure okay your gcp all this will do this is called live process we have batch process also in our data lake under that&#39;s what it will have all this structure semi structure and structure also this data of arrows normally we used to have oracle data values or the other teradata heard about this teradata terminal right so you are the this is for the data variables okay we do have something like this is what your teradata teradata is the first database to handle the terabytes of data so you might be having oracle article also variables and you might be having sql server microsoft sql server also so these are all some of the data barriers previously we used to have netizer you are the natives are right all this so in data lake nowadays if you come to this second part you have the the microsoft azure right that the azure we have so as you and you have the amazon aws right we can store for s3 bucket and all but the data lake and we have something like a gcp google cloud platform and we have we can have this snowflake see these are all the uh some data lake we have in our real time to store our data and in addition to the storage it will do all the processing and machine learning algorithm all this it will do that&#39;s a different part but we will use this one for our business uh process okay this is what we&#39;ll go for the batch processing and live processing all this okay i need to have the tool right here i need to have a tool which tool i can use there are different tools are there that&#39;s what our informatica power center will do one of the tool is called informatica power center okay one of the etl tool is called informatica or center see informatica is the company name informatica power center is the tool tool name see the informatica power center is used for data integration it&#39;s integrate the data from here to here it will integrate data integration rule informatica will not store any data in order to load the data from here to here we are going with that&#39;s what we are see if you look at iictc and all in that class they will not allow you to take the pnr status right why because this is oltp if you if you go and hit oltp the that collage that collapse is like more number of users will be accessing it okay if you are going to check in the something like reports on pnr status and all it&#39;s again overhead to the oltp system it may impact on the particle booking enough so that&#39;s why they are stopping it right same way in business if the business hours is happening business is happening here if you go and hit this database for this analysis purpose it will get impacted that is why we have so you can ask me okay here informatica also is getting this database right and then what will happen that is what we are not going to hit all the data we are going to get only up on our data on our data our one day data maximum and also we will have some replication server we will not directly hit this old pa we will have some replication server here some server it will replicate last 10 days of data here okay whatever the data here like a mirror server we will use that mirror server it will mirror the data whatever the data is available last 10 days of data only it will be mirrored we can have last one year of data but only 10 days of data will be there in this layer we will hit this layer only not this layer this layer is for database for business purpose okay clear on this that&#39;s what we will have the databases and everything okay so you have loaded here after that from here we will extract the data for our reporting purpose reporting in the sense we do have lot of reporting tools are there so what are the different reporting tools are there the market we are the tableau from salesforce then we have microsoft power va so we have power ba power business and legends then we have uh click view from click tech we have the click view or click sends so like this we have a lot of tools are there from looker from gcp we are the looker gcp in the sense you are google we have a looker and we are the quick insight right so from aws we are the quick insight see all these tools are very much used for this reporting purpose okay reporting purpose how we are the informatica power center right same way we do have other tools i will show you okay what are what is these tools i can show the data to the customer in terms of chart and visualization chart by means of chart it&#39;s very easy to understand them okay then where this machine learning algorithm will come into picture artificial religion will come into picture see here we are loading the data right we need to feed again data to our customer for the better user experiences so if you go to amazon they&#39;ll be giving you the suggestions right okay you buy this product based on your search history they&#39;re giving you the suggestions all this right so how they are giving us against your customer id they used to collect all the data they will process it based on that they will processing it again they are feeding back to the source system okay for the recommendations youtube recommendations amazon see a lot of a lot of application now they started with like a bestseller review comments whoever is getting good review then it will be on the top right see lot of filter condition lot of analysis they are giving that right see all this will be we are analyzing it all the data huge volume of data again we are feeding back to the oltp system for our business users okay that front end they will collect the data from here they will do use it this is what machine learning algorithm artificial religions algorithm will be applied on the database and then data lake and then they are feeding back to the source system again full stack developer under they will use it they will use this data lake as the source and then get the data they will develop it see these are all the data by using data lake life cycle here if you are going for data lake uh data center and all you can see this is what the data center so look likes it&#39;s again huge volume it&#39;s not only seeing uh small volume it&#39;s like a huge volume only google cloud we are these are all the data centers data lake look at here how much is the data lake it&#39;s alibaba you know right chinese china alibaba it&#39;s like a data center here this is what they will have the data lake hash lake cooled data center right so google also they have the data lake and everything lake data center it will it is used to store the massive data that&#39;s what we have the data center look at here this architecture even we can get the data from ingestion i told right ingestion we&#39;re tracking all this we will collect even from data values also we will collect okay storage and then data catalog data breaks so that&#39;s what it will move the data to be used in real time these are all the data lake in real time okay data centers we have huge volleyball hundred data values again it&#39;s huge volume only but when compared to data lake it&#39;s very limited one okay if you know this etl pipeline very well informatica and all very well learning this concept is very easy machine learning all this etl machine learning is very easy that&#39;s what extraction transformational loading machine learning also we will do deep learning machine learning all this related to our retail process only once we are the etl do that etl data sources etl data vargos and business intelligence look at here how the architecture will be right as i told we may get the data from your online resources right your online shopping your mobile your cloud so all this you are collecting the data flat file databases iot devices and everything this data source you do the etl and load it into the data values are elt i told right elt also anal analytics query and reporting your other business intelligence business intelligence means your tableau power bi all your mobile and everything that&#39;s what you&#39;ll be having all this see logs files and media unstructured unstructured business custom application is a structured one so you are taking that ingestion storing and preparation and training and then you are doing that all the modeling and everything then you are going to the power bi or web application or front end tool or everything that&#39;s what the pipeline of azure and everything okay ingest ingest means extraction okay ingest transform load into a storage then preparation and training preparation means machine learning and all this we will train that and then we will give the data to the applications front-end applications but the better user experience on that okay so this is what the etl process will do i hope you are clear on this etl process so that is what i have explained here so business intelligence is nothing but each and every company at the end of the day they want to track their business so that&#39;s what the business intelligence so each and every this is the business intelligence tool for mainly used for the business people the business people like manager senior manager directors vps and even for ceo also so they they used to start their business day-to-day activities by analyzing the data so for this particular business analytics as a etl developer they will load the data into data values or data mark and then as a report writer they will present the data in the form of data visualization tool in the form of dashboard or scorecard or different reports the business people will analyze the data and then they they will take addition accordingly okay this is what we can go for this is the process and this is what the sales we we got it and this is the profit this is the margin so a lot of analysis they will do on the data they will take addition accordingly so this is what the business intelligence as a etl developer we should know all this concept very clearly so what is etl as i have mentioned etl is nothing but extract transform and load so this is what the etl process one more time i will explain so the left hand side you are seeing that heterogeneous source systems so this is what the heterogeneous source system we have so like we can get the data from oracle sql server db2 uh even we can get the data from even we can get the data from the social media and then we can get the data from any live sensors and all the data we use to collect extract the data transform the data and blow the data into a data variousing layer or data lay so this process we will call it as batch processing so batch processing is nothing but it will be a scheduled run so every day it will be scheduled run it will be so even for half an hour once one hour once the data will be pulled from the source system and then it will be loaded into a different layers like staging layer and multiple layers intermediate layers multiple layers or players and data mark so data mark is nothing but a subset of our data values so this subset is nothing but say for an example i am working for merchant reporting then i&#39;ll be having merchant data mart and then supplier data mark finance data mark some different different data marks will be there for the particular subject area so from this so for this layer this is the source and this is the target staging layer and then staging is nothing but it&#39;s a landing zone for our data so even we have the data from heterogeneous so system we will land the data into a staging layer from the staging layer we will move different work layers we will apply a lot of business logic here for this layer so this is the source and this is the target again for this layer this is the source and this this is a target the data will be moved like this one so source and target so all these are etl process only this is what data will be moved so this is like oltp system this is like olap system so we used to have lot of different etl tools are there in the market so one of the widely used etl tool is informatica power center then we are then the cloud system informatica intelligent cloud services microsoft ssis sql server integration service oracle data integrator different detail tools are there in the market we should know all this concept very clearly whenever we are going for an interview or we are going for a project so definitely this etl process will be taken place and we used to have the live processing or streaming so nowadays we are working on the data lake so data lake is nothing but the more than the volume of data values it can handle structured semi-structured unstructured data also unstructured data is nothing but our audio file video file how we are getting the data in the social media right so that the social media content mostly are unstructured data so in order to process the unstructured data that continuously we are getting the data from our source system this is called live processing a streaming like how the water is flowing in the pipeline right the same way the data will be flowing 24 7 from the source systems like our iot devices like mobile devices or sensors we might be getting the data continuously and then we are loading the data into a data lake from the data lake we will process the data again you can apply some machine learning algorithm or you can apply some artificial intelligence algorithm here and again feeding back to the data to a source systems for the something like uh recommendations our best seller something we are doing it some news feed so a lot of analysis we are doing on the data and then we are giving the data to a source system so this is what the etl process will be will be there in our day-to-day activities so oltp is nothing but online transaction processing to run the business we used to have the oltp in order to analyze the business i need to have the olap system so oil ap is used too far mainly for the business person to analyze the business they will go for the in order to analyze the business i need to have the data values or data lake that&#39;s what we can do lot of analysis on the data after analyzing it i&#39;ll be taking the decision accordingly so who is involved in etl process so whenever we are going for a real-time project so first we will be getting the data from a data analyst or the business analyst or the product owner or the business owner or from application owner so they will be creating the requirements so we will be getting the requirements from our business analyst our business owners and how the source to target and what are the different business use cases they will be giving us so based on that we are to the etl developers they will be doing the etl development right so they will be moving the data from source to target we have to validate the data so how the etl tester move the data whether it is accurate or not so what are the defects are there so we have to raise the defect and then we have to assign it to the etl developer and the etl developer need to fix it so this is what the life cycle will be going off so once we give the sign off then only it will be moved to the other environment data will be moved to the other environment the data architect our data modeler nothing but who builds the data store for us so it&#39;s a data big data lake our data values are data marked they&#39;re only going to handle it so if i&#39;m going to do any business so what are the different tables it should get affected and what all the columns should be maintained the architect will be maintaining it as atl developer they&#39;ll be moving the data from source to target as a etl tester we are to test whether they have moved all the data correctly they have implemented all the logic in our etl process our business analyst team they&#39;ll be analyzing this particular reports only so they&#39;ll be coming up some some use cases okay uh say they want to know if they are we are doing the some transactions transaction is getting declined okay declined so by the particular customer or by the system automatically some reasons okay so we have to analyze why that most of the transaction is getting declined the business analyst team they&#39;re saying that okay 30 percentage of the transaction is happening just getting declined in the last stage or some stages it&#39;s getting declined why it is getting declined so it&#39;s a due to the card issue or the network issue or by by the for the person itself they are declining it okay a lot of reasons will be there right why they are declining it otp issue so lot of analysis they will do see all the analysis they will do and then based on that to arrest this they will be giving us some suggestions okay if there is a otp issue increase the server capability to deliver the otp very quickly okay if you are delay if you are delaying the otp like more than three seconds to four seconds so there is a lot of possibility to decline that particular product the last stage so whenever they are they&#39;re ready for the transaction you deliver it to otp quickly and read otp and then fill it complete the transaction within two seconds or three seconds that&#39;s what okay that&#39;s what you are completing the transaction nowadays they are having the e-wallet right so keep it to the ballot inside the particular uh the particular site itself load some monies before that so you can just one click take the date take the money that&#39;s all okay so it&#39;s a com completing the transaction very quickly then the customer so most successful transaction will be getting it from the customer right so that&#39;s what the business development team they are analyzing it and then they will be giving suggestions to the business team okay if you are going to do this we can improve the business like this then it will be implemented on the otp side okay the enhancement will be implemented on the otp site server enhancement all this so the volt this aggregating the data is nothing but we are integrating data from oltp to olap so database data marked data values what is data marked the data vargas we know what is data various the data variables so the data marked is nothing but a subset of your data various subset of your data values it&#39;s again logical division logical division of this data variables say for an example i want to send the merchant report i&#39;ll be having merchant data mark okay customer data mark some supplier data mod all this data mark will be there different different data mark based on the access and privileges we&#39;ll be having the different data mark so from the data mark we will take the reports if i&#39;m working for merchant report only merchant data mart i will have access other data mod i will not have access okay so it will have some pa informations for the secured information that&#39;s why for the security purpose they are splitting the data values into data mark based on your business area you take the data from this particular small area okay small subject so that&#39;s what data mark will be data mark is nothing but subset of our data values okay okay so once you have load we are aggregated the data into the data values once we have aggregated then we will present the data in the form of data visualization like a reporting table power bi see all the reporting tool we will create the reports so if you are giving the data in this format it is very easy to them right business people it is very easy to them to analyze it right then they will take addition accordingly okay so this is seven six days of five days of business why this business is coming down our last five years last two years to go with some business class are something like that different due to covet we are having a lot of uh transactions so that&#39;s what they will analyze it they will take addition accordingly okay how to improve the business so how to improve the business in this particular area so we have developed these many products out of this this is the product is moving fast this is the product is not moving we do have the stocks in this particular store we have to move the stocks to another store for the supply and demand so like this lot of analysis they will do on the particular business and then they will take addition it is not only one particular business team we do have different teams will be there business development team they will be doing it all the analysis based on their area and then they will take addition accordingly they will be feeding back to the source system we will move the data okay so in order to change anything on the source they need to get a lot of approvals from the business architect team okay so this is what data bar hosting architecture whatever i have explained so far this is what the databar goes architecture we have the source here erp enterprise resource planning are all other social media networking and everything e-commerce site and everything structured data unstructured data i told right it&#39;s a real time data we are pulling our batch data we are pulling from the source the data integration layer all the data profiling all the data cleansing we will do it by using some idq tool or some layers so from here from here we will consume the data to the to the data we are using layer like a staging layer and one staging layer completed we will go for different layers enterprise data vargos this is called edw from here the data will be moved to data marked from here it will be moving to the all the bi applications like you have the tableau power bi all this mobile applications and then data will be moved to some front end tool also on demand services also so this is what the data are using architecture see data various the ctl pipeline will be there okay even if you are going for new cloud technologies definitely this data warehousing architecture will be there so we can move on to the cloud technologies no issues still lot of companies are using informatica power center for their etl process so if you take this is one of the analytics this is one of the analytics reporting side say i&#39;m giving the data so this is one of the channel analytics say for example for that for whoever is having the channel we are giving the some analysis here okay these many subscribers for last 28 days 41 percentage increment then this many views this money watch time okay then this is the day by day how many uploads you have done and what is the views here so it&#39;s very easy to understand right for the channel holders right so how much is the overview the viewers and what is the interest viewer something like that see all this you are giving filter here last 28 days last one day last one month right our last this particular month current month you can go for last quarter last three months last year or from the beginning so whatever it may be they are giving the line chart here bar chart see the numbers they are say giving so this is called data visualization the reporting okay they are using different tools to provide the data it&#39;s a near real-time data see here it&#39;s the latest activity last 48 hours they are giving say every one hour the data is changing okay now updating live right so this is called live data update so this number will change often okay four eight three seven four eight three two something this number will change okay up front it will change this is called streaming the data and then you are showing this one will take the streaming the data but now here we are seeing this one right last 28 days this is not streaming the data this is like your data warehouse data only aggregations this is live data this is aggregation data so in a single page they will go for the scheduled data also on live data also that&#39;s what they will be showing them this is what we&#39;ll go for the data okay so this is the architecture of data vargos you can see here this is the functional architecture of our data values here we have the crm okay customer relationship management on inventory payroll sales general ledger production all the data we will be collecting from source we&#39;ll be moving the data into a data vargos from the vargos we&#39;ll be taking different metrics okay so how the customer profitability what is this we are we are having different suppliers how they are performing it okay productivity and strategic performance closed loop marketing product profitability so in real time environment we do have different different metrics all the different different metrics will be collecting from the data various data only okay a lot of aggregations will be running the data so you&#39;ll be having last three months whenever we are going for any quota results right the company is going for quota results they&#39;ll be taking all the three months data right all the three months data so if i&#39;m going to take aggregation of all the three months data if i&#39;m going to do that particular calculation the source system it will get impacted will not get one consolidated data so payroll will be there in sap okay payroll will be there in sap some different different systems will be there oracle sap different different system will be there it is not a consolidated data but here single place single unit it will give you all our historical data last 10 years i want to know last how last five years we are performing based on that we have to speak in the board meeting and all so that we have to take from the data variable okay this is what will be data various environment okay so what is data values as i have explained it&#39;s a subject oriented it&#39;s nothing but your data mark subject oriented like customer supplier product sales one subject oriented integrated we are integrating the data from source system to data various integrated time variant all our datas are historical perspective like we&#39;ll be having information like from day one to till date we&#39;ll be having all the time variant data and non-volatile non-volatile in the sense like we will never ever overwrite or delete the data in the data barriers remember we will never ever overwrite or delete the data in the data values we will just load the data and we will keep the data for historical purpose okay backup we will keep it so if any data change is happening the source system we will allow to change it if you are if you are going to a band site and you are changing your address the bank oltp site will allow them to update your address but in oil ap you will have your previous address and this address also will be inserted newly the new address will be marked as active address the previous address will be marked as inactive address so if you take amazon so they will be having all your addresses right whenever you are adding new address they&#39;ll be keeping and we are going for any uh checkout then they will be giving all suggestion of your address okay which address you want to take these are all business tricks so you know every time you are going to type your address it will take lot of time then customer will feel that very board right so that&#39;s why they are already taken the addresses and everything they are giving right everything they are making very easy for the customers to improve their business that&#39;s what non-volatile collection of data to support management decision making process to support management decision making process we are using this data vargos for the business intelligence i hope you are now you are clear what is business intelligence business intelligence is nothing but at the end of the day i have to make my business in a profitable way okay that&#39;s what the business intelligence so i hope you are clear this is the again the modern trend values definition integrated architecture and periodic copying data from various heterogeneous sources both inside and outside the enterprise into a centralized repository optimized for reporting and analytical processing this is what we are doing for the databasing okay data lake also this is what okay why the companies deploying the data values are data lake why because operational system operational system is nothing but your oltp side ostp are being impacted because of reporting request so if your customer is asking last three months of data for the reporting purpose then if you go and hit this operational system so if you&#39;re the backend huge volume of data it will consume right see so amazon per per second order is 120 hearts per second worldwide 120 orders per second like you could think about the volume the volume how much volume would be generated and then one day volume one week volume one month volume one quarter volume how many records will be created one order so how they are handling all the data right see logs and everything how they are handling it so in that particular system if you go and hit this oltp for the reporting purpose it will get impacted right so that&#39;s why and also it will not give the oltp cannot have all the historical data we cannot keep all the historical data we will have all the data mining operation data variables operation in the data lake our databar goes on so we will not get any consistent data so that&#39;s why and also it&#39;s very expensive to maintain this oltp system okay ap side our data lake side we have commodity hardware we can have the nowadays we have the commodity hardware site or data lake side so that&#39;s why we are going for the data values or data lake so these two members are called as fathers of data warehousing in mod and ralph kimball so inbound only the first person has suggested so instead of having oltp we can have one more system called oil ap one more system called oil ap where we will load all our historical data we will analyze it that&#39;s what he has suggested first during 1990s 80s itself he has suggested he has published international journal papers okay from that only they have created data virgos see his suggestion is top down approach see what is top down approach right so first we will we will create see this is what our oltp oltp from the oltp we will take the data and into a olap system that&#39;s what olap is nothing but our data variables we will load the data into data variables even data values different layers will be there that i will explain later data variables from the vargos we will move the data to data mark okay after creating the data vargos completely then based on the subject you are creating the data mark and then you are loading the data in this data values to data data mark this process we will call this top down approach this is a data modeling okay data modeler will do it our data architecting our data modeling team they will do it they will take the data from source okay then they will load it this is one on time automation automation process see informatica is fully completely automated okay we will schedule it after developing it we will schedule it so once it has been scheduled the data will be moved from source system to target system okay this is what it will be moved here the data mark will be there this is what the data will be more clear once after creating the data variables if you create data mark that will be called as top down approach after 10 years ralph tumble has come up okay so it is taking lot of time to create the data where which is taking a lot of time by the time so what we can do here we are the source system based on the subject you create different different data mark okay initially we will create a different different data mark load the data into a data market directly say for an example i want to have the reports for merchant i will create a version data mark only whatever the data i need to have this person i can take it from here from from this source and then i will load it into this data mark okay so it will have only merchant related data so here customer related data supplier deleted data that&#39;s what we can load it it&#39;s one particular subject area after that if you need you can move the data to data barrows so this process we will call it as bottom-up approach bottom-up approach okay in this process there is some here also some advantage disadvantage so based on the company&#39;s need either they will go for top-down approach or bottom-up approach okay it&#39;s a data modeling team they will decide it or they will go for hybrid approach so what is hybrid approach oltp to data mark data mark to data vargos data var goes to again datum data mark okay so this process you will call this hybrid approach so this is what we will do it in our real time projects okay who will do it data modeling team will do it they will do it all the tables data modeling and everything as a etl developer we have to load the data into this particular model so if i&#39;m doing one business okay if i&#39;m canceling a product how many tables how many tables it should affect okay so what are the different data i have to capture so that will be decided by the data modeling team as a etl team we have to take the data from source we have to load it into the data variables based on the business needs this is what we will do okay the inbound model as i have mentioned this is our source system so our heterogeneous source system we will do etl process load the data into a data values see here directly we are loading but in our real time it will be there multiple stages okay stage by stage data will be moved to the data values we&#39;ll be doing lot of etl logic and then the data will be moved to data values so once the data has been date moved into data vargos then again it will be moved on the data marked so data mark is nothing but subset of our data values okay how we have in our system so 1tb of hard disk we are splitting the 1tb into c drive e drive right the logical division the same way here also the other logical division of the data barriers is called data mark so in real time project if you are working for one particular data mart so merchant data mart customer data mart supplier data mark okay different different data mart will be there if you are the tablet developer or power bi developer or any other developer report developer so definitely you&#39;ll be having this particular data mart alone access okay you&#39;ll be having access to this particular data mark so other data mart you need any access to this table you have to get the approval from the customer okay from the customer and then you have to inform to the dba dba will provide you the access okay this is what in real time you have access and privileges okay so top-down approach bottom-up approach first we will load the data into data mark here also we can take from here here also we can take data can be so you you can take the data from this data mark our consolidated data will be available in data overloads here also we can take it for reporting so anywhere you can take so this is for the bottom-up approach okay this is for the bottom-up approach so we are splitting the data into a data mart based on business okay based on business not for the uh not with any schema so i told right it&#39;s a merchant data mart so merchant related data will be loaded into the merchant data mark okay merchant related data into a merchant data mark then customer related say you are working for customer reports then customer related data will be moved to customer reports see data marked is nothing but again it&#39;s a subset of our data values only it&#39;s a logical division it&#39;s a logical division okay these are all the logical division we are logically it has been splitted but it&#39;s a single physical hardware only okay now we have this is what next one is like we have next one we have data marked and data values and top down bottom of approach so what is the difference between top down and bottom up approach we have in one model it&#39;s a top-down approach gimbal model it&#39;s a bottom-up approach it&#39;s a more time consuming okay if you to build the data values it will consume more time and the maintenance is very easy maintenance and subsequent project development cost is lower but initial cost is very higher okay so so first we have to build data values then we have to take the data marked it&#39;s a longer startup time okay it&#39;s for enterprise wide so for the entire enterprise side we&#39;ll go for the data values so this is what individual business area so one business area we will go for i told rights one subject like merchant customer supplier right so all these are gimbal model so it takes lesser time you can create a data mark quickly and then you can take the data but after that it will take some no initial cost but after that the maintenance will be high okay this is what the difference between the inbound model and gimbal model now we have both both model okay so hybrid model so from source do the etl load it into a data mart so from the data map will be loaded into the data vargos okay then it will be moved to the data mart again again data mark so from here also you can take the data write the user from here also you can take the data from the user okay this is what it will be hybrid approach so who will do all this in real time so we are as a util developer we are not going to do data modeling team they will do it okay data modeling team some separate teams will be there so we do have two different tables in informatica that&#39;s what data various so this table is very very important so whenever you are going for enterprise data by those so definitely you are looking for only two types of table either it should be a dimension table or it should be a fact table so what is diamond and fact table if you go to the data various right data values i want to build a report so how we will load the data say for an example if i have employees data i will load all my employees data in the employees table okay all my customers data loaded into the customer table customer data so all the customer id and customers data employee employee data product information and location information see all this information will be loaded into a different different tables so these are all date table okay location store table different store so why we are loading all the data into a different different table right at the end of the day i want to know what is the store wise sales what is the product to sales what is the customer ratio sales regional is country ways right different different dimensions i want to analyze the data then i&#39;m loading all the dimension into different different tables then i will be having a fact table you&#39;ll be having a centralized fact table here it will have all the transactional data okay whenever we are doing the business see first we are capturing okay this this customer purchased this particular product okay on this particular date okay the from this particular from this particular employee from this particular store particular region particular country see all this particular store everything i will capture so this mini product he has purchased one product cost is this much and total price is this much so each and every transaction will have the data so this data we will call it as fact table this is the fact table okay it&#39;s the centralized fact table fact table will have the transactional data so always factor will have transactional data all the keys from this table will be connected the key values from this table so this is foreign key primary relationship so you can watch my sql session so today i will tell you how to install the software and how to practice our sql that i will tell you so please start working on it so this is the fact table this is a dimension type so all these are dimension table dimension table will have always a primary key okay remember dimension table will have a primary key and fact table will have a foreign key okay fact ff you remember fact table will have a foreign key so what is the foreign key these are all the foreign key then measurable values these are all the measurable values right so measures you&#39;ll be having it non-measurable attributes and primary key so what is primary key so i will tell you now so this is the data model we will create so who will load this table as an etl developer we have to load it okay as an etl developer we have to load it so how to do it so i will tell you how these tables are related if you look at here whenever we are doing transactions in our real time okay whenever you do the transaction now real time if you look at here i want to start a company so that i will have all the reports so how we will have the reports now real time right so i will load all the product related table in the product related details in the product table so each and every product which is available in my company will have a product id separate product id okay each and every product will have separate product id product descriptions all the related to product description this is called primary key this is called different dimensions okay attributes attributes of the particular dimension table if you look at here dimension table will have a primary key and attributes the primary key here it will not accept any duplicate value i cannot give for this any other product i cannot do the same number okay without primary key without this key i will not have the product name on double zip so primary key should be there so separate key should be there if you are going for any company you will be having a separate employee key right employee id the same way each and every product will have the product id each and every customer will have the customer id okay so every customer has been loaded into the table we&#39;ll have the primary key the same way employees table accounts table all this table will have it okay i will have all the tables the store table locations table this is what the diamonds and table see whatever the diamonds are needed for our data model i will load all the dimension table so from there this dimension table will be loaded from source only from oltp to olap we will load but while loading the old ap we will maintain the history that i will tell you how to do it but we will get the data from ol tp100 okay oltp also they will maintain that is what er model entity relationship model we also we will use in our oltp system okay so now we have the fact table so if you are going for a particular store right you are doing any business like you are purchasing a particular product then you are you are giving that product to the particular person in employee so they will be maintaining that they will be reading the barcode of the particular product and then all these details will be captured so what are the details when you are purchasing what is the date okay which product you are purchasing what is your customer id they will take your customer id card okay or the number mobile number which employee is helping you to purchase the product okay and store id which store you are purchasing a product which regions so all this will be captured in this fact table sales table see these are all foreign key so why i am capturing so whenever i am going for the transaction i will capture that and quantity so how many quantity so two quantity has purchased unit price and the total price okay so one unit price is three thousand five hundred two unit price seven thousand all this will be captured for an example your customer is saying that okay at the end of the day how much you have today&#39;s sales so what will you do you will go for sum of total price sum of total price date key equal to today&#39;s system date will take and i want to know in today&#39;s date how much is the sales for each store so group so here we have to make date key equal to today&#39;s date and group by store id we have to make the sum of total price this is what we will maintain never reports okay this is what we will take but i do not know which store right so 7667 which is which is a store then i have to join this table this store id fat sales store id to diabetes and table store id i will join it i&#39;ll get okay so mumbai kalyan store this is what kalyan region store we have the sales so highest sales so that i will take it from here but i have to make the summation so which product is which store is which region is so everything so this is what in our now real real-time environment we will be keeping that fact table and diamonds and table how these tables are related like a star schema snowflake schema the schema the star schema is nothing but the other star schema is nothing but like we have like a star this is the fact table so you&#39;ll be having a centralized fact table this is a sales table you assume that this is a sales table this is the dimension table if you look at here this is a dimension table we are the city table so city id city description all other remaining columns also will be there employee table country table products table see all these four tables are dimension table this is the fact table this is the fact table so in this particular country this particular city this particular employee purchased this particular product so two quantity he has purchased one quantity sorry one unit price is 200 dollars okay so he has purchased two quantity and 400 dollars that&#39;s what it will be maintained so from source only we are taking the data from source only we are taking the data but we will maintain in the fact table in our olap site okay see if in fact table the outer tables are dimension table the diamonds and tables are connected with centralized fact table to get a meaningful report to get a meaningful report this is for this task comma now we are the snowflake schema see in the star schema which table we have to load first so primary key table should be loaded first so it&#39;s a master table reference table okay always we will maintain all the primary key but remember this dimension table will not change frequently so do we need to load all the country data every now and then no right so one time the country data will be loaded so always the country data same if you are going to operate new business in new country that time i will maintain one more country otherwise we will be having the same country this dimension table will not change frequently that&#39;s what it&#39;s a slowly changing dimension okay slowly changing dimension product table also so product table so we will have new products but very slowly it will be changing but fat table is having huge volume of data okay this is star schema in star schema we have to load dimension table first then centralized factor clear so this is one of the interview question they will ask so how the dimension modeling they are maintaining in your project so which table you will load first okay that&#39;s what we will ask they will go ask questions okay so now we will have snowflake schema so snowflake is different one snowflake technology is different that is data lake i have yesterday i have explained snowflakes schema is different okay snowflake schema is in data various environment in our data various environments snowflake schema is nothing but okay the snowflake schema if you have this is one dimension table right this is one dimension table this one dimension table we are splitting this dimension into multiple dimension table look at here this dimension table we are splitting into multiple dimension table so instead of having all the data in the single dimension single locations table okay we are having the data into the another table okay store id store name store territory information before that we have handled here okay after that we have moved this data to another table okay we will maintain the store territory details in another table called territory whenever needed whenever store territory needed i will have this id okay with this id and then i will maintain this i will take this report from where this particular store is located whenever needed i will go for store name yes i will go for store name with the store id whenever i need the store territory then i will join this table i will get it okay so this is a primary key table this is a primary key table that is a dimension table this is also a dimension table see one dimension table refers another dimension table based on the normalization this is called normalization what is normalization so normally say why we are going for normalization so normalization we are going for normalization to avoid the data redundancy see in real time oil ap tables are denormalized okay denominate in entity relationship model in oil oil tp side will oltp side is a normalized table olap site it&#39;s a denominated one okay our data variables is denominated so denominations is nothing but this is what a determinist but here snowflake again we will normalize it so what is normalization to avoid the data duplicate data redundancy repeatness we are going for the normalization look at here say i am keeping my employees details here i have the employee id i have all this id column right so i have the employee first name last name everything i have here right so this one i have see here i have the employee id okay so here i have first name last name all this i have so i have these details are from office address building address poc and everything is nothing but our office location so you have three different office location you assume that you are the three different office location you have one lakh employees okay if you are keeping your employee details and office details in a single table what will happen so every time you are repeating the same addresses officer three of our office addresses to all the employees correct if you have online employees the three office addresses are repeating for one lakh times here instead what we are thinking maintain this harvest header separately okay maintain this harvester separately like this only three office address right three different obvious addresses three different times you will make so in how is understable address id will be the primary key okay in employee table employee is the primary key so this should be a par in key okay this employee located in office number two three one like this you can have it right like this you can have it so how many times i maintain now now only three times so this is what i can reduce the data redundancy clear this is what i can reduce the data redundancy so this is called one dimension table i am splitting the dimension table into multiple tables this is called normalization first normal second normal third normal right so this is their normal form this is what we are maintaining so why to avoid the data redundancy right so this is what we&#39;ll go for the data returns and everything see one dimension table splitting the dimension table into another dimension table so here if you look at here here you will be having the primary key here store id will be a primary key right store id will be a primary key and same dimension table will have a foreign key if the dimension table is there in foreign key now real time that&#39;s there in the snowflake schema okay here it is a primary key here it is a primary key so what is primary key foreign key so that you have to learn from oracle only from sqr only okay so what is the two different data variables and data lake data vargos is nothing but the structure data only will be handled and the data will be processed okay processed data will be loaded into the data values in data lake the other structured semi-structure unstructured data will be handled raw data will be loaded into the data lake from there it will be processed scheme that&#39;s what schema and write schema read raw data will be loaded after that we will read the data from the storage here we will we will have commodity hardware low cost storage here it is expensive cost is expensive that&#39;s what we are moving towards on the data lake side in data values mostly the business people will use it business analyst okay product monitor all the business people will use in data lake environment all the a and ml data scientist they will use it so if i want to use a and ml concept here so we will use it in data value side okay this is what the data variables data lakeside will use agile is nothing but so less agile fixed configuration highly agile configure and reconfigure as needed right see agility is nothing but you are splitting that entire work into a small small work okay small small work and then you are you are doing parallely you are doing parallel okay so all the work you are doing parallelly so agility in the data warehousing environment you can do very less okay but here in data lake environment we can go for high agility so agile process we have right so we will be we will be loading the data and there is no need for testing very less testing is enough and no there is no failover there is no support team much needed support team there is no much uh support team in our agility enough so we will develop the code we will put it in the production server and then it will be running so if any production issues are happening the developer itself will take care it&#39;s again depends on the team only but i&#39;m telling so agility is nothing but in most of the areas we are implementing the agility in our data lake environment is gc but in data various environment is less such agility only running a deployment in sequence okay splitting the modules into small small modules and then we are having a separate separate team okay that is possible in data lake environment okay this is what agility if you are going for a real-time project so definitely these data vargas roles will be there okay business users so business usage is nothing but your product owner okay your application owner your business owner or your business analyst data analyst somebody else who knows the business they&#39;re going to give business use cases the requirement to us okay so their business users they are telling that okay we need to create a report in this format this way we need to have a report the data is there in source okay do atl pipeline load the data into a data vargos and take the reports from there that&#39;s what the business analyst will analyze it they will they will give the information okay these kind of reports i need for this kind of report so these tables should be loaded from the source that&#39;s what they will tell you so what the report writers will tell okay report writers will create a data model that the data primary foreign relationship and everything they will say like okay if i want to create the report like this i need to have the tables and data values okay i need to have the data in the data vargos these are all the data should be there so report writers then our etl developers okay i will take the data from source i will take the data from source i will load it into a data vargos here based on the data model the data model will be created by data modeling team so if it is a new business definitely data modeling team they will suggest us okay go for this particular data model you load it into this table they will give us the data model we are going to load it report writers they will take the data from here to a report and then they will be creating a report dbs dba will maintain the database okay database database environment okay here all this database this oltp olap everybody is taking in addition to this we&#39;ll be having a report admin tableau admin rpa redmin informatica admin okay everything admin will be there and we&#39;ll be having a testing team also report testing etl testing team will be there okay data modeler so they&#39;ll be using you heard about this airbnb ervin data modeling tool so this tool they will use irvine tool for the data modeling we do have some different data model i will explain that that&#39;s what they will use it okay this is what the different rules and if you go to this uh in real time these are all the popular etl tools are available in the real-time project so you&#39;ll be having informatica power center so informatica is the company name our center is atl tool okay informatica is a company name we have informatica id cube informatica data quality mdm okay master data management information big data management informatica acs informatica intelligent cloud services so different tools are there power exchange address structure lot of tools are there so by using this informatica power center we can do the etl process same way competitors of informatica is nothing but all our remaining tools are competitors like ibm data stage and microsoft ssis sql server integration service so sap bods business object data services ibm cognos data manager cognos is the reporting tool cognos data manager is the etl tool oracle data integration is the oda it&#39;s the etl tool sas data integration studio oracle vargos builder avenue talent apache nifi pentago explaining so explain this cloud-based etl2 see these are all the etl tool if you know any one of the etl tool so learning other etl tool is very easy the same concept you will use even if you are going for cloud big data or rainy cloud data load right etl pipeline same pipeline only we will use instead of this database we&#39;ll be having a data data lake environment that&#39;s a connection and everything will be different but loading process is same okay if you are approaching oracle oracle for the entire data data process in your company they will say go for oracle as the oltp okay article is the oltp oracle vargos and then use word oda okay oracle data integrator is the etl tool and here use oba see obi is nothing but oracle business intelligence so this tool you can use it for the reporting that they will tell if you are going for microsoft sql server they will use sql servers database and data values use ssis is the etl tool and then use power ba as the reporting tool same way ibm so you can go for intes or data virus okay ibm db2 for source system so then you&#39;ll be having a data stage for etl so they will say all the related to ibm product so that&#39;s what see the companies will choose which tool they need to go for so out of all the tools informatica is leading retail tool it will handle huge volume of data support and failover mechanism and everything will be very high here okay all tolerance everything will behave so same way reporting tool you have tableau you have power bi you have click view or click sense cognos sapo okay business object zoom data and google looker we have ssrs sql server reporting service oracle business intelligence spot for your micro strategy a quick insight see all these are reporting tools so we need to know at least what are the different reporting tools are there why because if you are going for etl developer environment you will be working with these department these two teams only okay you&#39;ll be giving the data for tableau timon so if they are finding any discrepancy on the data so they will be contacting the etlt model okay so this is what enterprise data teams will be there so i will show you the data model if you look at here so what is uh conceptual logical physical data model who will create this so normally this will be created maintained by data modeling team normally the year wind urban tool they are maintaining right so first of all how they will maintain this is us just uh give you an overall idea just i will give you an overall idea so first they will go for conceptual data model see what is the conceptual data model so they will take like a pen and paper and then they will have okay in order to create a data model for the particular company so i&#39;ll be going like this product table store table sales table time table see all this how it is related to a fact table here see includes the important entities entities is nothing but table under the relationship how we are relating with each other no attribute is specified attribute means the column names no primary key foreign keys specified here okay just the conceptual design model they will maintain all them how many tables are needed for this data model okay finance data model and then sales data model so different different merchant data model something for that we how many tables are needed that we will maintain it first conceptual data model okay it&#39;s very highly abstract and you can easily understand so we will not have any uh software tool to create a conceptual model just dependent paper is enough okay and then we&#39;ll go for the logical model in the logical model if you look at here all the tables we will maintain the column names okay in time what all the columns should be maintained in product table what are the columns to be maintained in sales table what all the columns should be maintained and what is the primary key foreign key so here store id is the foreign key from where it is coming from here it is coming store id is a primary key product id is the foreign key it&#39;s coming from this table date key then after that you are maintaining two columns okay so normalization occurs at this level normalization means we are we can split this table into two tables also okay so that&#39;s what your logical model so we are maintaining a logical level then after that we will go in for the physical data model what is physical data model see after creating like this we will maintain each and every column we will maintain the length see what is the data type what is the length what is the all the models we will maintain and then by using the erbin tool we will create the model right after that no need to create by ourself create table table name all this so if you go to the web model if you are saying that create all the tables it will create a tone stretch in the back end no need to create manually see all the relation relationship which table should be loaded first sorry which table to be created first okay that will automatically created based on the primary foreign key relationship we are not going to create if you are creating first of all we are to create primary key tables then only foreign key tables right if you are trying to create foreign key table first we will get the error why because this is where we are having this foreign key here so that cannot be created but ervin tool will create automatically based on that clear so data type and the data column column name and everything is based on the database we&#39;ll be creating here this is called physical data model so who will do all this by using the urban tool or power designer the data model team will create it data modeling team just to give you an idea i have explained this see what up we know already this is also one of the interview questions they will ask see volapq is nothing but multi-dimensional data okay if you look at here multi-dimensional data so hold up what is nothing but based on the multi-dimensional data model so enables a user to easily and selectively extract and view the data from different point of views so decision making the business process okay see it&#39;s nothing but cube is for our reference only okay so this cube we can see this in different view right so top view side view front view all the view right the rear view and everything bottoms you everything so one cube we are seeing in different way okay product supplier category and everything so look at here see we are going to have this table model in this particular cube say category we are having the three categories so this is one category confections like this will be having this categories category dimension okay beaver edges so all this we are maintaining in the category dimension okay if i want to analyze in this category of data so from this dimension you will analyze it we&#39;ll go for one more dimensions okay here this is like confections so this one another dimension this is another dimension this data will be showing only for convection data so now you can go for see this is one one dimensions each and each and everything is to each and every dimension in the cube okay so now you can see this this one we have seen but now if you look at here this is like time dimension january february march april so this is category dimension so each category we have one two three same way here we are the time dimension january february march april like this right are keyways so q is everything will be there the same way we&#39;ll be having a measures in stock on order sales amount sales cost these are all the measurable values that we are that we are seeing in different dimensions see if i&#39;m showing one value here this is for one category for one particular one particular month for one particular sales amount one particular measure right multiple dimensions are related so you can see data in multiple dimension weight this is called a dimensional modeling olap cube okay if you look at here i am showing this way so this is the data say for an example i want to know the january data so what will you do so i have to sum up all this right for january february i have to sum up all this march summer okay so here i want to go for one particular category then i have to go like this one particular category okay this is what i will be analyzing the data in different different dimensions okay the same way here i have the book category the product category quotas q one q two q three q four on territory okay so we are differentiated so if you are showing some value how it will be like you&#39;ll be having the different operations like four operations these are all the questions they will be asking in interview what is roll up drill down slice and dice pivot operation everything okay so roll up is nothing but summarize the data see how the q1 q2 q3 and q4 if i&#39;m asking the earwise data you&#39;ll be summing up all the queue right that&#39;s what roll up drill down means you have the q1 data so q1 i want to go for month wise so you are drilling down queue into month month into week week into date that&#39;s called drill drum okay slice and dice instead of having all the dimensional data we are taking slice on slice and dice from that so pivot operation is converting rows into columns if you look at here roll up operation is nothing but roll up is nothing but you are summing up okay we have the data like this see the other data product table you have the pc book shoe and clothes all the data you are seeing only for q1 okay q1 q1 this part pc and this is for one particular locations this is for new jersey los angeles and this is for australia regions okay these two are us these two are australia so now we are split we are summing up see this new jersey and las angels we are summing up as 2000 okay this is called roll up and this path and then sydney we are rolling up okay for this one see 395 six six not five that&#39;s a thousand was saying here right so that&#39;s called australian region data so this is called roll up so you can roll up here also so q1 q2 q3 q4 you can roll up to a year level data yeah yearly sales so roll up on location roll up on time roll up on category so that we got we can do it are you clear what is roll up so roll up is nothing but summing up drill down drill down is nothing but again here here the q1 right 605 that q1 we are splitting into january february march so 605 so everything will be maintained like a table level so in the table level data like uh on date level they will be maintained so we are splitting the table into monthly quarterly yearly weekly all this will be taking so if you are summing up this 150 100 and 150 it will become 400 so what is that 400 so we used to have the 400 year right this 400 for clothes clothes for q1 400 this is what we&#39;ll be having okay clear on this what is drill down okay slice slice is nothing but one dimension we are taking so out of all this right we are taking only one new jersey las angel&#39;s birth and sydney we are taking for pc and category only so this is all we are not taking only one we are taking okay six zero five this one only one dimension we are taking okay slice and dice dice means this is what dies same like dice we have right we are taking one particular dice here okay sydney and perth for q1 and q2 for this two categories we are taking large cube into small cube the pivot operations for summing up some values okay we are going for the pivot operations like we are splitting the rows into columns columns and rows for different analysis so this is what the different uh roll up and uh cube and all so we are the old app is nothing but types of volume see we have the roll up on app olap and others okay roll up is nothing but relational volume relational database management system olap that&#39;s what we have multi-dimensional roll-up so normally it will be multi-dimensional like only hybrid it&#39;s the other hold up so these are all something like data modeling team they will take care we are not going to do but you have to understand what is this they will ask so do you know what is roller something you know okay so just you can tell them what it is this is what the older cube and everything [Music] so what is architecture what are the different tools will be there you will be having a server okay in informatica you&#39;ll be having a server here so this is what my server informatica server it will be there in one single place so if you are going to implement informatica power center you will be having an informatica server so power center server you&#39;ll be having it so where to be there so based on the company&#39;s dation if you are going for a us project then server will be there in us only one single server it&#39;s a developments of different sat r uq will be different uh the the production will be different server but via the server okay so on top of unix or linux only it will be running in real time the security results this server will be running on unix r index server what this server will do it needs to take the data from source read the data from the source it&#39;s a heterolineal sources load the data into data variables or target or anything so it&#39;s sourced to target it should load the data so here we are fetching the data from here okay from here we are switching the data and then loading the data into data values so informatica we need to inform right okay you go to this particular table which is there in the oracle database you load the data into a teradata database data table on this particular entire data we will call it as a database say for an example if you assume that this is the error data how many schemas are there to schema in this era data instead of saying it&#39;s a schema we will call it as database one database two like that we will say say just i&#39;m making like a it&#39;s not database administrator uh you can make the database one and database two something like that okay we will call it as it&#39;s a database database tbo or database the same way here we will call it as in oracle it&#39;s called a schema this is another schema this is another schema see what informatica needs to do informatica needs to read the data from source always we will read the data from source we will not we will not disturb any data in the source source okay we will not change any data in the source just we will read it select operation will know okay what we need to do we need to write the data into target so here we will disturb the data we will insert the data if it is new sync data we will update it the target okay this is what we will do in our informatica okay we have to tell that server right see if you want to this is post you assume that this is the post the post you will be having a two address correct you&#39;re having the two orders here you&#39;ll be having the from address see from address to address we have to give it to the postman then only he will take the data this post from from address to two address okay this is called metadata from address to address is the metadata inside the cover you&#39;ll be having actual data correct see what is the message you have to pass that will be there in your cover so that is what informatica will read the data from this table this is actual data load the data into this table but which table we have to read so thousands of tables will be there in the source slide which table should be loaded for this particular business use case you go to the schema schema to an oracle database by using this user id and password see how you log into the database you will see the data you&#39;ll be having a username password based on the username password it will be having the privileges either you have the read privileges right privileged service you will go to this informatica will have his own username password that is called service id okay by using the service ide getting this database username and password it will go and connect to this database and then read the data so that information so you go to this particular database you take this table you go and load this data to a dbo on the teradata that information we will store it in the repository the repository is nothing but the from address to address the address should be stored what is the actual data actual data is there in the this table so what is metadata this is what the metadata this metadata will be stored in repository clear repository that is what we need to create a repository also it&#39;s a db only database only okay you may get confused here also we&#39;re saying db here also we are saying db again then what is the difference i told right this is a data actual data you have to load customer data to a customer data table in the teradata this is metadata information okay you informatica you go to this particular database it&#39;s oracle you go and pick this data from this table load it into a teradata to this table that information is called as metadata the actual customer data will be there right in this table that is called a data actual data so while running that informatica to load the data so this is called repository it will have all the metadata metadata is nothing but data about data so from address to address all this right that is called a metadata okay this is also another db see informatica server will run on top of one repository only database only that&#39;s why we are to first install repository on top of the repository we should install the informatica server then we should on top of this we should be installing the client so why client so we have to develop right based on our development only informatica will do that see as a developer i i&#39;m going to work on this client tool only on the client tool how many client tools will be there based on the number of developers see we are all working on the informatica project you assume that for for everybody we&#39;ll have one server in our system we&#39;ll have a client so for all the members we&#39;ll have we&#39;ll be having the this particular client tool capacity manager designer workflow manager and monitor so this client tool i will explain later don&#39;t worry this client tool i will explain later you assume that by using this client tool only we are going to develop a business logic and we are going to save it in the repository when we are going to run it this rapacity will give information to a server okay through repository services this is called repository services repository services is nothing but from this repository services only it will pass the information to server okay server you go to this particular table you read this data load it into teradata in this table that&#39;s what information it will give you so server will do will have one services called integration service by using the integration service it&#39;s like a servant okay by using this integration service this is called is it&#39;s for data movement okay data movement we will use the integration service integration service will move the data to from to target that is your social target okay so it will have three different sorry three different servant that is your thread what is the three different thread right we have threads is called reader thread so reader thread reader thread will read the data from source transformation thread will transform the data okay that is what information right writer thread will write the data into target so this is the architecture of informatica power center when we are going to run this informatica server through client repository information repository will give the information to server okay you go to this one you pick this and then pull the data it&#39;s like a postman where the developer will come into picture here the developer will come into picture etl developers here only the etl developers sitting here then they will do all the etl mapping then they will do it etl admin they&#39;ll be coming picture here informatica server okay this is like who will create this table oracle this article who will create informatica server will create whenever we are doing the mapping right it will maintain informatica will maintain so whenever going to save that informatica will make an entry in the table saying that okay you go to this particular table you pick this data you do this transformation logic and load it into the target table that will will be it will be maintained in the oracle trip okay uh you assume that this is the etl process okay it&#39;s a i&#39;m saying you have you have different water resources okay you have different water resources you have more one okay you have some uh build okay you have some mill you have some lake you have some river see it&#39;s a different kind of water resource right different kind of art resource i have to take water from these water resources and and i have to send it to a consumer the the people sitting there there in different locations can i take directly all the time from this water resources and then send it to them no right what we will do we will have some our tank here right we&#39;ll have some over tank so we have to do the pipeline first you do the pipeline all the water resources will not have the waters on same time so this this one we have to do one one day in ones this one two two times in the day we can do this is like every uh four times five times we can do but this is like all the time we can do but somewhere we cannot have it something like that some different different scenario will be the right so here we are the motor here we are the motor and electricity we have given so we are on start button so we are doing the this one so this is called data arrows this is called heterogeneous sources we are doing some ro process okay ro process is nothing but your transformation logic okay so we are transforming all the different kind of sources into a single format our water is single format right single format loading into a over tank from there we are sending to the different consumers reporting team or tableau or powerbi or somebody else okay they cannot go and hit this directly to the water resource they will take it from the over tank only this process is called as pipeline right same way here we will call this etl pipeline either it is water pipeline the same we will call it as etl pipeline same meaning only now etl process cut it this is what will happen informatica will read the data from source and target right so that logic you are going to do in informatica server so informatica what is that informatica will do we are going to do the transformation logic we are not writing any queries uh add our queries here just we will write the transformation logic here it will read the data and load it into the target so this is claimed informatica client okay this is informatica power center client i have to the server i will connect it now through my username and password i will connect it has been connected to a server see you are also working in the project i am also working in a project same project multiple developers will be there right all the developers will connect to the server okay then they will do the etl logic say for an example i am working on this particular project i will join into this folder you are working in different folder to the different folder you open it and then do it see this logic we have to do it right where we will do the informatica client once you have done it save this if you are saving it it will be saved in repository okay repository it will be saved in as a metadata will it run will it move the data now no i&#39;m going to run the workflow i will explain this okay you will get to knowledge see this is all the client tool then where the server server is there on single one place in our system in our machine it has a server okay services that is what before starting informatica you have to start the informatica service it is running right so this is the server server running back end server we have connected through client we have applied now it has been moved the data from source to target is it moved see four partitions see all the data is moving from source to target once you are executing it then only to remove the data so this is called a client i told right so designer workflow manager and monitor and repository manager okay server is the the server will do the server operations it will read the data from source database and move the data client cannot read the data from source that the client cannot read okay no it cannot read only server informatica server can read the data from here it will have the integration service it will initiate the integration service okay you go there read the data from here load the data here that&#39;s all as an etl developer i need to do the mapping logic informatica client store it in the repository while running it it will give the information to the repository capacity will give information to server server will read the data to turn source and target when we are going to do this one we will come to know see this one is called repository services i told right repository services rep and this is called integration service is the integration service integration service will come into picture only on the data movement repository services will come into the while we are connecting it will be stored in the repository whether i have i have created this mapping now no right it has been created as some days before where it has been stored this logic in repository metadata so we have by using the informatica client we have created that metadata we are stored in the repository now i just connected through client i have seen that and i am executing it so this time whatever the logic i have already created that will go to the informatica and then it will pull the data from source to target in this table on this flat file you have 11 columns you have four columns these are all the column names you have to read from here you have to do joining up these two tables based on the some id columns okay department department id you get the information load it into the target table this table just we are telling that right that is what as a developer we will do it we have developed it we will save it while we are running it informatica server will have the integration service okay that integration service you have the three threads reader thread you go and read the data transformation thread you transform the data as per the logic as per the logic we are joining it right this is the logic actually okay load the data into target table so the table information should be there for informatica this table structure and everything should be there for informatica then only informatica can load the data [Music] hello friends welcome to nikki academy in our today&#39;s video we are going to see how to install informatica power center 10.4 in our windows mission in order to install informatica 10.4 in our machine we have to follow these nine steps if you are following these nine step you can successfully install informatica power center in your machine for the informatica installation what is the system requirement so we need to have 50 gb of space in the hard disk and we need to have windows 8 r10 r11 64-bit os 64-bit os and we need to have 6 gb of ram minimum 6 gb of ram or 8 gb of ram so if you have 4 gb of ram we can have it like a 4 gb of ram but this will be working with very slow if you have 4 gb of ram you can upgrade with 6 gb or 12 gb or 8gb so you can check the slot based on that you can go to the service center and then you can increase the gb and if you have any antivirus as of now for the installation you can disable the antivirus why because antivirus software will remove the exe file of informatica so that&#39;s why any of our installation will get disturbed so no need to have any antivirus software if the antivirus software has been expired you can uninstall and then you can try with informatica installation if you have windows 7 64-bit os yeah very well we can try with the 10.2 version so this is the requirement so let us start our installation as a first step we have to download informatica 10.4 software from oracle e delivery site or i have already downloaded and given the link in the descriptions you can download from the description itself once we have downloaded we will be having these folders oracle 11g 64-bit edition sql developer and we do have informatica server unclaimed we are using oracle 11g for our repository as well as source and target database i will explain the setup how we are maintaining the source target and repository so how do we create tables and everything so that you can also follow the same step okay consider this is the article 11g software this is a database we have so in so once you have installed the software it has inbuilt the user is called system okay inbuilt user is called a system that is a schema we have and inbuilt we have one more schema called hr so this schema we will create here and we will create one more schema for our informatica repository so that schema we will call it as info info user we will create and we will create one more schema for the target okay so we will keep so this database for source and target so this is the system schema info info schema for informatica repository hr schema for source and core schema for target we don&#39;t have any other database here so that is the reason we are going to use source and target on the same database if you have any other database you can make article 11g as the source and sql server or db2 or teradata are the target database okay so in system schema we have metadata tables all the the oracle metadata tables will be there so you can see here oracle metadata tables will be there so this is the metadata tables so here we we do have metadata is nothing but all the dictionary tables will be there metadata tables dictionary tables available in the system schema on informat info schema so we are going to create the schema for schema is nothing but subset of our database for the easy access and privileges we are creating the multiple schemas so informatica schema we are going to use it for informatica repository so we are going to create informatica opacity to hold informatica related metadata tables so then we have the source source and target this hr schema we are going to make it as source and this course schema we are going to make it as a target through informatica we are going to pull the data from hr schema tables to core schema tables so this is the informatica tool we are bringing here in hr schema we do have so here we have the tables like employees locations departments lot of tables are there okay departments here you can create any tables like p underscore employees like target employees you can create a customer table you can create any tables here so product table any tables you can create so we are going to make this schema tables as source this schema table says target so through informatica we are going to load it so why do we need this in first schema to have informatica metadata repository so this is the rip we are going to create it here so informatica repository we are going to create so in this particular schema so that is the reason we have created in first schema by default system schema on hr schema will be available this informatica info schema and core schema we will create it so in order to query this database we are going to have the tool called sql developer so sql developer is an application development tool so this tool we are going to so we have downloaded already so you can download if not in download you can download so first we have to install oracle software then we have to configure this sql developer from sql developer we have to give the connections you can query this database so that this is sql developer just an interfacing tool gui tool graphical user interface so what data is available on the database we can query from the sql developer we have to use oracle 11g for informatic installation we do have latest versions like 19cr21c or 18c 12 cnl but we will though this this particular informatica version will work on article 11g only we can go ahead and install the oracle 11g as a repository as well as database then in order to work with this database we are having the sql developer then this is the software like server and client say right click on this particular software once we have we have given straight click and extract to this particular folder if you do not have winrar in your machine go to google winrar download and go to this particular link and click on download winrar so this is the authorized software only no need to worry about it and click on download winrar it will get downloaded once downloaded the exe file you can click on it and then install it it&#39;s very simple step since i have already installed i am not doing it again you can install it so first we are installing this in winrar or winzip if you have then you can use it right click on this go to extract you can extract all the four folders once extracted this is what the folder you will be having it so we have done with first two step and third step we will go and install oracle 11g for repository as well as source and target database so once extracted you&#39;ll be having this particular folder double click on this folder click on this oracle this folder go to disk one double click on the setup file click on yes then it will be preparing to install this is a page you will be getting it this is the oracle database 11g express edition click on next accept the license and terms and conditions click on next year so here this is what the destination for where it should be installed click on next year no need to do anything so this is the admin password for the user sys our system so we have to carefully choose this so always make sure that you are remembering the password so enter password i am giving like admit password is admin confirm password assignment click on next click on install it will get installed it will take five to ten minutes of time depends on your system speed we will wait for it once installed then we&#39;ll go for the next step once installation completed we&#39;ll be getting this particular wizard click on finish we are done with these three steps we have downloaded the software extracted and install oracle 11g and we will configure the sql developer to connect with this database and we will create source and target connections go to this particular page sql developer so we can copy this sql developer into our c drive so it will not be disturbed whenever you are changing your files we can copy it here paste it here the c drive okay we have copied sql developer into c drive double click on this go to sql developer folder this is what sql developer right click on this we will make one shortcut to the desktop so send to desktop so you&#39;ll be having a desktop icon if you look at here we&#39;ll be having this shortcut double click on it so next time you can double click on this sql developer directly no need to go to the folder the sql developer will be opened if you are doing it first time you&#39;ll be getting that yes or no click on no here this is what the first page you can see here and click on this plus symbol we will be creating a connection from sql developer to the underlying database oracle database the connection name just i&#39;m giving like admin username is system so this is the system username administrator username password is admin so this is the password we have given while installing the software you can click on test sql developer is testing it is pinging the underlying database and we are getting that success this status should be success so this is the admin user by using this admin user we can control any other user so we have to remember this admin password so you can click on this plus symbol again we will make the connection for hr schema initially hr schema will be unlocked state just you can check it here so this is a username and password host name will be localhost port number is one five two one s id is xe nothing but express edition if you are testing it showing like account is locked so initially the hr schema account will be locked state we have to unlock it so how to unlock it from admin schema only we can unlock it you can go here we can mention that alter user user name identified by password password is hr account unlock just click on account unlock it will unlock then you can click on the plus symbol and use hr go to username hr password also hr click on test now the success click on connect so we have connected hr schema left hand side if you look at here some seven sample tables will be there you can make use of these seven tables for our practice okay now we will create one more schema called core we can go to create go to admin schema make sure that you are doing it creation everything user the admin schema create user core identified by core this is nothing but password this is the username this is the password just we are creating the one user schema and we have to grant the permission grant dba permission to core first time permission i am granting the all the permission to the core user from the admin schema i hope execute then click on the plus symbol you can make sure that giving the name a score and username will be core password will be core you can create any other user also no issues we can create any other users look at here this is what the admin user and we can create one more user for the informatica repository create user the repository i&#39;m creating for info okay info user just i&#39;m creating it grant dba to infor just we can execute this particular user and grant the dba permissions so this user this schema i&#39;m going to use only for informatica repository and hr schema is for source and core schema is for our target okay we can have any other database also for target our source also that is up to our requirement we can use sources oracle target is sql server towards something different so we can have it different different source and target connections only in informatica we are going to define the connections so we can create one more connection like infar the username is called info and the password is called info just click on test should be success then click on connect this info and you core we have created so nothing will be there the tables nothing will be there and info also we do not have any tables so what i&#39;m going to do i&#39;m going to create in code i&#39;m going to create a table okay normally we have to create like this in order to load the data i can go and create admin from admin i will create a table so create in this way create table t underscore core dot schema name dot p underscore implies t underscore implies nothing but target table i&#39;m just creating a table in core schema to load the data from hr schema let&#39;s select star from hr dot employees from employees table just i&#39;m creating the table structure where one equal to two and just maintaining click on control enter one table has been created you can go and check this table in the core schema now one table will be there here so this table we are going to use it for our target so hr schema we do have employees table if you look at here select star from employees table i have a source table by using informatica i am going to load it into a this table we underscore employees okay we have created all the setup configured sql developer and everything we have done with post chip now we will go and install informatica power center server in order to install server go to this particular 2 of 2 folder part 2 you can see the server you can click on server here click on install so you&#39;ll be having the install application file so this is hot fix no need to install it just click on this install double click on it click on yes here you&#39;ll be getting this particular page install anywhere is preparing so this is the informatica server installation step you&#39;ll be getting this particular page so we do have 12 step so this is the first step install informatica 10.4.1 agree the terms and conditions click on next year so this is the prerequisite for this particular informatic installation you need to have disk space is 50 gb and ram memory is 6 gb minimum if you have 4 gb ram then your system will be very slow we can install it but it&#39;s very very very slow click on next here we need to provide the license key so where the license key available click on this three dots go to this particular one off to part one here we will be having this oracle america this key dot key file i have given this key file in the description you can download it and then you can keep the file and then you can choose it click on next so this is the fourth step the available space everything it will be displayed and then click on install here so this is the fifth step it will get installed it will take minimum half an hour to 45 minutes depends on your system speed so we will wait for all the step all the 100 percentage to be completed then we will go for the sixth step let it be installed so once 100 is completed we&#39;ll be getting this particular fifth step no need to do anything here just click on next here we can allow access click on allow access and it will generate the key store so here so this is what the repository just we are creating it enter database information for the domain configuration repository so we are using database type as oracle and either we can use system schema or input schema just i&#39;m using info so user id is info password is info you can go for system admin also then we can have the host name as localhost localhost port number is 1521 without space service name is xe express edition xe and then click on test connection the connection should be successful that means informatica is bringing the oracle database for the repository creations and again we are getting the connection successful click on ok click on next so here we have to give encryption key see encryption key we can give any key here i&#39;m just giving administrator at one a caps administrator up to one just we have to give in this format click on next here click on ok and this is what the domain name host name and everything so informatica port number is 6005 and domain is domain domain password is this administrator only so we cannot copy it from here copy and paste it here we are to type from here so type it a caps so username is informatica username is administrator password also administrator a caps administrator we can make administrator administrator administrator we do not want to create any model repository so click on uncheck this click on next here so it will go and create an encryption key it will take some time [Music] we will wait for some more time and then we&#39;ll see the next step in this eighth step no need to create any other new user just uncheck this option go for next it will again this step also will take some time in this step we have to provide we have to create repository service and integration service in order to create repository service we can give the use database user id we can go for info and password is info and database service name we can give x e and power center repository service we can give rep r so we can give any name just i&#39;m giving like repository i&#39;m creating like rep and integration service is just click on next so if you are getting this particular path no issues click on ok click on continue if you are getting this particular error now you should just click on continue once installation completed you&#39;ll be getting this installation status access and you&#39;ll be having this informatic administrator home page so just to click on this home page so this is admin home page we will check the informatica services and everything so click on this is https just to click on advanced click on proceed to desktop so this is the page you&#39;ll be getting it see always informatica administrator pages username is administrator administrator password also administrator you can just copy and paste ctrl c ctrl v copy and paste click on login we can see here services and nodes go to the services and node we will check whether these two services are available click on this repository services rep it should be available the operating mode should be normal mode and is both should be available see this is also is should be available so this is a repository service this is integration service just to make sure that both of the services up and running okay both are up and running now no need to do anything if you are facing any issues please raise your questions through chat i will respond to it okay we have created the repository service and integration service we have configured this all the six step has been completed now we will install the informatica client seventh step go to the folder informatica client one of two go to this one you can extract this folder and go to this client folder go to this client again so here we&#39;ll be finding the install so click on install click on yes here then this is the page it will get installed the client software will get installed easily we want to go for informatica 10.4.1 client just click on next year click on next so here we can choose power center client we want to have informatica data quality then use this informatica developer so just we are using power center client just click on this option then click on next next uninstall so it will get installed very quickly the same way server side you can click on now ok done then client will get installed so informatica client has been installed successfully so this is what the page you will be getting it click on done here so completely has been installed so we have completed all the seven step we have installed informatica server and declined and configured the capacity and integration service so we have done with all the installation we will configure the source and target connection then we will do one test mapping so that you can test it whether the source and target connection is correct and then we can move the data so as we discussed earlier sql developer is the tool the frontend tool and we do have uh oracle 11g as the database we will have this article levitg for source and target so here we have the hr schema so we will use this hr schema for source we do have seven sample tables so through informatica so we will make this hr source and core course target so consider in hr we do have the tables called employees so employees table we will make this employee stable source and in our core schema we have already created a table called t underscore employees if you haven&#39;t created then create this table i have given the create table in in the description so this is the etl tool it will load the data from source to target so we can load either from table to table here the article itself or we can go for any other target also take the data from hr schema and load it into a other database also like sql server db2 through etl so only thing is the connection will be vary from informatica level okay so here we will make the hr schema source in the admin schema i have created the table like this create table table name and the course schema if you look at here i do not have any data so i have given the all the queries in the description you can also create it we will make this hr schema table source for schema tables target we will extract the data from employees and load the data into t underscore employees we will open the repository manager so go to search type repository manager the informatica repository manager will be opened this is a four claim tools we have we do have repository manager manager designer workflow manager and monitor so this is the four client tools are interconnected with each other so go to repository we will add the repository here this is the procedure we have to follow add the trip click on ok the capacity will be added here click on configure domain the domain so you have to click on this box and then domain domain name is domain only gateway host is localhost so here you can check this domain this is what the domain we have entered gateway host is localhost only localhost without space and port number is 6005 so for informatica the port number is 6005 just click on 6005 click on ok here then you&#39;ll be getting the rip rip from here so it should be available here then we have connected correctly and right click on this click on connect this is the page you will be getting it so definitely you should hit the page with security domain is native if you are not getting this page then we are the services is not up and running go to services.msc if this now if this page is not available and go to services.msc check here informatica service is up and running see informatica 10.4 is up and running if it is not running then the left hand side will be having the start button started and it should be running so wait for it five to ten minutes then after that you can connect to the repository so here if it is not available just start it so wait for five to ten minutes then we can go to the here the username is administrator informatica username is administrator password you can copy and paste here click on connect it has been connected you can see here deployment group is available so it has been connected so we will create one folder for us go to folder create we can create the folders training click on ok only administrator will have access you can see here folder has been created successfully and we can go to the designer open the designer whatever the folder we have opened will be opened here automatically all the four clients are interconnected so you can watch my video the other videos on the client tool you will get all the stuff very clearly click on training open and here so this is the five client tools i have explained all this details which is available in my in the channel so you can go and check it here so what is source analyzer everything firstly we will define the source go to the source here import from database so first of all odbc connection will not be there we have to create it so click on this three dots will be opened here click on add go to this oracle wire protocol click on finish data source name we have to give oracle underscore src the oracle underscore src is nothing but the database connection name i&#39;m just giving us oracle source host name will be localhost this is oracle details number will be one five two one and service name will be xe so click on ok here will be created oracle src same way we will create for one more connection for oracle tgt so click on here so you can go to and same way you can create the oracle tgt connection host name will be localhost port number will be 1521 service name will be xe click on ok here so both the connection has been created click on ok choose the source connection from here username will be hr tab click on tab automatically populate the owner name order name is nothing but schema name okay so honor name is nothing but schema name this is the schema hr schema click on hr so in the hr schema then click on connect it will populate all the tables and views in tables we have the table called employees so we do have seven tables just take the employees table click on ok here go to target the same way target here import from database choose the target connection and click on the username is core for target we are making like more click on connect here so target we have created only one table that table will be available here so choose that table target t underscore employees click on ok this is what the page will be getting it the target table has been imported now next step we will create a mapping go to mapping create a mapping m underscore test mapping i&#39;m just creating as test mapping so source if you want to get what is source target and what is the source qualifier and everything please watch my channel so i have explained all this very clearly just drag and drop source here drag and drop target on the right hand side so always target should be on the right hand side keep the cursor here select all keep the hudson on the header drag and drop here all the columns so all the columns should be connected correctly then control yes this mapping should be valid so mapping should be valid so we have created a mapping then go to workflow in the workflow manager you can see here workflow manager this is our opacity this is integration service go to workflow designer the workflow designer we will create our flow but before creating a workflow we will create a connection go to this connection just informatica we are telling that by using this connection you connect go on hr schema a source go and connect this core schema as a target you have to provide all the username and password just go here click on connect relational connection no connection will be available see what is your source and target based on that we have to create a connection click on new so source is oracle right just we have we have to choose the oracle connection click on ok now oracle underscore src just giving the name user name will be hr password also hr and the connection string is xe express edition click on ok so in our case target also oracle database at core schema if you have target this sql server or db2 or teradata here we have to give whatever the new we are going to have here we have to choose it so target also oracle in our case just click on oracle click on ok here with the connection name just i&#39;m going to make oracle underscore pgt click on core over here and connection string is xe and click on ok then two connections we have created that&#39;s all if you want to have more connection just to create it here now go to the workflow designer create a workflow create here wf underscore workflow name is just i&#39;m giving like test workflow just i&#39;m creating a test workflow it should not have any iphone just click on it should have only underscore test workflow click on ok here and click on the session icon keep the icon here then choose the test mapping click on ok then click on the link task link it double click on the mapping we have to provide the connections in order to run the informatica workflow we have to run the connections go to mapping the fourth tab click on source source should be pointing to source connection source is always a reader since database it&#39;s a relational reader target is always writer subject target also oracle relational writer so here we are to bind to the oracle tgt connection click on this arrow mark choose this tgt and go little bit down you can truncate and load so what is in the truncate and load we are truncating the data before loading the data into the target table apply okay control yes so this workflow also should be a valid one and you can see here before starting it i&#39;ll show you the target table doesn&#39;t have any data right doesn&#39;t have any data so right click on the white space start workflow monitor will be opened automatically look at here monitor will be opened running you can click on the task view it got succeeded but here it got succeeded the task you open choose the left hand side the workflow name this is a session name session so this is the workflow this is a session task you can go for right click on the session session this is a session right click on the session get run properties you can go to the source and target statistics you can see here 107 record read from the source and return into the target so this is what there is no rejected record right you can go and check here so ctrl enter you&#39;ll be seeing that all the data has been loaded if you check the count all the onenote summon record loaded into the course schema tables from hr schema through informatica we are loaded into this table so always make sure that this table we will create it so normally we have to create like a create table table name and all but we have created by using the admin schema so i have given all the script sql queries what we have executed in the admin schema in the description you can also follow the same and once you have completed everything this is a test mapping it&#39;s running correctly then once you have completed everything right click on the repository disconnected okay this is what you are properly disconnecting it so this is what we are properly disconnecting it so you can disconnect all the four client tools correctly disconnected so you can close it under pass to here close it that&#39;s all so repository manager also you can just a disconnected this is one time activity repository manager next time you will not open it directly you can go for designer for example tomorrow i want to start i want to work on this particular tool so directly you can open this sql developer from here and then you can work on but informatica we have to go to the services.msc as i told earlier just click on this informatica 10.4.1 we can start the informatica service from here since already it is running no need to start but make sure that it is running if you are starting now wait for wait for some more minute wait for five to ten minutes after starting it after ten minutes go to the designer and then double click on it double click on it you should be getting this particular page native the domain name should be a native that&#39;s what has been connected clearly okay so this is how the setup file please so we have installed successfully you can watch my channel for the rest of the session you can practice it and then you can learn more on the informatica thank you for watching if you haven&#39;t subscribed my channel please subscribe the channel and click on bell icon so that whenever i am uploading a new video it will be get it will be notified thank you all [Music] hello friends welcome to nick it academy i am chandra shaker i hope you have seen my previous installations video and you have installed the informatica power center successfully and let us begin our complete training so kindly watch the playlist as well as the description for more informations so let us start our session so informatica power center has four client tools the first one is repository manager designer workflow manager and workflow monitor if you look at here so i&#39;m just selected the repository manager here so here we have the repository so repository manager designer workflow manager and monitor so these are all the four client tools available in informatica power center so these client tools are interconnected with each other so all these tools are connected with repository so first we can connect with repository so let us see all these playing tool one by one first tool repository manager so this repository manager will be denoted as r here so this is the repository manager and it is used to do all the admin related tasks so what are the admin related tasks so we can create a folder for our project we can provide some access and privileges and once the code has been developed we can export the xml or you can import the example so for an example you are getting the data example from production environment we have to do some modification of the existing work existing logic then you can import the production xml into a repository and then you can do so all this you can do by using this repository manager historical run status we can see by using this repository manager so this is a repository manager just i&#39;m connecting so connect this repository manager i&#39;m connecting with admin user so in our real-time project there will be a lot of lot of folders should be there and just i&#39;m creating a folder for this so go to this folder you can create a folder so here i&#39;m just creating a folder called nic it academy so all these sessions we are going to see in this folder only so just i have created so here i have created a folder so you can you can see here the folder has been created and it will not have anything as of now so this is the one thing we will do here after some time we will export the xml file import the xml file in this repository manager so that we will see a later point of time so these are all the four steps you can do the client the second client tool is designer the designer by using designer we can do three things one is source definition target definition and mapping the mapping logic source definition we can define all our sources the sources might be a flat file or it could be a relational database table or it will be a xml file or cobalt file at the same way the target we can define and mapping logic so normally if you are the developer we will be using this designer tool very much around 70 to 75 percentage you&#39;ll be working on this designer tool only so all this business logic we will create by using this mapping designer okay we will see all this tool one by one so from here just i&#39;m connecting to the designer you can see here the designer tool so it will be automatically so whenever you&#39;re opening the folder here this the corresponding folder will be opened here automatically so at a time you can open only one folder here so you cannot open two folders at a time so if you if you want to open the second folder you can right click and open it so folder should be opened if you are closing it if you are just closing it disconnect it it will be disconnected here also it will be disconnected if you are disconnecting this workspace will not be available if you want to open so there is a difference between connect and open if you connect the folder will be connected but the workspace will not be opened so you can open the folder by using this open option so you can open it so this is the designer tool you can see here there are five component in the design tool by using this designer we can make three things one is source definition target definition and mapping so go to this designer if you look at here there are five components in this design so first component is source analyzer so by using the source analyzer we can define the source if you are clicking on the source here source analyzer the top you can see the source if you click on the second one target designer the top you can see the target then transformation developer so that i will explain one by one then map let designer mapping designer so what are the five components source analyzer target designer transformation developer map designer mapping designer so as a first time it will be little bit confused for you but after you are using this tool for extra two to three days so definitely these component names will be very easy for you by using the source analyzer we can define the source definition so source search might be relational database table or a flat file or cobalt file or xml file for example here i&#39;m going to connect the sources source here so import from database or a file or import from cobalt file or xml file or you can create an instant source definition here but it should be a same structure of your either your file or it&#39;s a database table or anything so here i&#39;m going to connect with import from database i&#39;m just connecting it this source table from hr schema i&#39;m going to take the table called employees table any table you can take so consider this is the existing architecture so this is the database i have so in this database i have three schemas so one schema i have created repository so i have the same i have explained in our installation so here i&#39;m having so this is a structure we are going to follow for entire project entire session so so you can see here so we have the hr schema so from here i&#39;m going to take so from here schema i&#39;m going to take the table called employees table so from this table from employees table i&#39;m going to take the data and i&#39;m going to load the data into t underscore employees table so it&#39;s a table to table note just for for the explanation i&#39;m going to explain so how to load the data from table to table by using this informatica client component you can see here so source analyzer i have created the source and i&#39;m going to the the second component is target designer by using target you can define the target here you can go to the import from database and see here the target might be anything so i&#39;m just connecting target table for the explanation i i just connected one table t underscore implies just i have taken this this t underscore employees table from the core schema so this is the first component source analyzer second component is target designer the third component is transformation developer so what is this transformation developer you can see here these are all the transformations by using this transformation we can apply a lot of business logic so this is the update strategy expression filter and aggregator the same whatever we are doing in sql the same logic you can do by using these transformations in informatica the transformation developer is nothing but a single transformation if you want to do the reusable logic then you can create the reusable transformation here so this is the very early stage to understand this reusable logic but try to understand so going forward i will be explaining this reusable transformation very clearly if the single transformation can be reused in multiple mapping so that will be called as reusable transformation so here in project we can use this reusable transformation encryption decryption time conversion date conversion some conversion logic all this right so most of the projects most of the developers are using the same logic in this case we can one developer can create the transformations by using this transformation developer and then others can just drag and drop this like function in oracle the second one the fourth component is maple designer map plate descender is nothing but so maple designer also is used to create a reusable component but here in transformation developer a single transformation you can use but in so for an example that the reusable logic you can create by combining two to three transformations then you have to create in create it in maple designer the maple designer is nothing but a set of so here i have explained everything so i have posted this document in the descriptions so you can take this document transformation developer and map the designer transformation developer is nothing but it&#39;s also reusable component a single transformation can be reused and it can be reused in multiple mappings and maple designer is nothing but so this is the maple designer a set of transformations can be reused in multiple mappings so this is also reusable component the fifth one is called mapping designer so this is the mapping designer by using this mapping designer we can create the logic in mapping designer you can see here left hand side the source and target definitions and everything then you can go here you can go to the mapping and create so m underscore employs details just i&#39;m creating this employee&#39;s details just to drag and drop this one employees to mapping so you&#39;ll be having this source qualifier source qualifier is nothing but the source qualifier converts the source data type into informatica native data type so that is the functionality of source qualifier this concept we will see in the later sessions okay so i have just i drag and drop the target so this is called a mapping you can just drag and drop this one this is a table to table load you can see here this is called pipeline so if you have if you have connected wrongly you can select here you can delete and again you can connect it just you can drag and drop this one so just it will be connected you can save it should be valid so i will be explaining this transformation developer and map designer in later sessions in detail we will have separate session for transformation developer and maple maple designer in that session i will clearly explain okay so these are all the five components which are available in designer level so just i have explained here so now i can go to the workflow manager click on workflow manager the workflow manager will be opened so workflow manager is used to create a session and the connections and workflows in workflow manager three components are there task developer worklet designer and workflow designer this task developer is a reusable component so the in designer in designer we have seen transformation developer and mapper designer here single transformation can be reused the same way in task developer we can create a single reusable session so that i will explain in later session but just remember this by using this task developer we can create a single reusable session we can provide the connection also in task developer but if i want to create a logic a reusable logic by combining two or three sessions then i can go with worklet for an example in the project you can have some control table nodes so after loading the target tables you want to load some control tables it&#39;s a common for all the projects so you can the one person has created already the same worklet you can use it so any of the developers can use it and this is called worklet designer so this task developer and workload designer is used to create a reusable component the third one is called workflow designer so here we can create a workflow go to workflow here just i&#39;m creating a workflow look at here so this is what workflow we of underscore implies so just i&#39;m creating a workflow the start button will be created and you can see here the session so here also you can create by using this task and create so you can create a task but here we have to enter the name but here we have a shortcut for this under session you can click on here you&#39;ll be having the associated you have to select the associative mapping so since we have created only one mapping so only one mapping will be available you can select here you can click on this link task and you can link it you can see here lot of connections we need to provide the connections here you can see the connections can click on here we can provide by using this connections we can provide relational connections the file transfer protocol connections and application connections and loader something like interior data we have some loader fast load or multi-load or something loaded that connection we can create and the front-end application connections we want to create that application connections we can create so mostly we used to create relational connection if it is a development environment you can create a connection but in real time if it is a production environment you have to ask the informatic admin team to create a connection they should have already created you have to use that connections but normally we will have a parameterized connection so that i will tell you after some time for a security reasons we&#39;ll be creating a parameterized connection so that is one that connection only we are we are using so not like this hard coded connection we will use normally parameterized connection so that will be passed through parameter file that concept we will see in later point of time just we have created a workflow you can save it so this is what the component in the three components in the workflow manager okay next one the monitor the fourth client tool is called monitor by using this workflow monitor you can do you can run the workflow you can see whether it is running or succeeded or failed or aborted or stopped so anything you can see the status it&#39;s just a monitoring tool you can see the row count source and target row count how many recorded it read from the source and return to the target so that you can see here and throughput is nothing but number of record per second so per second how many records it has been read and how many record it has been returned into the target that you can see here and you can see the session log so session log is very much important in our real-time environment so whenever anything happened you can check the session log okay so these four steps you can see here in monitor so whenever you are running this workflow the monitor will be automatically open you can see here so remember whenever you are creating a workflow remember whenever you are creating a workflow that says the mapping should be valid so then only you can able to create a session and then you can able to save this workflow as valid one so here it is showing like start workflow has been started you can see here it is running so you can see whether it is running or it is aborted or it has failed anything so either you can go for gun chart or task view so it is showing like failed so we have to check why it has been failed as i mentioned here we have to go here get session log so you can see get session log you can press e here so that is what it will tell you it tells you that table or view does not exist where i am getting the error so that is on the right thread so reader thread it has written it has it has read the data you can see here read the data here from the source but while writing it is getting failed so you can check why it has been failed can go here so it is showing that session is failed but the workflow the parent the parent of workflow is showing like failed sorry succeeded it should not be the real time cases you have to go here you have to enable one option called the failed parent this if this task fails fail parent fail parent if this task fails if this task fails fail a parent this one workflow you have to fail it okay and also you have to check this option go to config object you can make this stop on errors equal to one so that two options so every every sessions we will give these two options whenever if any one error occurs it will stop the workflow so i can provide the control s it&#39;s a valid now just we have pointed out can the the because of the connection it is failing and go to the mapping you can check the source and target connection source is always a relational reader you can see a relational reader so you can provide the connections here it is pointing to hr schema you can see your hr schema and password is available in target just you can connect to the target and you can provide the connections here run key target table option then apply okay here so should i give trunk a target a table option for in in for every sessions no need if you want to truncate before loading the table then you can do so again i&#39;m just starting you can see so just i intentionally i did it to explain you the session log and you can see here it is running it will succeed now so right click get run properties you can see here the source and target statistics so here it will be showing like how many record has read from the source and then it has written the target and there is no rejected record throughput is nothing but record per second one not rec 1.7 record and here also not sound record so there is a low error you can go and check in the database table so here is a table the target schema so this is the core schema then you can check here so it has been loaded so this is a target table you can see here the data has been loaded into the table and it has one or seven records count the rows it has 1.7 records so this is the so four client tools in the informatica level you can see here so these steps you have to follow whenever you are creating a workflow these steps you have to follow the first one you have to do the source definition so that is what we have did source definition we have did first one second one target definition so these three things first one source and target and mapping these three things you have to do in designer so that is what we have done source and target and mapping then fourth one is the session we&#39;ll be creating a session if you want to create a session reusable session then create a reusable session and task developer then here it will be available so we have just created non-reusable session so what is reusable what is non-reusable that i will explain in later sessions we have provided the connections by pointing to source and target database and then we just created a workflow and we have executed it and monitor it so remember these are the naming conventions we have to follow the mapping names should always have m underscore the lowercase m underscore table name or a meaningful name a session name should be yes underscore the mapping name it should be yes underscore m underscore the meaningful name and workflow should be w k f r w f underscore so this is what the workflow name mapplet is mplt rml worklet is wklt or wl dimension table should be underscore dim or dim underscore or d underscore fact table should be fact underscore f underscore view should be table name underscore b or v underscore table name materialist use table name underscore mv so these are all the naming conventions these naming convention will be slightly different from project to project so but this is the standard one you can follow in your project also kindly practice this session we will meet in the next session welcome to the informatica power center tutorial session in this session we will see the flat file to table note watch the complete session without skipping and subscribe the channel for more updates as well as look at the playlist for all the videos happy learning in our previous session we have seen how to load the data from a relational database table to a table so that we have already completed so if you haven&#39;t seen our previous session so kindly check in our playlist it will be there you can practice and then you can practice this this current session so there are different types of data load so first load we have seen already the table to table so we have one table in hr schema that is called employees then we have one table we have created in course schema that t underscore implies just we have loaded the data from a table to table here so now we are going to read the data from a flat file and going to load the data into a target schema so you assume that our customer our third party vendor our customer is placing the file the secured directory so from the directory we have to read the data and then we have to load the data into a table so file to table load then next session we will see a table to flat file load so current session we will see this flat file to table load we will create on file so you can execute this whatever the select statement you want to have just you can execute and export the data the format will be you can mention that csv format the first option will be csv and you have to save this file in a folder so where i have to save this file so you have to save this file in this particular source file directory you can go to c drive informatica 10.2.0 server infrashare and this is the common path this is the root path you can see here we have the backup files bad files cache files lookup files and session log directories and source file directories and all these directories will be there in real time so this c drive will not be there but from here server in fashion it will be available you can go to the src source file directory here we are going to keep the source files so i&#39;m keeping all this file inside one folder and here i&#39;m going to create a file so just go here and go to the file you can see i&#39;m going to name it the file s this is the file name i&#39;m going to have so click on browse it you can provide this name click on next and go to the finish you can see here the file should be created here so this file we are going to take a source now so since here we are creating the file i&#39;m just opening the file you can see here the data will be there so everything the data will be a comma separated file you can see here so from this file i&#39;m going to read the data so first data i&#39;m going to make as nic it academy so just i&#39;m making that just i want to show you this data is this particular session is reading the informatica is reading the data from this particular file so this is the source file and the file name is source underscore flat file underscore implies so from this file i&#39;m going to read through informatica and load it into the t underscore employees table so this table this table i&#39;m going to load so go to informatica now so we have already seen how to so what are the different components are there in informatica just you can go here source analyzer if already source definition is there you can right click you can clear so don&#39;t delete any source or target definition if you want to delete it you have to delete it will be deleted if you are deleting it it will be deleted from all the mappings wherever it has been used so before deleting it be careful you can go to the sources so this time we are going to take the data from a file so you can give the second option import from file so so you need to have the so select the file type as all files then the file will be available you can select the file and click on ok all the information i have given in the descriptions you can follow the description also you can go to the you can check here the file type the file type is delimited file are fixed with the file so this is the delimiter file so sometimes customer will place a fixed width file also each and every column will be a fixed width so that we will see later point of time but now we will we are taking the delimited file you can mention this delimited file and if the file contains a header you can enable this option so we are informing informatica that okay the first line should be a header so you can check this option then informatica understands first lines first line is a header then go to next one so here you have to provide the com here you have to provide the delimiters the delimiter here is comma so if any other delimiter is there you can mention this delimiter in the other option so since here we are using comma delimited file you can mention the comma if you look at the difference if i&#39;m enabling the comma so automatically the fields are separated we can go to next since we have double quotes in the file it is automatically taking the text qualifier as double quotes you do have seen here the double quotes is there right so this is a text qualifier it is automatically taking informatica is automatically predicting that double quotes so it is removing the double quotes here you can go to next you can provide all the fields data type so this is very important step here the first one employee id you can provide the employee id is a numeric and it&#39;s a three digit if you are thinking in in future the length as precision may vary then you can provide here the maximum length and precision you can give here and first name it&#39;s a text so if you are thinking in future it may differ more than 14 then you can provide now itself then here the last name is a text email also is a text and this is a phone number so this is what we will do lot of mistakes here so kindly make a note of it since i have two dots i have to mention this is a text field if you forget to mention this text field then session will fail then next one is date column so date column should be very careful you have to give so this is a date and time the format you have to note down so what is the format dd iphone mon iphone y so this format we have to give in the properties i will show you where to give the next column is job id it&#39;s a text column you can give and salary set numeric column the next one is commission percentage so that is also numeric only so you have to provide numeric and the next one is manager id also numeric department id also numeric so normally this source definition this ddl definition will be provided by the client so we have to get clear the requirement so what are the maximum length of the column that we have to provide here so finish finish it then you can see here the source file definition has been created you can see here the file the source definition name and the type of this source flat file the target since we have already taken this t underscore employees i&#39;m not going to import once again and go to the mapping the third step you can follow here so go to mapping the third step is creating a mapping go to mapping designer mapping create a new mapping m underscore yes underscore source underscore flat file underscore implies just i&#39;m creating a mapping and go to the mapping here mapping has been created just drag and drop your source from the flat file so you can drag and drop the source from here and target from here right hand side you can drag and drop source always should be in the left hand side target should be here in the right hand side you can just click on select all you can connect it control yes the mapping should be valid and if you want to arrange it you can arrange it here arrange all it will be arranged clearly or arrange all iconic it will be there on iconic so if you have lot of transformations you can put easily to understand you can put iconic r can go for arrange on so it will be there this way so before creating a session it should be valid so it&#39;s as up now it&#39;s valid you can go here we forget to provide the source properties date and time go to the properties here you can see here date and time just you have to provide dd iphone mon iphone y so this format whatever the format the source data format that you have to give here apply okay save it again it will be updated you can create the workflow and everything so go to workflow manager create workflow wf underscore source underscore flat file underscore employees the naming convention just i&#39;m giving to understand what is this you can create a session by clicking on the shortcut icon session icon so here look at here so here you can click on the session icon you can keep it here so informatica will ask which is the mapping you want to associate then i&#39;m just selecting this mapping source flat file click on ok so click on link task and connect it double click on it go to mapping source now it&#39;s a reader it&#39;s a file reader so file reader will not have any connections here so it gets the file so you&#39;ll be having a path so this is the source file directory so here it is mentioning like dollar pm source file directory so this file is a parameterized now so this is nothing but this file path is nothing but so this is the path so where it will be defined in the informatica admin console it will be defined so you can check here admin console if you open so there it will be mentioned like this is equal to this one so that is why informatica is understanding that okay this path is nothing but this path and then source file name you can put the file name if you want to create any folder and then you want to keep your your own file over there your project file over there then you can create the folder here so that folder you have to mention here for example here one is there if you want to keep the file in one then you have to put the folder name is one one so here so that is what the changes you have to do but i&#39;m not doing it here directly i have kept the file in source file directory source file name you have to mention then go to target the target you can see here the connect the target table is pointing to source so this is what we will do lot of mistake here your session is getting failed because of connection error so you can table our view does not exist that particular error you can see whether you have pointing the target table correctly or not you can go to this particular table you can bind to the target and you can go little bit down you can do truncate target table option it will truncate the existing data and it will load freshly i&#39;m just starting the workflow so it will be running here so i have just started our flow it is running you can see here get run properties you can see the source and target statistics it should read the data from a flight file to table so this is the flat file and table 107 record has been loaded so we can go to the target table you can see here so this data has been loaded in the table so this is what you can practice so kindly practice the session so in the next session we will see different concept thank you for watching kindly practice the session and we will meet in the next session with another topic thank you for watching welcome in this session we are going to see table to flat file load so in our previous session we have see we have seen table to table load and the flat file to table load so now we will see the table to flat file load consider we have the table here in hr schema in the employees table we are going to fetch the data from here and we are going to generate the flat file so we will do lot of processing and then we are doing it in the we are loading it in the flat file so how to take the data from a table and how to load the data into flat file that we will see now go to designer the designer we can see first source analyzer the source will be a table and we have already taken this table so no need to take again so we can consider this table to this table to flat fire we are already table here only to take again in target and go to the target target designer just we will create it how to create so i can go to the target we can use the file structure if i have any file structure we can import but since we are creating a target file we will not have this file structure here you can use create and you can provide that flat file and you can mention the target flat file name and create but you have to provide each and every columns but instead of that since the file type is in this format t underscore employees table format just then selecting this table ctrl c ctrl v just to make a cup i&#39;m making a copy it&#39;s asking do you want to copy this target yes i have to give yes since the target table name is already available we are getting a conflict you can see here you have to give rename and you can apply here edit and you can provide this is the flat file target underscore flag file underscore employees just start giving the name of this target click on next finish it you can see here the one of the target has been created if you look at this this is also a table only oracle database only you can see here it&#39;s a table can i convert table definition into a file yes double click on that you can go to a database type instead of oracle you can mention flat file so click here flat file so you can use this flat file definition go to advanced what is the delimiter you want to have in the target you can provide the target definition delimiters and if you want any optional quotes that you can give click on ok apply okay here so we just created copied from this table structure and we have created a flat file so this is the oracle table so this is a flat file just to save it source and target has been created now go to mapping here i am creating a new mapping called m underscore target underscore flat file underscore employees just time providing the naming convention you can drag and drop the source the table employs table and target is a flat file you can keep the target in the right hand side and click on employ id select all the columns keep the cursor here drag and drop from here to here so it will be dragged and draw so you can if you have wrongly connected you can select the columns you can select the pipeline and you can reconnect again so ctrl s it should be valid the mapping should be valid before creating the session first three steps we have completed we can go to the workflow manager the workflow manager go to workflow designer create a new workflow create wf underscore target underscore flat file underscore employees so this is the employees so file name just i have given the workflow name you can click on the session you can select this particular mapping m underscore target underscore flat file underscore employees and click on ok then click on the link task you can connect it so double click on the session go to mapping you have to find the source to the source connections target is a flat file writer so target will not have any connection since it&#39;s a file you have to provide the path so where it should be created you want header in the file yes if you want to have the header then you can provide output field names you can go little bit down you can see here so output file directory so this is the target file directory so this file directory is nothing but if you look at here so previously we have seen that source file directory now the same in first at the root directory we will have one more folder is called tgt files so this is the same structure we will use in our real time also i&#39;m just deleting all the files from this folder i&#39;m keeping the empty folder so it is not necessary to have empty folder so you even you can create one more directory here and then you can place your file so just i&#39;m loading the file into this target directory so so this is the file directory here so dollar pm target file directory is nothing but so this is the one which we have already defined in the admin console so target file directory is equal to this one so this is what we have already that admin team will point at this particular parameter value to this particular path then informatica knows that it will decode that particular path okay so what is the file name you want instead of dot out i&#39;m making the dot csv file just to make that csv apply here you can see the set file right so this is the file property you can see the same file property will be there delimited file go to advanced you can see the column delimiters so whatever the delimiters you want you can place this delimiter here click on ok just i&#39;m not doing anything here cancelling it apply okay so click on yes so if i&#39;m going to run this workflow it will create a file and then it will load the data into this particular file directory so just it&#39;s running now you can see here the data the workflow is running you can click on run properties you can see the source and target statistics here so 107 record has been read from the source and returned into the target you can go and check the target file directory target has been created as a csv file you can right click and open the file and then you can see so this is the file it has been created so always the file will be created with hash symbol in the beginning so we can remove so how to remove this has symbol from the file that i will we will see in later point of time so just now if just we have seen that the file has been created you can see here the date and time format since in the source we have only the date we have given the date and time format here it is loading only zero zero here all the zero zeros but if you are thinking that i need only date the format we have to defend where we have to define the format the target so remember the target definition you cannot change it here so you can change here but the port you cannot change it here the properties you can change but both you cannot change go so always if you want to change anything the source source and target definition go to the source analyzer or target designer so here you can go to the properties or port you can change it so just i&#39;m going to the properties i&#39;m just changing it so what is the format i want i do not want this hours minute second just removing it i&#39;ll have only mmddy by the way format so i&#39;m going to save this just i&#39;m going to run it no need to refresh it why because i haven&#39;t changed anything on the mapping level just start workflow the workflow will be started it will replace the file so always it will replace the file it will overwrite the file you can see here it has been whether it has been completed or not it&#39;s running it has been succeeded you can go here the file has been updated the timestamp has been modified and see here the date alone has been loaded now so this is what you can do so you can ask question if i want to append the data in the existing file then what i need to do so we have one option called the session level go to target definition so here you have one option called append if exist if the file is exist you can append the data in the existing file just click on this even you can create a directory here so that the directory is not present that you can enable it so this is the one you can enable it so just i&#39;m not enabling just i&#39;m giving that append if exist so click on it so how many records are there in the file so one not someone will be there after making that upend file how many we will check so here i have one not seven records including header okay just i&#39;m closing it i&#39;m going to after enabling the properties up and if exist the the workflow is still running it has been executed you can see here the data has been appended you can go little bit down so previously we were seeing only up to here we have seen so now again the same data has been appended into the file so 107 or not 7 you are getting that 214 including header 20215 so this is what you have to create the file in target table so if the customer is asking that the downstream application is asking different delimiters you can provide the delimiters in this session level so up to now we have seen table to table flat file to table and table to flat file load so kindly practice this session at least two or three mappings you have to create for each and every scenarios you have to practice so then only this tool will be much practiced this tool will be very easy to you and kindly practice it thank you for watching we will meet in the next session the transformations are two different and transformations are available one is active transformation passive transformation so first of all what is transformation right the transformation used to transform the data so from one format to another format it is used to transform say for an example if you take this is the transformation if you take this is a transformation see whatever we are doing by using sql or anything this transformation is used to transform the data from one format to another format see transformation you can give n number of input input port this is called input port will be having different number of ports here then you&#39;ll be getting the output port here so what the transformation will do transformation will transform the data from one form to another format if you look at here this is the transformation so this is the transformation if you are giving 100 record to this transformation you are doing some filtering out or you are doing some rank or you are doing some aggregation or anything you are getting different number of records from here out of this transformation is changing the number of record if the transformation does change the number of record password pass through it then that is called an active transformation and also if the transformation does changes the row number of the record so first record it&#39;s coming like a 10th record here but the number of record is same but the row number of the record is getting changed then you can go for the then you can call this particular transformation as an active transformation okay so this is an active active is nothing but so whenever you are having the transformation the transformation does change the number of record pass passed through it that is called an active transformation okay if the transformation does not change the number of record passed through it you are getting you are giving 100 records you are getting all the 100 records out of it see some transformation something like you are calculating salary or a salary increment or you are calculating some bonus or you are calculating for tax see this these calculations it will be applicable for all the record rate so whenever we are going for all the records then we have to go for the we can call this particular transformation as a passive transformation if you look at here this is what an active and passive active and passive a transformation does change the number of record fast through it then active a transformation does not change the number of record passed through it that is a passive okay and also we have one more uh difference like connected and unconnected unconnected transformation what is this connected and unconnected see connected means the transformation will be then pipeline so pipeline means it will be connection some transformations we have that will not be in pipeline something like a stored procedure or so different transformation we can go for lookup we can go for unconnected unconnected means it will not be there in pipeline okay simply it will be alone standalone it will be uh looking so this is called the connector connected means the transformation will be in pipeline okay that is called the a transformation will be in pipeline so this is called connector and unconnected means the transformation will not be in pipeline so separate it&#39;s a standalone so that is what unconnected you can see here see we have we are going to see some 17 transformations 17 transformations so either it will be connected or it will be unconnected whether it be passive or active we will go with very first transformation is called source qualifier so what is the source qualifier transformation right the source qualifier transformation is nothing but it&#39;s a very first transformation so this is the whenever you drag and drop a source into a mapping you&#39;ll be getting one transformation called source qualifier so what is the main advantage of the source qualifier what is the primary task of source qualifier it converts source data type into informatica native data type that means your source might be your oracle sql server db2 flat file sales force or anything right the source might be from anything any different data sources see look here this is an oracle so oracle we have varchar2 right but in the sql server we don&#39;t have any error in teradata we do not have anywhere to simply aware only so whatever the data type we have it converts that into a string here okay string number it will convert into a decimal date will it will convert into a date and time so i have given the explanation here you can go through it it converts source data type into informatica native data type source qualifier is an active and connected transformation it&#39;s an active under connected transformation source qualifier has many properties if you look at here these are all the properties if you are not writing any properties you are not overriding any properties in source qualifier it takes all the record from the table with all the columns you how it will write the sql statement select all columns from the table name this is what it will fix select all the column names from the table name this is what it will fetch i will show you but we have many properties we have many properties you can see here the properties are like first properties are sql query then user define join see first one by one we will see the first property is like sql query if you double click on the source qualifier right you can find the properties here go to properties the properties you can see first one is sql query then user defined join source filter number of sorted port tracing level it is a common for all the transformations i will tell you what is this tracing level then distinct pre-sql post sql output is deterministic and you can see here output is repeatable so these are all the one you can see these properties are mainly used the first seven to eight properties are okay first i will go to the sql query what is this sql query see here if you if you don&#39;t write anything on the source qualifier it will write select star from the table name star means all the columns column names it will mention so that&#39;s why if your source team making more columns your informatica will not fail select all the columns from the table name that is whatever sql query we can overwrite say for an example in the source table normally it will have more data right not only one year data you have two years three years all the data will be there in the source but we have to take only whatever the data we want how can i take so that&#39;s what we have to go with sql query so by writing sql query we can restrict the data from source itself by writing sql query we can restrict the data from source itself that&#39;s what sql query okay next one is user defined join see here previously we haven&#39;t joined anything just we have taken one table and loaded into a table but we can able to join but the thing is here you we have to join homogeneous child i will tell you what is this each and everything we will see clearly so that you can understand what is this user defined join means homogeneous joint homogeneous means tables from same database tables from same database that&#39;s what homogeneous join then source filter source filter means for an example here you are applying some filter condition in the sql query so instead of that you can go for the source filter that filter condition you can write it here any filter condition look at here salary is not null something we are writing right so this is the so those who are getting salary it will be it will be picked up from the source country equal to india okay state as equal to active something like that if you want to put some filter conditions you can put the filter condition on the source filter select distinct it&#39;s nothing but enable or disable that is a full row duplicate it will select say for an example in the source you have the full row duplicate see everything i will show you practically we will see then you can get to know it will remove the full row duplicate pre sql and post sql so what is the pre sql and post sql right pre sql means so this sql will be executed before this particular sql okay see this sql executed to fetch the data but before that i want to execute any sql then i can go for the pre sql post sql see after data fetch i can go for the first scale that means see you can do the index creation or you can do the correct stats so all this you can go for the pre-scale and post sql then number of started port number of sorted report means you can do the order by class now i will show you what is this okay if the source is flat file if the source is flat file all of the above properties are disabled okay remember this if the source is flat file all of these source qualifier properties are disabled except the tracing level why because if the source is flat file you cannot do anything see whatever you are writing here so everything is like yes sql query one sql go only that we will see now we will see one by one okay first of all we will execute the table we will execute one flow just i&#39;m doing table to table look at here going for create m underscore source qualifier underscore transformation just by making the transformation i will take sources a table employees table t underscore employees table whether i&#39;m doing any any changes on source qualifier no if you double click on this there is no changes just i have taken as it is control yes then it should be valid you can right click and go for arrange all right then go to the workflow manager you can go to the workflow manager clear create workflow wf underscore source qualifier underscore transformation we can create a source qualifier transformation then we can take the this particular mapping and connect it double click go to mapping source should be pointing to source target we are pointing to target just we are taking the data from source table to target table without any transformation after that we will do some some transmission logic okay i&#39;ll go to the i&#39;ll start the session it will be started and it will be loaded into the target table right it&#39;s a target table it will write how many data it will write see whatever it is going to read from the source it will write you can see here or not seven record read from the source and one not seven record return into the target so okay now we will analyze the session log see session log analysis is very very important i will show you some other things in session log you can able to see three threads three thread what is the three thread so one is reader thread first one is reader thread second one is transformation thread see transformation thread reader thread it will read the data read the data from source transformation thread it will transform the data from one form to another format if you do not have any transformation logic then transformation thread will not do anything it will just pass the record then righter thread writer thread will write the data into a target table see wherever you are getting any error either read a thread or transformation thread are right at first so there you have to concentrate on it see if the source itself we are not able to get the data then there is an error in the source qualifier right so if you look at here it will show you the information and then timestamp node so which node it is running what is the thread it is running see whenever you are seeing the director or manager are mapping right these are all about the informatica level something you are seeing that reader threat reader so it&#39;s a source side writer is nothing but target site then message you can able to see here the messages like so what is the date time format and everything it will take and if you go little bit down see whenever you are getting some time say please go through one by one all this dtm you know right dt means data transfer manager you can go little bit you can see all this right at that right and thread source side will be having the look at here target writer thread say insert into right insert into table name all the column names then values will be there values question mark question mark means run time it will it will load the values see if you want to copy this if you want to copy this you want to make it like a google or something you want to google it then you cannot copy from here here you cannot right click and copy from below you can copy it from below you can copy it okay if i want to copy below only how to copy okay let&#39;s go little bit down you can see here the source qualifier query right sq instance means source qualifier instance yes q employees source qualifier query you can see you can copy this query so what is this query look at here what is this query read rr means relational reader sql query issue to the database first row returned from the database that means symformatica can able to read the data establish the connection right or not sound record right see this what you can able to see and uh what is that end the loading the target table it has been in depth and load summary it is showing how many they cut it from the source how many record return under the target everything you&#39;ve been able to see then it will complete session completed without any issue you can go and check in the source qualifier the query so where you have to execute source qualification you have to execute it in source site you got executed correct so informatica will execute this query in the source qualifier the source database and whatever the result you are getting right so that result will be picked up from the source if you look at here select all the column names from table name from source table see how it will be how it is writing select table name dot column name table name dot column name see why it is it is mentioning this way right if you are writing this way the performance will be improved instead of mentioning like select column names along employee id first name alone if you are mentioning like table name dot column performance improved instead of writing star if you are mentioning the column names so that is also will improve the performance okay and also if the source team they&#39;re adding more columns it will not impact on our informatica load see this is what okay it will fetch the how many recording fetch you can see how many recording will fetch all the one not sound records so this is what informatica will pitch the data from the source okay see instead of this i can go for somewhere class where i can write any class okay so where commission percentage is not null where commission percentage is not now that means who are all getting commission i&#39;m picking up is not null not null means value should be there that is a not null see just i am writing the not null here okay okay so just time executing it out of 107 how many records you are getting i will go on execute on not seven you are getting 35 records okay see even out of uh those who are getting commission and i can go for some other some other condition also the salary should be more than seven eight thousand dollars and salary right you can put condition greater than 8 000 some condition see whatever out of 35 you are getting only 20 records see you can write this sql query based on your scenario based on our scenario you can write the sql query clear on this so now what i will do i will select this query normally this is what we will do we will write the query in sql that is what uh source sql source database and then go to this informatica you can double click and go to properties here i am writing this sql query okay okay apply okay save then you can refresh the session right click refresh mapping and execute it now informatica will fetch only 20 records from the source see look here instead of 107 it will pick only 20 records see whatever the query we are writing execute the query in the source quality source database and then if it is executing correctly then place it here that&#39;s what we will do okay okay look at here it has been executed only 20 records it has taken okay that is what it will go on right into the target table so i&#39;m right i&#39;m saying that if this is the record should be loaded then you can take say for an example you have employees data employees table a particular company it has all the employees who are who left the company or they they went for some medical url you have taken all the records and instead of that you want to take only active employee status record see whatever the condition whatever the query we are writing all the source qualifier properties will be available in session level also if you go to here go here the source qualifier you can see all the properties sql query whatever we have written over there it will be available here i&#39;m overriding this instead of 8 000 i&#39;m writing 9000 i&#39;m overwriting this or 10 000. okay so session level i have written one query but mapping level i have written on query so now you you guess it which which one it will take any idea informatica anyhow it should take only one right so informatica will give reference to session level properties remember this i&#39;m going to run it now it will take different records so that&#39;s why in real time and all if you go and check sometimes what will happen production production database sorry production server we used to take the we we will check it the queries and then we are getting this particular record only and then we are going to check the this query this query here we were seeing some different properties here but the query is running it&#39;s different if you go and check why it is having different they should have written something different in session level so what will happen some new see look at now that only taking eight records right that is what it will take sometimes what will happen in real time under we may face some issues see a center in fix we have to apply status equal to active some some filter condition we have to apply if i want to do any changes in production it should be go with releases releases change liquid change ticket all this right so we cannot change anything in the production asset is so it should go with all the cycles uh development testing and pre-prod and then approval everything it should even one small properties also it should go with but it will take a lot of time your business is waiting for the particular fix to be applied conditions should be applied over there so what we will do we will raise one request we will get the approval from business saying that this is an issue to fix it immediately we will go for session level override we will give that information to our business owners system owners you will get the approval they will say like go ahead once we get the go ahead then we will go to the informatica support team we will assign it to them and ask them to change the properties in session level normally okay so normally we will ask them to change okay put some condition here uh some filter condition you apply source qualifier you change it something like this because of this it is causing an issue something like that we will apply will tell them they will immediately they will apply here they do they do have the access right write access then they will change it they will save it then they will run it okay then they will ask next time you go with the change request see real time always check whether we have something like a different logic in session level okay so now i will remove this i will remove i will apply i will save i will save here if i&#39;m going to run it how many record it will read from the source so it is taking all the record it is not taking 20 records why because 20 records previously you are seeing 20 only after that we have changed the query in session level we have removed it why it is taking 107 now can be able to guess it if there is no query then what informatica will think it will take all the records correct select star from the table name that&#39;s why it is taking 197 okay remember this these are all interview questions they will ask okay so now i what i will do i will revert it back to the previous one that 8 000. now it will take i have remitted okay now it will take this is called source qualification so what i will do i will remove the source qualifier for you even you can write whatever the queries you want to have you can ask me can i write something like a giant query yes we can write but whatever the column you are taking from the join query so that column should be there out of this sql out of the source qualifier that column should be there say for example department name you are taking the sql query that department name should be there from here okay so that is why we will take a source qualifier query and what i will do now i&#39;ll go for simply instead of this much query it&#39;s very simple condition right simple conditions then that condition i can write here source qualifier filter source filter i will apply something like job id say for an example job id equal to sa underscore sales representative okay job id equal to srf i will write apply okay save and i want to refresh it refresh mapping control yes so you will check it whether it has been that source qualifier query is there or not so there is no query right okay now i&#39;m going to execute it you can see here so 30 records now if you go and check the session log so informatica will write that session the query okay you can see here from employees where employee dot job id is here we haven&#39;t returned this query we have written only the filter condition informatica will take the filter condition it will write the sql query so this is called sql source qualifier filter source filter so you can write any filters the same way you can try to overwrite the filter condition on session level also okay now i am not writing any filter conditions just simply removing it i&#39;ll go for the number of sorted port what is the number of sorted vote means say for an example i am writing one one means first column what is the first column employee id see while taking the data from the source based on the employee it will start it order by on employee id order by on employee that&#39;s what it will take two means employee id comma first name not second column alone if you are writing number of sorted put is two it is one comma two three means one comma two comma three that is an order by class i have explained order by class in our sql please go through that sql order by class okay if you look at here it will take same data only but it will be on the order by if you go and check the session log you can find the order by class you can get session log you can see here there is an order by class right you get session log you can see here it will write order by class okay so from employees order by employees.employee okay employee tooltip why we are writing the same order by so in order to improve the performance we&#39;ll go for this order by class from the source itself it is order by then its performance will be improved that&#39;s why see i i am writing 2 here number of sorted vote is 2 it will not go for second column alone it will go for employee id comma first name see in this case in this case our real in this scenario since employee id is primary key even if you are making 1 comma 2 that is same only why why because first column itself it is a unique column unique values right it will not go for the second one if you look at here i will write i will show you in the the source level so that you can able to understand select star from employees order by where per order by okay order by one one means first column this is employee see here i&#39;m making two two means second column alone i&#39;m adding so it will make second column but informatica how it will write like see if you are making two one comma two it will make one comma two means first column if you are getting first column duplicate value then it will go for the second column since it will not have duplicate value it will not go to the second column okay informatica if you are making like number of sorted for this two it will write one comma two only look at here i&#39;ll show you get session log you can see here look at here order by employee id comma first name that is the weight will write do not write order by first name alone clear on this what what is mean by number of sorted port next one is called user defined join can i get a data from other table as well yes you can get it but if you want to join it by using the source qualifier both table should be on the same database either from the oracle or from the sql server so only one you can go for n number of tables not an issue but all the tables in same database same database okay all the tables in same database look at here i will go to the employees table look at here i have the employees table here right see this one i will take employee stream i want to know which department she is working for which department she is working for nancy its department so department id 100 do you know which department it is if i want to know department id what you have to do you have to look up say how we will do the vlookup in vlookup in the excel right the same way we are to do the giants here giants senior joints departments 100 is finance department where it is located 1700 thousand seven hundred where i can check okay so department is finance department located at thousand seven hundred thousand seven hundred which location i have to check it in the locations table 1700 location is so washington us so that is what you have to do joining okay first of all you will join this employees table and departments table for each and every employee we will load the department name we&#39;ll go here so how to do the join say for example this is what our employees table employs table department&#39;s table employees table department table both are in oracle so that&#39;s very important okay both are in oracle both are in sql server i either both are in an article or a sql server or teradata or anything but everything it should be a same database it&#39;s a department&#39;s table department so what is the table here is an employees tables employees teams okay the condition is both should be on same database okay so now you can take the column names like employee id so you have all the employee id i i hope you should have already gone through this employees table you have what is the columns different columns are there department up to department you have in real time also we have to analyze that table first whenever we are going for any development we have to analyze analyze it all the tables what are the different columns are there primary key foreign name department underscore name and location id location id all this okay here i have the target table here i have the target table see this table you need to create it you want to create it i have given the syntax for this table creation look at here i have given the syntax by using this you can able to create the table okay i have added two more columns department name and location id in the existing table like from this t underscore employees table i just copied i have created like t underscore employees underscore department there i have added two more columns so i will this is the table name target table so this is the target table this table contains these columns like employee id first name last name uh some different background so i am taking all this basic concept for the joining okay see this is what the columns i have so for each and every employee i want to find out which department name they are working which which location they are from location id alone i have to bring these two tables column into one single table see here you&#39;ll be having one source qualifier right for this you will be having one source qualifier see remember whenever you are going to do source qualified china so how many tables you are joining no no not an issue you can have one single source qualifier so you can remove all other source qualifiers you can remove all other source qualifier you can have only one source qualifier common for whatever the number of sources you are using so even you are using three three sources four sources even sometimes you use ten different sources so stable but everything should be at same database remember this this is called source qualifier right in this source qualifier you can join it first of all these columns from first table then department id and department name and location need these columns only so what you have to write simply you have to write the join condition join condition is employees dot department id department underscore ide equal to okay departments dot department id so this is only simply you have to write departments dot department head you have to write only join condition not a query it will write informatica will write that query even if you want to write a query you can write in words sql override that source qualifier override you can write but no need to write simply if it is a giant condition or filter conditions you can write it here say for an example i want to load only finance department what you will do this is joint condition this is joint condition after that you have to write filter condition you have to load filter finance department in the filter condition you can write any filter condition you whatever the condition you want say for an example i want to write department name equal to finance department finance department so that you can write it see the above one is called join condition the below one is called filter condition you are filtering out see only the employees who are working in finance department will be loaded department name location id department id say department id you can take it from here also clear on this okay i will go to the mapping i will create it now you will get to know so m underscore source qualifier underscore joiner first table what is the first table employees table so what is the second table department table is that department stable now no i have to go here first department stability controller you can connect departments table this time you have to take the buttons table also at the department&#39;s table this is the one you can it so departments table also see this is the source the source side so what is the connection here it is a database level connection okay database level connection not in informatica level so database level this employees table department&#39;s table is there connected with department id employee and all okay then target target which table i have to take see don&#39;t delete any of the source or target tables don&#39;t delete it so instead you can clear it clear all clear all means it will be cleared from this workspace it will be available on the left hand side okay don&#39;t delete it if you are deleting it it will delete it will get deleted wherever it has been used in mapping okay how many mappings you have used already it will get automatically from there it will become invalid mapping clear on this okay see all the mappings are now it&#39;s well right valid if you have deleted then it this will become invalid that is one concept okay we have we have taken the source two sources but we need to have a target go to target bring the target table here core and you can take target t underscore employees underscore department i can take the second table second table department drag and drop so this is the department table see this blue color is different this blue color is different this blue color is atl pipeline that is informatica pipeline this blue color is database level connections okay so we can have only one source qualifier the second source qualifier you can&#39;t remove it delete it so you can have this source qualifier common for both this source qualifier will have department ide from here also you will be having the department id department id department name location id only these are three columns i&#39;m taking i do not want this manager id no need to take it department id okay you can ask me so here itself you have the department id why you are taking one more department id column right so these two columns only we are going to join it department id and department right so that&#39;s right double click go to properties the properties user defined join only write the conditions what are the conditions so just click on it double click on the department id click on ok employees.department equal to department start department id click on ok apply okay and then you can go with the target see here target employee to department id you can take it from here so no need to take the department id one more time you can make department name and location id from here and take it control yes it&#39;s a valid now okay it is a valid noun so this is what you can go for the source qualifier joiner joiner double click go to mapping so it should be source target should be pointing to target truncate and load i will truncate the target table and load it start workflow it will load based on the join it will make inner join it will make inner join okay i will show you what how to do a left outer right outer randall but it will make in the join inner join left outer right and outer door if you want to get clear idea so please watch the sessions on joints i think it&#39;s the fourth session or fifth session i have clearly explained what is giants why do we need this giant soldiers okay got succeeded you are seeing only one or six record so that means one record not loaded why because some some giant condition is getting filtered out okay here you are having it here underscore dept you have the department name and location that you got loaded for an example i want to do left outer join left out design means all the record from the left table matching record from the right table so how will you write left outer join here simply this is an oracle table right either you can write sql query or if it is left outer join the right hand side you can make plus symbol on the bracket like this if you are making plus symbol on the right hand side table then it will become a left outer join okay so if you if you want to know how it is please watch the sql sessions sql join session it is called a left outer join okay so i&#39;ll go here i will refresh the mapping control yes start workflow you can see here it will read from the source itself it will read one not sound records okay see here it has taken one not sound record you can see here there is a null value on the right hand side see null value on the right hand side this record will not be there in the previous one okay null there is no matches that is why inner join it didn&#39;t take but now it is taking okay clear on this i can apply some filter condition also i can go here apply some filter condition source filter see what is the filter department name equal to some department name i can put i can put department name so i can put department name is purchasing [Music] okay purchasing department click on ok ok save purchasing department it will make the giants see without join you cannot you cannot say how many of the employees from purchasing right without join can you say can you load a purchasing department alone employees alone to the target no joins only you can able to load it that&#39;s why here you can do the joints okay so look at here it&#39;s running this time it will load only purchasing departments only six records six employees are from purchasing department without joining it you cannot able to understand you cannot able to find it so that&#39;s why it is called source qualifier joiner okay clear on this so what is it this select distinct what is select distinct say for an example here i have one table called select star from employee slide select star from employees underscore one employees underscore one this is same table empty like same same like employees table but i have added one two records with duplicate value look at here this is one record one record this column is different last name column is the front this will not be taken as duplicate but here it will be taken as duplicate full row duplicate okay this is full row duplicate this is not a full duplicate so if you are selecting distinct informatica will remove the full draw duplicate only this one only okay if you look at here look at here this one source qualifier transformation i will go to the source qualifier transformation here what is a table we are fetching the data employees table correct employees table if you go a little bit down the last you can see one option called source table name source table name here the source table name i&#39;m overwriting it employs underscore one do you know which table it will take now data whether it will take employees table or employees one employs one but that mapping we have created for employees right mapping it has been created for employees what is the predefined conditions the condition is all the columns should be matching with employees see employees one and employees one sorry employees and employees one should be matching see now i will take this employees underscore one if i&#39;m taking that employees underscore one year it will read the data from employees underscore one table only source table here i have written some different table it will not take from employees table it will take from employees underscore one clear on this okay how many cards it will read one not nine record so one not nine we have right or not nine so one not nine record it will read from the source i have to apply some distinct if i&#39;m going to apply distinct 0.9 if i&#39;m going to apply distinct here select distinct how many record it will read or not eight why you are not eight this is not full row duplicate this only full draw duplicate see from the duplicate it will remove one record you will get one not eight record even target also this is target see instead of loading into a t underscore implies i want to load it in 30 underscore employees under history table you can mention the table name here sometimes they would have mentioned that table name differently here okay some history table they will load it see they will take a mapping session and everything copied source qualifier query they will write some different queries target table they will change it here that&#39;s all then it will go and load it into the history table that is what they will do normally see pre-sql post sql before reading the data you can create an index after reading the data you can remove the index that we can go for three sql post skill so this one you leave it i will teach you separately okay but other properties you practice it now it is fetching how many records one not eight instead of or not nine it will remove full draw duplicate hello friends welcome to nik iit academy this is chandra shaker welcome to the informatica power center tutorial session let us move forward and learn filter transformation today watch the complete session without skipping and subscribe the channel for more updates as well as look at the playlist for all the videos happy learning filter transformation is an active transformation and a connected one so why we are calling it as active right so your filter transformation it will filter the number of record pass to pass through it so it will it will filter out right so we are giving 100 records we are making some filter condition then it will filter out so source qualifier itself via the filter so this filter transformation source qualifier filter is used to filter out the record from the source itself only source after source anywhere in the pipeline you want to filter out then you have to use filter transformation filter transformation is used to filter out filter out the record anywhere in the pipeline so that is what anywhere in the pipeline if you want to use it you can go for the filter transmission so it&#39;s very simple transformation it is used to filter out the record that&#39;s all for an example this is the filter condition the properties you&#39;ll be having the filter condition by default it will have the condition like true true means it will pass all the record from the filter okay it will pass all the record from the filter if it is false means it will block all the record okay it will block all the records it will not pass it the condition if you are making any condition the condition passes only the condition satisfied record so that alone it will pass then it will unsatisfy the condition unsatisfied record it will block so that is the use of filter transformation for an example you have salary column you are the salary column you have salary like 10 000 50 000 null value you have 8 80 000 6 000 12 000 i have so i&#39;m making one condition like not of isn&#39;t love salary so what the filter will do filter will pass only these salaries and it will block the null values why because we are making not of is nala is means it will pass only this record null value but not optional it&#39;s opposite so that is the one and also so why do we need this filter so anywhere in the pipeline if you want to filter out the record you can go for the filter transformation you assume that i have a flat file and it has some record okay it has some record with all the countries okay i have i want to filter out i want i want to load so india&#39;s data alone okay india&#39;s data to the target if it is a flat file can i use source qualifier no if it is a flat file you cannot use source qualifier filter why because for the flat file source called source qualifier properties are disabled one you cannot use it so that is the reason we are using source and source qualifier you cannot do anything on the source qualifier you are going with filter transformation in the filter you can make country equal to india under saturday equal to 50 000 something some condition you can make it if the country is coming like india this india and this india something like three different format it&#39;s coming you can make the source qualifier filter something like lower off you can use like this lawyer of country equal to india so you are making that uh lower country and you are making the lower case understandably greater than 50 000 something like that you can make it it will pass only that record now we will see practically so what is the use of source qualifier what is the use of filter transformation so i will go to the folder here go to the folder i can use the source is a flat file i can use the sources of flat file just we have already taken this flat file this employs flat file and a target i can use this t underscore employees table flat file to table i am going to the mapping create a mapping m underscore filter underscore transformation just i&#39;m creating the filter transformation so taking this flat file and taking this target but so in the flat file if you look at here the flat file it has some the commission percentage is another column it has some null values and normal values those who are getting commission those employees only i have to pass it to the next level this target table those who are getting commission that means this column should not contain any null value so that record alone i have to pass it can i use source qualifier properties here no i cannot use it why because source is a flat field if it is a flat file all the properties are disabled one all the source qualifier properties are disabled one but i have to filter out how can i filter i can go with filter transformation so this is called filter transformation you can click on the filter transformation you can keep the filter here drag and drop the all the columns to the filter transformation here you can double click on it go to the properties you can see the filter condition right true means it will pass all the record but we are we we need to filter out right so what is the condition you can go to the fourth so you can make is so you can make is null of commission percentage that means it&#39;s commission percentage means those who are getting those who are not getting commission it will be passed i have to put negate negate function that is not not of personal law of commission percentage you can ask question how can i find what all the different functions are there you can go to the functions okay lot of functions are there all functions are character functions right you have all the character functions you are the instinct substring l trim altering all these are like same like oracle single row functions right by using this function you can do any uh logic here okay any filter conditions here so you have that data cleansing operation conversion function then you have the date function so all the date functions are there so different functions will be there numeric functions are there right all the numeric functions and you can go to all function you can see all the functions by using these functions i have taken this null of right so you can see this number of state of spaces of islam right this one i have taken so below itself you can see the explanation is null of expression as expression so that is what you can see if condition so everything will be there by using this you can derive your logic okay validate it it should be valid click on ok go to apply ok then we have to complete the pipeline i can select all this drag and drop itself like columns are matching with the target columns just to save this it should be valid it should be valid mapping should be valid then only you can able to create the session i can directly go to the workflow manager click on the workflow manager you can go to close this existing one you can see here go to workflows create one wf underscore filter underscore transformation this filter transformation right you can create a workflow this is a session icon click on this you can choose the mapping corresponding mapping click on ok the session will be created click on the link task link with start button double click on it you can enable this properties right fail parent if this task fails that means so this property we will enable for all the sessions why because if the session is failing corresponding the parent or blue also will get failed automatically and go to the mapping you can find the source source is the file reader now so we are reading the data from a file that is a file reader for the file you will not have any connections you&#39;ll be having a path so in this path we have to keep the file right source underscore flat file underscore uh implies underscore csv so target you can check target is relational writer you can provide the connection here you can go to down you can if you want you can truncate target table and you can give and before that we will check the file okay so since we have taken the file definition we will check the file this is the source file directory right so dollar pm source file directory this directory is nothing but so this is the source file directory i&#39;m going with source file this is the file i can open the file you can check here what is there okay here we have used to deliver is some different delimiters so dollar hash and percentage this is a delimiter we are using so same delimiter we will check it here whether the same delimiter is available source we have seen this concept in our file to table load itself so that is why i&#39;m taking directly here delimited file advanced go to column delimiter you can put this dollar hash percentage you have to enable this treat multiple delimiters and why because these three delimiter we are making a single delimiter so that is why this option we have to enable first line we are skipping it this concept we have seen already in the flat file to table load just i&#39;m giving that and they can go to the config object make this top on error equal to one so what is this top on error equal to one right if any one error occurs it will fail it will fail the session it will not proceed so click on ok it should be valid save it it should be valid now i&#39;m going to start it so right click on the white space anywhere in the white space you can click on start workflow it will start it now the workflow will get started now automatically the monitor will open here so this is the filter transformation you can go here and go to the source and target statistics you can see only 35 record has been loaded so 107 record right from the source actually and then it is filtering for an example the same filter condition if you have applied on the source qualifier if it is a source is a table then you&#39;ll be having statistics like 35 and 35 why because informatica will filter out record from the source itself since it is a table but here since it is a file it will not filter out it will fetch all the record into informatica then in the informatica cache memory it will filter out the 35 and will it will be fetching that if you go and check in the database you can see you can open this core for course schema so just i&#39;m connecting to the core schema you can check whether it has been loaded with those who are getting commission that is what we have applied the filter condition right so that i&#39;m taking so go to select star from t underscore implies this table i&#39;m checking if you check here in the employees table you&#39;ll be having those who are getting commission it will be loaded here okay those who are getting commission even you can put one more condition also like those who are getting commission and salary should be more than 10 10 000 so where can i put go to filter the filter you can go to the properties you can put under condition and one more condition also i can put go to port salary greater than 10 000. like this you can put under our conditions any number of conditions no issues apply okay since after creating the session i have changed the mapping i can refresh the mapping here refresh mapping ctrl s then run it now it will not load 35 records it will reduce the number of record why because i have changed the filter conditions it is not sql operation just filtering out we are fetching the data from a flat file right for an example your customer is sending the flat file from the flat file you want to filter out that we are doing it that&#39;s all same like sql only but we are filtering out the card if you check this session now it is loading only eight records to the target you can go here you can check the target now so the employees who are getting salary greater than ten thousand dollars and also those who are getting commission so these two condition it is satisfying it is fetching all the records informatica is fetching all the record from the flat file then it is filtering out so this is the filter transformation you can see here this is the filter transformation anywhere in the pipeline you can use this filter transformation so this is the property of filter transformation kindly practice the session and we will meet in the next session with another topic thank you for watching hello friends welcome to nikeite academy in our today&#39;s session we are going to learn expression transformation this expression transformation is very much used in our day-to-day project to implement business logic so expression transformation will use lot of functions to write expressions if you want to learn all the functions i have given the description link in oracle single row functions please watch this session to get complete idea about the functions which is used in expression if you haven&#39;t subscribed our channel please subscribe the channel and click on bell icon so that you you&#39;ll be getting all the notification thank you let us start our session [Music] previously we have seen source qualifier and filter both the transformations are active transformation but this expression transformation is a passive transformation passive means i we have already seen that so how many record you are giving to the transformation the same number of record you will get it in the outside so that&#39;s what we will get passive right passive one and we can see this this is the connected one it is a passive and connected and whenever we want to apply some business logic in our informatica then definitely we will go for expression transformation so whenever we want to apply some business logic in our informatica then definitely will go for expression transformation in expression transformation there will be three port one is input port output port variable port and we&#39;ll be having expression expression is not a port just you can write the business logic okay what is this input port and output port so input port we are getting the data from input and this output port will be enabled if both input and output enabled you cannot write any logic here why because it&#39;s coming like input it&#39;s going right so that&#39;s what we cannot enable any business logic here if you want to write any logic either it should be output port or it should be a variable port variable port is nothing but if you want to do any intermediate calculation intermediate calculation then you can go for the variable port okay here look at the logic just we are getting one column called salary it&#39;s a input and output if the salary is 10 000 it will come as 10 10 000 and it will go as 10 000. there is no changes here but we are creating one more new column called new salary this column is nothing but just we have created this columnist output column since we are creating the output column this this has this will be available only in expression transformation okay since it is output expression will be enabled just i&#39;m making the expression like salary this column i&#39;m just using it if it is ten thousand ten thousand plus ten thousand into zero point one that means ten percentage increment on the salary so saturday it will come say for an example salary is ten thousand so here we&#39;ll be getting ten thousand here you&#39;ll be getting eleven thousand that&#39;s what the new salary what is variable port variable port is intermediate calculation say remember if you are writing any variable port you cannot take it outside see here so this is what one expression transformation you can see this is one expression transformation in the expression transformation we are passing one of the column called salary and salary we are taking its output so there is no changes here but we are creating one more port called newport we are creating new salary that&#39;s what we are creating new salary this is what our what called salary this is what our salary that means if the salary is 10 000 so here we&#39;ll be getting same ten thousand here you&#39;ll be getting same ten thousand but here you&#39;ll be getting new salary say for an example this is what incremented salary or salary bonus or you can we can say anyway it&#39;s a normally we&#39;ll go for o underscore o underscore means output column so normally we&#39;ll create o underscore is nothing but output column v underscore is nothing but variable port so o underscore salary is nothing but we are getting the logic salary plus salary into 0.1 just we are making one simple logic here if the salary is 10 000 the new salary will be 11 000 will be taken from here that&#39;s what if it is variable port if you are creating any port as variable port variable salary just on creating as variable salary this port you cannot take it outside you cannot take this port outside this is what the expression transformation we are having input port this is output port this is also output port just we have created a new port here this is new output port this is output only just we have created output port this is a variable port variable port you cannot take it outside that&#39;s what i have mentioned and see here this is a variable port that&#39;s that that is what i have mentioned variable we underscore variable port a plus b so i am adding two numbers and storing it in the variable called c and if i want to take this variable port outside just assign this variable to output port and if i want to use the same value v v underscore c then i can use it but i cannot use o underscore c anywhere okay that&#39;s what output port you cannot use it for next level but variable port you can use it so remember in expression transformation the order of execution of the port is so this is what first input port then variable port then output port this is what it will apply so remember expression is a passive transformation we cannot apply any filter condition in expression so that&#39;s f it will not filter out any record okay it will not filter out any record look at here this is one simple logic i&#39;m going to apply if the employee is getting the salary greater than ten thousand if already the employee is getting greater than ten thousand dollar then just incrementing by ten percentage it&#39;s already if the salary is less than 10 000 i am incrementing the salary by 20 percentage the new salary okay this is what the calculations how to do it so remember in expression in any of in informatica this is the logic actually if logic iif if is nothing but iaf here if condition you write the expression here condition if the condition is true what i have to do if the condition falls out i have to do that&#39;s what if condition iaf there is no if okay there is no ef here this is called a logical if condition same like our other languages this is what in our informatica also if condition iaf there is no if okay remember this if condition you can go for nested if condition also like if condition if it is true one more if condition you can go with and one more if conditioning can go so this is what we can go for multiple nested if conditioner so definitely we will be writing this kind of logic in our expression transformation okay so this is for just a theory we will go for practically we will see so that you can get a clear idea we can go to informatica just i&#39;m closing that existing mapping and i&#39;m taking the source tables employees table so i&#39;m creating a mapping called m underscore expression transformation expression transformation just you can try take the source drag and drop source into a mapping you can see here so this is what the expression transformation we have and right hand side right hand side you can take t underscore employees this is a target table just we are going to implement this particular logic salary logic and we can load the data into the target so implement the logic we are bringing the expression transformation here just to take this is the expression transformation f of x is nothing but expression transformation we can click one time here we can post it here and then you can place it here and then you can take all the columns select all drag and drop here to here and we can use the column here okay just we can make double click on this expression expression is a passive you can look at here input port output port variable port and expression if the port is coming like input on the output then just it will go from it&#39;s like same like our normal transformation it&#39;s coming like input getting us output here there is no changes there is no changes if it is coming like this but if you if you want to do any logic changes you cannot apply on the same column remember in expression if i want to do any changes you cannot apply on the same column since it is a input so if you want to apply some logic only input you can make only input you can remove the output port that means only input we are getting from this source side okay then you keep the cursor here cursor here you can click on this add a new port to this transformation this icon this icon will add a new port this icon will delete the port this icon will copy about this one it will move the port down or up okay you can see here okay up and down you can keep the cursor here just to create one new port called o underscore incremented salary o underscore incremented salary just i&#39;m creating that oh underscore to denote these this this column we have created just i&#39;m making like lowercase if the salary is decimal oh underscore strategy also should be decimal right decimal so you can go for decimal point if you want to decimal point i told already it should be output or expression then only the expression will be enabled if coming like input and output there is no expression okay it will not be enabled either it should be a output port or it should be a variable port so then only this expression will be enabled so just i&#39;m making like output port there is no intermediate calculation that&#39;s what i am taking that output port go to this icon this expression you can click on this arrow mark this is what the expression window will be opened as as i told previously we can see all the functions are available we will cover most of the function here okay just a look at this if you want to learn all the function very clearly so watch my session on sql single row functions okay it&#39;s like a one hour session i have posted you can watch it clearly that what is this apart from that we do have some expressions are there some more functions are there additionally some functions are it will be changed but that you can learn it from here okay so now i will take this if condition since this one this condition we are going to apply if salary is greater than 10 000 this is what right so i have to write if condition iaf so here this one iaf if you select this you can see this you can find the explanation here just to keep iaf if salary okay if go to port always choose the column from the port don&#39;t type it you you might be doing some mistake okay if salary greater than ten thousand if salary greater than ten thousand comma if it is true then what i have to do greater than 10 000 10 percentage increment so i can make 10 percentage increment salary plus salary into 0.1 0.1 is nothing but 10 divided by 100 right so you can make this what salary plus this one this is a true condition if it is false what is false then you can make comma here then this is 20 percentage you can make 0.2 then you have to close it you have to you have to close the bracket so always make sure that you are opening the bracket and then the closing and you have to check everything validate it always you have to validate any expressions the expression should be valid so just click on ok so this is what and apply here okay if you look at here if you are making any port as input port alone then you cannot take output from here you can see there is no this arrow mark from outside of this expression you could you could not able to see here right so if you drag and drop this column you cannot drag and drop it so in the below it&#39;s showing like cannot start a connector in input only both that you cannot do it but look at here this is output only port right so you cannot pass any input to this port so it&#39;s a look at here there is no arrow mark in the input side so you can take this port to a next level so you can ask me the question can we make this whole underscore salary to salary port yes we can make the column name might be different no issues the data type should be matching okay even you can ask me here it is a decimal why it is number so as i told so inside the informatica everything is like a informatica native data type okay here this is what inside the informatica it is inside we have the native data type after that source and target will be based on the target and source database you can make control yes and it should be valid go to workflow just to create a workflow create new workflow wf underscore expression transformation i&#39;m just creating expression transformation click on ok and click on the session choose the expression mapping click on ok click on the link task double click on it go to mapping source should be pointing to source target should be pointing to target connection and just time truncating and loading the existing data it will truncate and run the road for our academy purpose we are just a truncating and load we should not do it truncate and load always in the real time project okay you can go to the config object you can enable the stop on error equal to one as i have already explained stop on error one means if any one error this will be stopped it will be failed the session will get failed go to general you can make failed parent if this task fails if this task fails fail the workflow so control yes the workflow also should be valid workflow also should be valid here and then right click on the whitespace anywhere start workflow it will load all the data so you can choose whichever the workflow here double click on this r you can go to the get run properties so source and target statistics we can watch it and then you can see what is the source what is the target so this is what you can see here what not sound record or not seven record got loaded it will not filter out any record you can go here in the target we can check what is the data it has been loaded if you look at here for steven what is the salary here in target it is 26 400. if you go and check here select star from employees table this is source table right source table steven is twenty four thousand so twenty four thousand is greater than ten thousand right greater than ten thousand so it will increment only ten percentage twenty four thousand plus two thousand four hundred right two thousand four hundred so that&#39;s what twenty six four hundred right okay this is what it is making then if i have less than ten thousand this is what for alexander we have less than ten thousand here we have nine thousand so nine thousand plus thousand eight hundred right so it&#39;s a twenty percentage increment so what is the salary here ten thousand eight hundred so this is what it should be loaded you can see here we will check it in the target so steven is having like salary like 26 400 and alexander we have alexander we have salary as 20 percentage incremented ten thousand eight hundred that&#39;s what you can check it for all the records it will be loaded with all the records so this is what one simple logic we have applied like if condition okay we do have many other logic just we will see one by one okay the first name so look at here these are all the condition just i&#39;m applying it here the first name in the source whatever the source we have in first name it should be an upper case in target so convert the first name in source into uppercase last name the source it&#39;s like last name we have but in the target it should be like concatenation employee id underscore last name just i have to make 100 underscore king just i have made it like how to do the concatenation informatica okay we will do one by one so that you can you will get a clear idea how to apply some logic okay email column email column look at here if the salary of the employee is greater than ten thousand i have to make the email as gmail.com if the less than ten thousand i have to make email at yahoo.com okay so this is what you can make email here and phone number so phone number is nothing but we have this format we have to remove the dot from the phone number we have to add plus one iphone okay in the phone number just we have to add it and job id we have to take first for from fourth character we have to take java id and the commission percentage we have to take if it is zero you assume that in target it&#39;s a not null column so we should not load null values on the target we have to add some not null values here okay just you have to make some zero here okay manager id just i&#39;m passing the sequence number i want to make sequence number i have to generate sequence number from one to sequence number you have to generate it so we will do one by one first name we&#39;ll go to the same mapping double click on it first name create a new first name so o underscore first name create a logic for o underscore first underscore name so you should have the underscore not hyphen underscore here it is a string and string you can have the same 20 characters and you can make this as only as input this is output port and then you can make this is output you can see here upper first name how do i know the functions you can go for function name here just you can click on up or off you can make upper first name validate it it will convert the first name into uppercase okay you can go for if logic any other logic like this also you can go for what is the last name last name you have to make employee id underscore last name we have added so how can we do it i&#39;m just making like underscore last name it&#39;s nothing but output port oh underscore last name and then string here i&#39;m going with 25 characters and since i&#39;m making that employee id underscore right i will go for 30 characters is it possible yes it is possible so only 25 characters coming is input 35 characters we are taking but next transformation done it should be accomplished 30 characters okay last name what is the last name it should be employee id underscore so employee id here concatenation how to do the concatenation pipe symbol so you can use concat or pipe symbol so pipe symbol is very easy so just to use pipe symbol and i have to use space or underscore right just you can make here one space or underscore just like this and you can make like this and this is what you can go for first name underscore last name you can make like full name that is also possible okay last name that is what we can make just to validate it should be valid okay so now i&#39;m going with email column so same way if logic right so underscore email so what is the logic here if the salary of the employee is greater than 10 000 i have to make it like email gmail.com if it is less than that email at yahoo.com okay i can go here you can make this is 35 characters are 40 characters this is only input this is only output then expression will be enabled here you can write the logic okay how to write the logic if condition iaf as we have seen already iaf if salary greater than ten thousand then i have to make email at gmail.com along with this i have to add at gmail.com gmail.com if any other conditions like is there any regular expressions yes we have in our functions we have regular expressions we can use some special characters you can go for regular expression replace regular expression extract regular expression match we replace all this we have these three we can use it something like only alpha alpha number one be numeric number that you can use okay that will take it as exercise i will tell you email thiago.com this is ellspart at yahoo.com you can make the sales part and close the bracket so clearly you have to close the bracket you can see here so this is what if condition the condition is true what i have to do if the condition is false what i have to do okay validate it it should be valid and click on ok then phone number you can click on here you can add a new port below to that and oh underscore phone underscore number so it&#39;s not necessary that this is what same weed we need to have just i have to add some character right i can make 25 characters so i can make here and apply the logic what is the logic plus one iphone it&#39;s hardcoded value right so you can use single quotes plus one iphone single quotes this is character right so any character any hard coded value you can make it like single quotes here okay so now concatenation we have to remove the dot symbol from here we have to remove the dart symbol so for that we have to go for replace character replace character you can see here you may ask question how do i know all these functions you have to practice it one by one you have to practice it one by one that is the way you will be getting it replace character it will replace the character replay string the entire string will be replaced so based on the scenario we have to go with just i&#39;m taking the replace character replace character you can give four argument look at here four argument first flag is integer so what is the flag right replace character you have to give four argument remember four argument if the flag is true if you look at here if the flag is true this function is case sensitive okay that means case sensitive replace or case insensitive replays if i&#39;m asking you to replace a with x okay you may ask question so i&#39;m replacing a with x if small a comes should i replace or not you&#39;ll be asking questions right yes even both the case is coming you have to replace with x this is called a case insensitive replace okay case instead is to replace case sensitive replacements only capital a it will be replaced with x okay only capital here this is called a case sensitive refresh so this is you have to make case insensitive you have to give zero zero here case sensitive you have to make one one here okay clear on this that is what we will make first zero will make case insensitive then what is the column we have to change phone number what we have to replace the dart single quotes you make the dart single quotes comma two single quotes two single quote is nothing but null value either you can use this wave or you can use null that is also fine okay just i&#39;m making like this way this is dart and this is null okay you can see here just to validate it it should be valid and okay that is what we have made it here then job id job id from fourth character we have to take right i&#39;m just adding that to o underscore job underscore id then string here for the character from fourth character how will you take it&#39;s a substring right so you can go for sub string s u b s t r substring of jaw by d i have explained some substring very clearly that is what you have to make from fourth character from fourth character means four four means from fourth character right hand side it will take that&#39;s what so if you want to check all the functions please go through that on our video on single row functions okay in real time if you want to apply any logic if you want to clearly apply that you want to think so what could be the logic so definitely you have to practice all the functions whatever you have okay so please make sure that you are practicing it okay next one salary we have made it and the commission percentage so whenever we are getting null value we have to replace with zero right so i can go with o underscore commission so this is also decimal make sure that it should be a data type everything is correct then here we have to write if logic is null of commission percentage if is null this value is null then we have to add zero okay replace with zero otherwise we have to go with commission percentage itself okay if is no law of commission percentage this is condition if is null of commission percentage then zero otherwise commission percentage itself so this is what you have to write the logic if condition in expression validated it should be validated okay this is what logic i have applied a lot of logic i have applied you can go for manager id sequence of number should be generated how can i generate sequence of numbers you can go for o underscore sequence of numbers okay it&#39;s nothing but sequence of numbers to generate the sequence number we need to use variable port to use the variable port that&#39;s what i want to tell you that how to use the aerial port i&#39;m making this is a decimal port i&#39;m making like variable port so what is variable this value will be taken as first zero zero plus one it will be incremented one by one variable plus one next time this value will be taken so variable port one plus one two two plus one three like this it will convert then we so as i told variable port you cannot take it outside look at here the variable sequence is not visible right so you can take first name here yes you can take last name here yes you can take from here email port so you may ask question previously we have connected why this connection has lost right since we made it like that port has only input the connection has been blast that&#39;s what you can make the phone number here drag and drop then java id you can drag and drop then commission percentage can drag and drop then manager id look at here there is no manager id here right you can remove this and i want to take the sequence of numbers i i cannot take variable port outside that&#39;s what i told so you have to assign this variable to the output oh underscore sequence it is also decimal then we can make output port assign a variable to this okay click on ok then you can take it this this one so this manager id will have a sequence of numbers okay you may get some some little bit confusion on this very first time if you are watching no issues you have to practice by watching this practice okay so whenever you are going to practice it you will get a clear idea it will get memorized and you can go for some other logic also i will give you exercise okay now i will go to the right click refresh the mapping since we have already created the workflow then load it since we have given the truncate and load it will truncate the existing data and then it will load we can see all the new logic will be loaded it gets succeeded it will not filter out any record you can see here or not some record got loaded and we will go to the target table we will see one by one all the columns whether it has been loaded or not we&#39;ll check it so it is in upper case yes and then it is making like last name underscore but you might be saying underscore is not there but if you click on it you will be seeing that underscore okay then so whether the employees salary is greater than 10 000 you are seeing that at gmail.com are yahoo.com plus one iphone has removed and date column also we can apply some leap year only piranha job id and commission percentage and sequence of number by using expression how will you generate sequence so this is what we have to use it yeah i have given some expression logic so one or two exercise i have completed already just you follow it okay i have given some uh answer also in phone number we have two types of phone number one is this format another one is this format so whatever the format we have to split up this way plus one iphone first five characters iphone next to five characters plus one iphone first five characters iphone next five characters like this you have to load it okay and then you have to look at this one you have to complete it if you are not able to complete it i will explain in our next session okay tomorrow session i will explain and look at this one generate sequence number i have made it already and load the input data in reverse order so that you can apply you can complete it apply ten percentage salary only for first ten employees okay that how to do it uh you have to reduce the logic okay i have given the answer for the first question also here don&#39;t go for answer directly so you go for first you try it and then you can go for the answer so kindly practice it you will get a clear idea about expression all this so remember expression transformation will not filter out any record thank you please practice it you will get all the stuff very clearly [Music] hello friends welcome to nikkei academy in our today&#39;s session we are going to see another important transformation called china transformation so this giant transformation is very much used in our real time project so we will see this concept in our today&#39;s session so that is what is joiner transformation why do we need this china transformation and what are the different join types are available we will see one practical session to understand on this channel transformation if you haven&#39;t subscribed our channel please subscribe the channel and click on the bell icon so that you will be getting all the notifications thank you let us start our session [Music] so what is the use of source qualifier joiner source qualifier joiner used to join the homogeneous sources like joining from same type of database same tables data coming from the same database right but here if you consider i have one table okay i have one table from one database some other tables from another database i have one table from this is from oracle okay another one is from platform or sql server another source from sql server so i have two different sources so this is one source this is one source one is from oracle the another one is from sql server or db to r cloud file okay so this is oracle you assume that it&#39;s an employee&#39;s table okay any table you can go for so it has employee id right see whatever we have seen the same concept for source qualifier same thing we are going to see but the source qualifier we have taken both of them from oracle but here we are going to take so one is from oracle the other one is from flat file okay so this one we will take it like a flat file that file is nothing but our department&#39;s table so normally if it is a source qualifier what we can do so we can have only one source qualifier here so it&#39;s qualified java we can have only one source qualifier how many number of sources are there so with respect to of number of source we can have only one source collector but in joiner we need to have so this source qualified into hub and for this also we need to source qualifier we need to keep we should not delete it and we can we can take one joiner here one joiner here see whatever the first table if you drag and drop into the joiner the first table will become like it is a detailed table the first table is detailed normally we will call it as left table right table right left table right table but in informatica the first table will call it as detail table the second table we will call it as master table master okay master table then we have to do the join condition join condition what is the join condition we can write the join condition here you will be having the department id department underscore id we have department name on location id right department underscore name and location so like this you&#39;ll be having a column location id all the columns will be there so here you can take all the see all the columns you can take and you can take all the columns here okay so first table is detail table second table is master table join condition on the join type we have to take the join type also so what a join type i will tell you but join condition is nothing but see join type is nothing but whether it is a ignorant left out right outer will take i will take this joint type after this see join condition is what is the giant condition sorry implies that first you will take implied stability implies that department underscore id equal to departments dot department id so like this you&#39;ll be taking the columns right so you&#39;ll be taking the columns so this is called join condition this is called join condition normally is what we go for the joint condition okay then after that you can take the columns department name all these columns you can take if i so there is no filter condition in designer if i want to filter out the record either you have to before the joiner or after the journal it&#39;s based on the source which source we are going for so what is the first table it is a detail table what is the second table is a master table clear so we have four type of joints the joint type will go for four type of joints see we have four type of join type so one is the first table if you take the first table it will it is called detail table second table it is called the master team the first table what we are taking that table is detail table second table is monstrative always remember this first table is detail second table is master the first giant type is called normal join what is the giant first turbo it&#39;s a normal chart normal giant means inner join so inner join means matching record from the both the tables right it&#39;s matching the cut from both the tables you&#39;ll be having matching non-matching record so only matching record this is called normal join second join type is called second join type is called if it is left outer it is a master router master router what is mastered right it is a left outer join left outer means all the record from the left table all the record from the table matching the cut from the right table this is called master router okay third one is detail outer detail is nothing but right outer okay detail outer so this is what we will we will explain if they are asking about your joiner transformation you have to tell them either it is a normal join or master router or tt loader or full loader so what is this left out right after how it will perform so please go through my joiner video okay please go through that before this joiner please go through them so then you will get to know all the joining as well as your a source qualifier properties okay detail router is nothing but this is a detail order see how can i easily understand how can i easily know this which is the detail which is the master see here if it is normal join means it is a inner join you know already master router means it&#39;s a left order all the records from the left matching record from the right see master router means master will be out okay that means all the record from the detail detail outer means detail will be out all the records from the master clear fourth one is fourth one is called full outer full after you know right floater so what is the floater joint full loader is nothing but matching non-matching record from both the tables so where it will be used in real time so normally if i want to compare two tables so if i ask some questions you can you can take the data from two or more tables right so that is what these table joints are used okay remember this see whatever that table we have smalls table two tables we are taking right whichever the table is having small table like small number of record list number of record so that table you take it like a master table so why write why we have to take like master table so because informatica it will take catch memory catch a memory on this master table only okay whichever the table small table informatica will take this table into a informatica catch memory then it will create a caching so if it is small table then cache will be very less space to log back otherwise informatica will take lot of space performance wise you will go for the degradation see if they are asking how to improve the performance of china you can tell them i will go for the small table that master table small table as master table okay if you have n number of tables n number of heterogeneous table n sources you have to use n minus 1 china remember this one n minus one china why so if i have two sources how many giant needed two sources one china needed three sources two china needed remember this okay n sources n minus one china small table you have to go for master table clear on this if i have three tables like this how many china needed two joiner first employee is its oracle table departments is a flat file just i&#39;m taking these two like detail and master just connecting through the join condition okay through joint condition i&#39;m connecting after joining i&#39;m joining this to another china china two this is a detail so this is the master this is the giant condition we are making okay so this is like joiner one and joiner two so this is what you can go further if it is three sources say for an example this is oracle this is oracle you can use by using the source qualifier and then you can have only one joiner okay if it is possible to use source qualifier use it in the source qualifier itself say concept wise you have to be very clear how it will perform then joiner transformation is very easy only okay i will take m underscore joiner it&#39;s very simple only i will take one is from a table another one is from a flat file i will take employees as a flat file now okay this is a flat file departments are stable oracle okay there&#39;s a table then i can go for the joiner here so this is a joiner okay two into one right first table if you drag and drop this will become a detail detail second table if you drag and drop this is the master you can see here this is the master first one is detail second one is master okay then you can have target simply double click go to join condition just to make the join condition add a new condition see operator will have only equal symbol okay so department id equal to department id here not employee then properties you can choose the join type you have the four type of chart normal join master router detail or triple so i can choose the normal time so you&#39;ll be having null order that null order i&#39;ll take i will tell you what is this while taking the next transformation i will tell you what is this null order started input and null i will t i i will tell you this properties in the next transformation then you will get to know joiner will take data cache and index cache indexing is nothing but the join condition it will take the indexing okay so here i will go to the employees underscore department for that online we are going for the joiner right so i can take department name and i can take all this column from here control yes it&#39;s a valid one okay then go to the go to workflow or close create wf underscore joiner underscore transformation it&#39;s very simple only double click on it go to mapping source should be finding the source target should be pointing to target so you can choose the source and target a little bit down you can do the tranquility and load apply okay save you can start workflow if you look at here so 107 record from the source source qualifier 27 records from the departments it&#39;s a flat file and then 27 you are getting one or six records so one record since it is a normal join it is getting filter out filtered out okay you can go and check in the target it should get loaded if you are doing like a master router okay for an example you are doing master router master router is nothing but left or right master router is nothing but left order i&#39;m choosing master router then if you execute this you&#39;ll get all the 107 record why because from left hand side table you will get all the records okay you can choose here select star from t underscore employees underscore department right you can see here just one not soundtrack one not six records will be there right still the informatica is running so there is no null value on the right hand side but if it has been completed it will have null value on the right hand side see here 107 record has been loaded you can go and check here not seventh records you&#39;ll be having some null value on the right hand side all values from the right hand side so this is what you&#39;ll be having left out or right outer floater all the joints [Music] welcome in our previous session we have seen the joiner transformation now today&#39;s session we will learn the shorter transformation okay sorter transformation is an active transformation and also it is a connected one so if you look at this the sorter uh in sql we have the order by class right so in in sql we have order by so order by is used to do uh sorting algorithm either by ascending order or descending order the same way in informatica we are the transformation called sorter so it will do the sorting algorithm right it will make the data either ascending order or a descending order so where it will be used in real time so whenever we are facing some performance issue like we are using some aggregator we are using some rank transformation we are using some joiner or we are using some lookup so data is like not sorted order then it will take a lot of time it will have some performance issue so to avoid this performance issue we are going to use this water transformation so sorted transformation is used to sort the data okay so whatever the data is coming to start the data then you can ask me the question why it is inactive or if you are giving hundred records it will start right how would you be active that we will see the later point of our session if you look at here in sql we have the order by class if you look at this this particular these queries if you i&#39;m going to execute this query in the hr schema if you look at here just i have made it like select start from employees table order by salary so if i&#39;m selecting only select start from employees just i&#39;m selecting only select start from employees it will select whatever the way we have inserted the data but i want to order based on the first name or something so for an example instead of salary i&#39;m going with first name so first name i&#39;m making it and it will make the order by on the first name so look at here this is from alphabet order a and then it is alphabetically it is increasing right so we are getting some j here why it is like j is last right small j is lost because of this lower case okay the ascii value of lower case is higher than the upper case that is why this john is on last okay if you are making first name on descending order then this this john will become the first one right then it will go to the order by under descending order so this is what the order by for an example instead of order by first name i&#39;m going with order by salary so it will make the salary built on the order so if you if you are not mentioning any ascending order descending order it will by default take ascending order so this is what it will go with lower salary to higher salary it will increase in this way the highest salary is five nines right so this is what the uh the order by class will do my skewer you can make order by salary descending you can see here the order basically descending it will make the sending order of the data so for an example here if you look at the order by salary and descending we are making the employees based on the salary from highest to least if you look at here there are two employees are getting the same salary right two employees are getting same salary if you are ordering this way this employee may ask question that okay why i am the second person right among us i am the second person he will she will ask question and also if you look at here there are four members are getting the same survey so it is making order by on the salary only but within this particular salary how we are ordering so that is called the second column so second column or third column you can make it so order by salary descending then one more column we are giving like higher date so what will happen now so it will make order by on the salary only first if the same salary if i&#39;m getting more values here more repeated values then it will go for the second column so second column is look at here there are four employees are getting the same salary rate then it will make the order buy on the higher date ascending order we haven&#39;t mentioned anything ascending or descending by default it will take us something within this particular salary it&#39;s making the ascending order look at this so this is what the salary column you can do it okay for an example so this is more than one column you can make any number of columns you can make okay and also if you are making this uh order by any column if this particular column itself the first column itself i have the unique value then oracle will not check the second columns so this column it will not check why because here itself it is unique right so you will be getting higher to higher salary to least salary then it will not check the second one if if i&#39;m getting any duplicate value on the first order by class then it will go for the second one so this is the order by one okay you can ask one more question how the null values will be ordered so always the sql in any database null will be treated as highest value look at here select start from employees order by i am making commission percentage why because this order by commission percentage this column contains some null values so i&#39;m making like commission percentage it is making from here then it is making from point one to point four after that point four only it is making null values why because null treated as highest value by default if you are making order by descending then this is what the descending first null values then only the highest value so all the null values then highest value then it will go for the least values so null always will be treated as highest value so why i am telling all this right the same concept is applicable in informatica also you can make one column order by on one column or two columns or a number of columns okay and also how how the null value will be treated so that is also we can use it in sorter transformation so in sorter so if you look at here sort of transformation we have uh port some different port will be there like you have the input port it will receive the input so we know already the input port will receive the input and we have output port then keyport then direction we&#39;ll have if i&#39;m giving salary as input port and i&#39;m enabling the salary as key port here then make that direction either it&#39;s ascending or descending from top to bottom whichever the port is enabled the first one it will be ordered first okay so first it will order by salary descending if the salaries are same then it will go for the second column is higher date if you are making this way first column is higher rate then salary from top to bottom which is the first one now higher date so higher date will be ordered first if some some employees have joined on the same date then it will be order based on the salary so this is what it will do i can make this way so this way also i can make here okay so control is that i can make here also okay now i will go to the informatica and i will do the mapping go to mapping designer since i have already taken this employees table and t underscore employees table i&#39;m going to make this employees table as source p underscore employees tables target and i&#39;m going to create a mapping create a mapping m underscore sorter transformation just try making the mapping name is m underscore starter transformation then drag and drop implies source and we can drag and drop the p underscore employees on the target side so this is the target and this is what the source you can keep the sorter here you can keep the sorter here this is air2z it will do the sorting algorithm you can take the sorter here you can make the sorter in between source qualifier and target you can drag and drop all the columns to sort or now you can see the keyboard right keyboard here keyport is nothing but order by whichever the column we are giving that is a keyboard double click on the transformation you can go to this key port and direction you can see whichever the port for an example i&#39;m i&#39;m making that keyboard like salary you want to make it like ascending order descending order you can enable this keyboard and direction you can give as of now i am giving like salary order by on the salary descending order then go to this properties the properties you can see sorter catch size this catch memory is nothing but so sorting algorithm it will compare all the records and then it will do the sorting so that is why we are the working directory informatica will keep all the data it will fetch all the data to informatica cache memory in the cache memory it will make sorting algorithm then it will make ascending or descending based on the keyboard we have selected keyboard and the direction we have selected and it will create a sorting so definitely it will fetch all the record to the informatica level and then it will start it will take some time okay sometime it will take and you can see the catch size you can give auto don&#39;t need to give any other values here why because if the data volume is getting increased so this will take automatically the cat sizes automatically it will uh assign it case sensitive so what is this case sensitive case sensitive is nothing but if i&#39;m giving like this okay if i&#39;m giving like this some data so this is arun okay this is babu and babo this is how the case sensitive which one should be order first either this one or this b or this b okay so that is what the case sensitive if you are giving case sensitive yes then this will be taken as first this will be taken a second why because based on the ascii value the high the upper case will have the lower value right so that is why it will take the first one then it will take the second one then it will take so this is what you&#39;ll have the case sensitive you can give always it will be enabled we can give that working directories dollar pm temp directory so it will keep the data in this particular directory it will make the order then distinct i will tell you after some time the null treated as low so we have seen right null value will be treated as highest value we have seen here if i want to make the null treated as low value then you have to give here like null treated as low value so this is transformation scope for all the inputs so these are all the properties we have one property called distinct this property if you are enabling this property this sorter transformation will allow only the distinct record pass through it it will remove the duplicate from the sorter we have this distinct option in two transformation one is sorter the other one is source qualifier source qualifiers distinct will be used to select the distinct record from the source itself but anywhere in the pipeline if you want to filter out sorry anywhere in the pipeline if you want to select the distinct record you can go for the sort of transformation this distinct because of this property only the sorter transformation is call it as active transformation since it is removing the duplicate right if i have any duplicate it is removing the duplicate so that is the reason the sorter transformation is an active transformation so based on your scenario you can select the properties here now apply just i have given only this salary descending order click on ok then select all the port and then you can give it to target control yes it should be valid then go to the workflow manager we can create our flow and we can run it we will check whether it is making the order by on the salary descending order so wf underscore sorter transformation click on ok then click on this session task you can keep here session then select the starter transformation click on ok now you can link the task here from here double click on it go to mapping source should be pointed to source source connection the target should be pointing to target one so we know already the connection details we have seen in the installation itself and we are using this for more point of time then truncate target table option if you want you can enable it so incremental data load you should not enable it only if you want to have some staging load or truncate and load in the sense it will truncate the target table and it will load the data so for for our practical purpose we are just truncating and loading it you can double click on it you can enable this property like fail parent of this task fails and they can enable one more property in the config object stop on error equal to one it&#39;s nothing but if any one error comes it will stop the execution and also fail parent of this task file in the sense fail parent if this task fails if the session fail the workflow will get failed so normally these properties we will enable the real time okay now it&#39;s valid and right click start workflow so the monitor will be opened now you can double click you can go to the get run properties you can see here how many records it is reading from the source and writing into the target you can see here one not seven recorded from the source and uh 107 record return under the target you can see so this is what you can you can see the statistics and there is no rejected record and throughput is number of record per second if you go and check in the target table you can go to the course schema and check this t underscore employees table you can see here the data has been loaded based on the seller right based on the salary descending order it is loading but look at here here i&#39;m having like if i&#39;m having two employees are getting same salary it is not able to do the ordering right it&#39;s not able to do the ordering so within this 10 000 it is not in order or if you want to order based on the employees based on the higher date you have to go to the so these four employees will ask question right so herman may may ask question why i&#39;m also getting 10 000 why i&#39;m the fourth person right among us then i will tell like okay whoever joined to the company first or higher date will be the first i&#39;m making that order within this particular employee then i can go to the designer double click go to the higher date and enable the higher date first so as i told already from the top to bottom whichever the port is enabled first the informatica will make the order by on the first column so first column is higher date it will make order by higher date then it will make the salary look at here i will show you the result then i will make the second one okay now it will make the order by on the higher date only since i have changed after creating the session refresh mapping control yes start workflow it will again it will start and then run so since from the top to bottom now it is higher date is first one it will informatica will order based on the higher date if some employees have joined on the same date it will order based on the salary look at here it will order based on the salary you can go to the target now you can check it it will not based on the order by the salary it is based on the order by on the higher date 2001 then 2002 then three like this but if you look at here 7th june 2002 there are again four employees right seventh june there are four employees have joined on 7th of june but based on the salary it is making the order why because here we have given order by higher date first then salary descending right salary descending that is the reason order by higher date then it is making the salary deciding but our case our requirement is different right order by salary then higher date how can i make keep the cursor here you can see the error mark here you can move the selected port down you can make it down so from the top to bottom you can see the salary is first one the next one is higher date apply here then apply so something if you look at here the link will be twisted one no issues then you can refresh the mapping then you can run the workflow it will order correctly now it has been completed you can control enter you can select it you can see order by on the salary now employees are getting same salary if employees are getting same salary it is making order by on the higher date ascending order june 2002 then jan 2004 jan 2005 march 2006 so this is the functionality of sorter transformation okay so starter transformation i have explained everything that uh the one exercises i can give you so as i told we are the property called distinct you can take some sometimes the interview they will ask okay if i have a flat file it has some duplicate record how can i remove the duplicate record from a flat file okay you cannot use source qualifier distinct right you cannot use source qualifier distinct why because the source is flat file so after the flag file you can put source qualifier after the source qualifier you can sort you can place the sorter transformation in the sorter you can place the distinct option if you are enabling the distinct option here if you are enabling the distinct option here then all the port will be keyboard will be enabled to remove the full duplicate it will make the keyboard here so this is the functionality of this distinct if you want to remove the duplicate anywhere in the pipeline we can use starter transformation you can take one example you can take one flat file you can keep some duplicate record then you can place some starter transformation here then you can use distinct you can load only distinct record so full row duplicate only it will remove if any one of the column has been changed then it will not remove it will not consider that as a duplicate record so this is the functionality of sorter so sometimes interview they will ask why is arteris inactive because of this property saturation active so kindly practice this you will get all the functionality of sorter [Music] hello friends welcome to nick it academy in this session we will learn about aggregator transformation so we have seen sorter transformation previously and now we will see the aggregator transformation so what is aggregator transformation this aggregator transformation is an active transformation and connected one so we have seen in sql we have seen some aggregation function right so minimum of maximum of sum of average of count of all this are aggregate functions irrespective of number of record this aggregate function will give you only one record in the output if you are doing some minimum of some salary minimum of first name something like that if you are giving like minimum of some transaction date or date column it will give you the minimum rate so maximum it will give you only one record minimum will take you on record sum of average of count off it will give you only one record we will see one example then you will get some idea about this aggregation then we will go to the informatica aggregator transformation if you look at this table the students table student id student name year of study and mathematics physics chemistry biology marks are there okay our aim is to take the total mark of each and every student the total mark of each and every student and average mark of each and every student so this is something like a row level right this is something like row level so i used to ask this question in many of the interviews they will ask they will answer it like i will go for some off so make a note of it my question is each and every every student&#39;s total marks i need like total marks like this row level if i&#39;m asking the if the interviewer asking questions or you are getting any scenario like this row level you should not go for aggregate function candidate will answer like i will go for sum of all the marks no you have to do like this the select statement we have did it right so we have to do like this you have to do something like this you select mathematics plus physics plus chemistry plus biology as total marks this is our last name then you have to add all the marks divided by 4 is the average mark interviewer asking questions like row level then you have to go with expression you have to add it plus you have to use it if you are getting any scenario like this aggregate function aggregate function will do the scenario like this column level so remember this column level if i&#39;m asking what is the maximum mark in mathematics then you have to go for the aggregation minimum mark sum mark average count everything you have to go with aggregate function so what is the maximum mark in mathematics then you have to use select maximum that max of max from table name so if you&#39;re making that max of math then you will be getting 98 right 98 you will be getting minimum you will be getting 72 right so this is what you&#39;ll be getting what is the sum sum will be getting okay so this is what the sum 779 the nine members have nine students have appeared for the examination then average mark in mathematics is 86 so average or sum or anything this aggregate function will give you only one result remember this so this is average mark in mathematics average mark in physics only one column only you can make in aggregate function you cannot make average of mass come up with six that you cannot make it you can make it like average of max comma average of physics that you can make it but you cannot make two columns in the two are more than one columns in the average function okay if you look at here there are some color right color difference like green color and yellow color green colors the student list from first year then the second one i have taken like year of study second year so why i have grouped it it&#39;s a group by call so we can group by year of study then we can make okay what is the maximum mark in the second year what is the maximum work so this is the same concept would be applicable in informatica also group by on the column and then you can take aggregate value so normally you will get how many group it is creating so those many records you&#39;ll be getting it okay in informatica level if you look at here i will go to the aggregator so this is summation is the aggregator i will create one aggregator transformation i have explained whatever so this is what group by for example sum of salary group by department id i&#39;m making so what is this meaning so sum of employees salary in each department so that is the meaning okay so that you can go with group by column okay now we will practically see then you can get clear idea so same employees table i mean i&#39;m going to take it so i can create a mapping now go to create mapping m underscore aggregator transformation aggregate underscore transformation just i can create it drag and drop the employees table here drag and drop the t underscore employees on the target side you can make here the aggregator is nothing but so the summation this is summation right so this is an aggregator transformation you can take this aggregator transformation here you can drag and drop all the columns from the source qualified to aggregator in aggregator you can double click you can see you&#39;ll be having input port variable port output port variable port and you can see the expression and group byte so one column extra is there like a group by port is available so this is a salary it&#39;s coming like input and output i&#39;m creating one more port is called sum salary sum of salary o underscore o is nothing but output o underscore sum underscore salary so that is what the sum of salary i&#39;m taking it if salary port is decimal you can make one more the output port also decimal you can make this is an output port you can make this is an input port the sum of salary you can make some aggregate functions so what is the difference between expression and aggregate transformation in expression you will not have this folder in aggregation aggregate transformation you&#39;ll be having one folder extra called aggregate if you click on this you can do these aggregations in aggregator transformation okay you can do average count first last maximum minimum median and then everything can be do now i&#39;m going to do some of the salary so i&#39;m going to do sum of salary just i&#39;m making it validate it&#39;ll be valid then click on ok then i&#39;m past i&#39;m passing salary here i&#39;m taking that sum of salary out from here and click on apply okay i&#39;m not selecting any group by port here select from employee id to job id from here to here and you know already this is an output port i&#39;m just taking output the sum salary to the output okay you can ask question this is sum of salaries like a aggregate function right how it can be passed to all other columns so if you look at here in employees table if i&#39;m making like sum of salary what is the sum of salary sum of salary for all the employees it will make the sum salary right so it will make sum of salary is something like this is the sum of salary it is making okay so can i take the sum of salary for something like first name here can i take first name here in the adjacent to it no i cannot take the first name or other columns i cannot take why because so you cannot take non-aggregate column with aggregate column this is on the we know already that we cannot take it if i want to take first name of the employee who&#39;s getting higher salary or something then you have to go with some sub queries but sum of salaries it is not meant for one particular employee right so this is meaningless so this you cannot select it so that is why we are getting error then how would it be so if i am going to give all other columns how it will be in informatica it will be taken as so in informatica level if you look at here select star from employees table right so what is the aggregation value seven lakh seventy four thousand 000 415 this is the value so informatica will take all the salaries sum of salaries and since i have given this column these columns to next level so informatica will take last record okay informatica will take last record instead of this salary 8300 it will make some salary here okay so in most of the scenario you have to take only this salary output this summation value to out next level since we are taking all other columns the last record it will make the the output it will make the summation value look at now i&#39;m going to create a workflow i hope you are clear now we will see how it will be created aggregator underscore transformation you can find source and target go to truncate and load just time truncating for this year for the testing purpose just to save it run it even if i have millions of billions of record in the source it will give you only one record in the target i haven&#39;t selected any group by port so irrespective of number of record it will give you only one record output that one record is last record it will make the summation value look at here it will take the instead of william gets salary 8300 it will make the salary yes only one record it is giving you the result right so one not seven record you are getting only one record even million billion record it will give you only one record same last record instead of his salary it will take the last salary you can ask question okay this is wrong right why it is his salary is making some salary the aim is to take some of sunday only since we are taking all other columns okay to the next level to the target level it is making the the last record it will make the the summation value okay in interview they will ask this question so how many records you will get how this summation value will be loaded into the target level so it will load the last record it will load the summation value okay i&#39;m going to give some group by group by on the department id so i want to segregate the 7 lakhs 44 000 in each group in each department id so as a just i have given group by on the department id refresh on the session refresh mapping then run it if i have some ten departments each department it will give you the sum of salary you can see here you can select here right the sum of salary the seven lakhs forty four thousand this is source level just i can make it group by department id you can make group by on the department id you can select the department id here you can make order by on the department id order by one so one is nothing but first column so each department it is giving you the sum of salary you can see here it is giving the if you add all the salary it will become the seven lakhs seventy four thousand four one five so this is what distribute the salary for each department so 12 records you will be getting it in target also you&#39;ll be getting the same you can look at yes 12 records will be populated so each department you have to consider the department id this column and this column how it will be populated each and every department last record it will load the data okay if you add all this right it will become some that seven lakhs forty four thousand four one y so this is the the functionality of aggregator transformation you have to consider the aggregation group by if you want to filter out okay for an example i want to load only the departments who is getting more than fifty 000 okay who is getting more than 50 000 normally these four departments are getting more than 50 000 right so in oracle we will be using the having class so group by department id we are using having right so in order to filter out record will be grouped by our result and we will be using having plus having you can use this sum of salary greater than 50 000 so if i want to use like this sum of salary greater than 50 000 then in informatica you can use filter transformation i can use something like filter transformation for the aggregator we have to use the filter transformation you can use the filter transformation here and they can keep all the all the columns whatever the column you want you can keep all the columns double click on it go to filter in the filter you can use this column go to port sum of salary greater than 50 000. so this you can make it validate so only those the department those who are getting sum of salary greater than 50 000 the filter will allow you can save it you can refresh the session refresh mapping you can execute the start workflow now you&#39;ll be getting only four records out of it you&#39;ll be getting only four records you can see here the aggregator is running now even though it is reading millions of record in this scenario we&#39;ll be getting four records so four records right only four departments are getting more than 50 000 of the sum of salary what are the departments you can see department id 50 80 90 100. you want to know what so what is the department name then you have to go for the join so i can go for source qualifier joiner or join joint transformation that we can see so this is the functionality of aggregator transformation okay so now in aggregator transformation we have some more concept in properties you can see option called sorted input so aggregator transformation normally will do two operations one is sorting first it will make the sorting algorithm it will sort the data then it will do the aggregation then only it will do the aggregation so normally this is what the two operation it will do if i want to improve the performance of the aggregator if i want to reduce the overhead of this aggregator transformation then you will have option called sorted input okay so here i am enabling the sorted input is nothing but if you are if if you checked showing like you&#39;ve checked the input to the transformation is assumed to be already sorted by group by port so whatever the group by port we have selected like group by port we have selected department id right so informatica will assume that okay if you are enabling this property informatica will assume that okay i am giving some sorter data based on the department id based on the department id so just i&#39;m enabling it but i&#39;m not giving any sorter i have informed aggregator that okay i will give you sorted input but i am not giving started input what will happen it will fail the flow look at now it will fail the flow you can see here failed parent of this task space i am just executing it so normally sorter will normally the aggregator will do two operations sorting and aggregation but for this scenario i have informed aggregator that okay i will give you the sorted input don&#39;t do this operations then aggregator will try to do the aggregation for the sorted data but i&#39;m not giving any sort of data here right i&#39;m just directly giving this because of this it will fail the flow look at now it has been failed you will be getting this expecting keys to be descending order that it will fail that okay what i have to do you have to enable before aggregator you have to make one sort of transformation i can select it remove this i can enable the sorted input you can go for the sorter we know already that sort of transformation we have seen in our previous session so drag and drop then you can take the salary up to here from here then you can take commission percentage to commission percentage so double click on it which port we have to select sorted we have to select the department ids descending order or sending order any anything okay i&#39;m taking descending order the keys to be descending order apply okay now so if you are enabling that sorted input in aggregator we have to in we have to place one sorter before aggregator now it will succeed i&#39;m going to refresh the mapping i&#39;m going to start the workflow so this is the property of sorted input in aggregator transformation in many of the interview they will ask this question so what is the sorted input for sorted input option in aggregator okay if you are enabling this this property is used to do the it is used to improve the performance of sort aggregator if you are enabling this property we need to place one sorter transformation before aggregator so whatever the port we are making like group by look at here now it has been succeeded you are getting four records whatever the port we are using group by here so that port we have to choose here one more concept here if you are not enabling this group by if you are not enabling this group by simply or making some salary right sum of salary then if you are placing the sorter or if you are not placing the sorter it is not an issue so here just i have enabled the sorted input but i am not placing the sorter why because i am not using any group by port here if you are using the group by port it is necessary to have this order but if you are not using the group by port no need to have so salary also you can drag and drop then commission percentage you can drag and drop control yes i have enabled sorted input but i since i haven&#39;t used group by port here no need to give sorter so this is another concept you can refresh the mapping you can control s you can start it you&#39;ll be getting only one record out of it why because i i haven&#39;t selected a group by right you&#39;ll be getting only one record even though it will you&#39;ll be getting it even though i&#39;m using some filter i&#39;m using some filter is somehow salary greater than 50 000 only right so i&#39;ll be giving getting seven lakhs forty four thousand because of this it will will get one to one record you can check that record so that will be loaded here in the target so this is the functionality of aggregator transformation okay remember this what is the property of aggregator transformation i&#39;ve interviewer if you ask that okay how will you improve the performance of aggregator you can place before aggregator you can place one starter transformation then double click you can go to the properties you can see here a caching directory you know right aggregator will do the sorting and then aggregation and that is what will use the cache directory data cache the aggregator will do data cache will have that data and index cache it will make the sorting algorithm index cache so that is a two cache it will it will do it so that is why the agree whenever you are using aggregated transformation in your real time project so definitely it will you will be facing some performance issue you can place sort of transformation before aggregator whatever the group by you are using so sort it accordingly so this is what the aggregator transformation i hope you are clear aggregator transformation kindly practice this aggregator transformation if you have any questions you can post then i will answer i will give some exercises to you on the aggregator transformation so this is an ex exercises for you load the department&#39;s name where more than 30 employees are working i need like this you can take our our sources flat file so one of the employees as flat file and department as table then you can do the china since it is a heterogeneous right you can do the china transformation an aggregator you can do the aggregation you can load the data to the target like this department id department name and count of employees these three columns only i needed you can create one flat file or table with this three columns and load the data you can use joiner since both are like heterogeneous sources then you can use the joint transformation aggregator then filter to filter out more than 30 employees right so these three transformation you can use it and do the exercises here you can create one target table like department id department name and count of employees you have to load only this this result you should be getting it so complete this exercise definitely you&#39;ll be getting all the stuff in a joiner as well as aggregator and filter transformation practice this if you have any questions you can post your questions then i will answer it hello friends welcome to nik iit academy and we are learning informatica power center in our today&#39;s session we are going to learn router transformation i have explained the various properties of router transformation and the created example mapping explained real time usage so kindly watch till end of the session and you will get clear idea about router happy learning so router transformation is again it&#39;s like a filter transformationally it&#39;s an active transformation unconnected you look at filter transformation filter transformation will filter based on the one condition one condition or you can put the multiple condition with under r now the filter transformation like this consider this is the filter transformation if you are sending some record to this filter transformation then this set transformation will filter out the record for an example this is a customer table you are sending all the countries customer you want to filter out india customer alone you have to send it to the next level then what is the condition you can make country equal to india you can make so this is the way we will filter out if any other countries record is coming out coming up to this filter it will get filtered out it will not pass other countries out of this filter it will block the unsatisfied record on satisfied record in this particular filter so that is what the filter transformation will do you cannot take out of this filter transformation the unsatisfied records it will block you can get only the satisfied record you can put condition also like you can put under condition or condition condition you can put whatever the filter condition is satisfying that condition that conditions record only it will be passed other record it will get filtered out okay i want to take that filtered record also i want to put lot of groups i have to take for an example i have to take india separately you are separately uk separately in that case you can go for router transformation router transformation is nothing but you can pass single source pipeline okay single pipeline to multiple pipeline so that is a router transformation you will be having one default group so default group by default it will be added whenever you are creating any group in router transformation default group will be created so router is nothing but multiple filter conditions multiple filter in single transformation that is what the router transformation is very simple transformation whenever you are having single pipeline for an example you are having the single pipeline source you are getting the data like a card details debit card or credit card or anything you have to separately you have to segregate this into multiple groups so this is one group and this is another group you want to filter out you can filter out like this you can create multiple group so this is one group you can get credit card this is a debit card details and an example all the four columns will be there so here you can get all other remaining call cards default is nothing but whatever is not matching these two group it will be taken out of this default so whenever we are creating one group the other the default group will be automatically created so this group will be automatically created you are not going to create this default group you automatically create you can put this is like a credit card this is like a debit card okay this is another whatever it is not there like a card type other than this credit card and debit card all the record will be passed to default group that is also you can take it out but in in filter you cannot take the unsatisfied record so that is the advantage of router transformation you can trout the transformation is nothing but you can give single pipeline input multiple pipeline output so that is the router transformation okay we will see one example here i&#39;m going to take some employees from different countries employees from different countries i can take this is the employee&#39;s country&#39;s information employees are there in different countries i can take in this way there are different employees like employee first name last name all the countries are there you have employees from india employees from usa employees from uk again uk then usa then india like this i have lot of employees are there like uh we have different countries employees are there in this table uk usa then uk like this i have yeah like this i have employees details they are from different different countries you have the country column another column is country column you assume that i have three target tables one table has i have to take all the employees who are in india so that i have to take it separately here in this particular target table india then i have to take another target table so in this table i have to take the employees blocks to uk so that i have to take it like a separate target here and then i have to segregate the other roof file us employees that you can take it separately so i have to segregate can i use here filter transformation no it is not advisable why because if you are using filter you have to use three filters i just use three filters one filter i have to use country equal to india another filter you have to use country equal to uk another filter you have to use country equal to usa so that is what you can take if any other countries are there it will be blocked here but you want to take any other countries to next level that you cannot do by using this filter transformation instead of making three transformations parallely you can put all the transformation in one single transformation so that is called router transformation so this is nothing but router transformation so router transformation will have single input single input we are passing input into the single input and this is one group one output group this is another group so this is another group okay taking that different group i&#39;m passing i assume that this is the input group you have different current so only i&#39;m mentioning four columns so this group we will call it as input group this will be where you can create multiple groups you can create for an example here you can write the condition country equal to india you can write the condition here here you can write country equal to uk that we can write here you can write country equal to you say whatever the filter condition you want to segregate the data you can put this way and you&#39;ll be having one more port called default group right so this is the default group by default you will create we are not going to create this group automatically this will be created whenever you are creating one single group it will be created so you can take this group into so whatever the level you want to have you can take it next level so this is one group so i&#39;m taking that into country equal to india so this is i&#39;m taking and these groups i&#39;m taking to country equal to uk this group i&#39;m taking country equal to usa even if you want some different countries is coming like china or australia some other countries is coming up you can take this default group also to one more target so here you can have one more target table you can keep all this default group also on target so this is called router transformation router transformation it will segregate the data from one single group into multiple groups that is called router transformation we can put something like in in real time we can segregate the null values into one group and not null values into one group duplicate value into one group not duplicate value into one group odd numbers even numbers something like this you can put what i have explained right debit card credit card and the customers active status inactive status that if you want to segregate you want to apply some different different logic you can apply here so this is what we will use in the router transformation okay you can ask one question here if i have the same if i have the same country called india again also a country called india country also india what will happen you will be having the same record in this group also in this group also in this group also that you&#39;ll be getting three times here so single if i have 100 records here you&#39;ll be getting 300 records here why because here 100 here also hundred here also 100 so that will be passed so that might be interview questions they might be asking an interview so how will you solve this now we will see practically i have the table here in hr schema i have table select star from employees underscore country i have the table in place underscore countries you can see employee first name last name all these columns are there you can go to here you can see the department id i have one more column called country separate column i kept i have created one more column here i have mentioned like a country you have the india usa uk all these countries and the japan and everything i have mentioned but i&#39;m going to take only three groups other than that i&#39;m going to make it like a default group so i have created four targets in that course schema i have created four target you can see here i have made it like india t underscore employees for india then i have kept one more uh usa and then i have kept a uk right uk and i have put one more default okay as of now we don&#39;t have any record in these three tables okay so first i&#39;m going to the source analyzer i&#39;m going to pull the source table employees underscore countries table go to source here import from database since source is a database table click on hr you can&#39;t connect so it will be connected here you can take this host table go to source tables the table name is countries employees underscore countries i can take the source table so you can see the one more column i have just added country okay i can go to the target the target i have the four tables you can pull all the four tables at a time so i can go to the import from database you can connect to the target schema you can select all the tables the t underscore employees underscore default then india then uk then usa all the four tables just i have selected click on ok so it will be all the four tables will be imported now it&#39;s importing you can see all the four tables has been imported okay now i can go to the mapping designer create a new mapping m underscore router transformation router underscore transformation you can create router transformation now you can see here we have the source table you can drag and drop the table called employees underscore countries and drag and drop and you&#39;ll be having the source qualifier also and target table you can have this t underscore employees underscore all the four tables you can select all the four tables and drag and drop here i just drag turned up all the four tables that is what i have uh you can drag a drop at single time also then you can put all the the default table i&#39;m bringing below that first table i&#39;m going to take india so that is a table the second table uk then usa then i&#39;m making that default so first i&#39;m going to make the router so this is the router single to mini if you look at here this is a single group to mini so this router you can place here in router you can drag and drop the columns remember if you drag and drop it will have only input you cannot take output from here this is why because the input group you cannot take out you can see there is no arrow mark here single input group that&#39;s all okay you have to create a router group output group go to the groups here it&#39;s very simple you click on add new group just one group if you are clicking on one single group it is creating the default group by default okay first step you can make whatever the group you want i need three groups just i have created same like filter you can see the condition is true you can put the group name group name in i can put country equal to india so india just you can make whatever the group you want and then uk you want usa right just this is not condition this is just a group name just it&#39;s a group name you can go to the first this is the condition here you can put the condition you can make country equal to india so it&#39;s a very it&#39;s a case sensitive you have to make correctly if you are thinking that it may you may get country in this way also india also you have to put upper of country equal to india so that if you are making like this then there is no issues in this okay so whenever the country is coming like lower or upper so that will be make it like it is converting country into upper then equating with uppercase even though the country is coming like lowercase it will be equated then validate should be succeeded then you can select this one this group click on ok go to second one you can make country equal to this group we can have the uk condition you can make you can ask question can i put one more condition yes you can put with and r so that condition you can make separate condition you cannot make separate group only you can create you can go with country equal to usa here other than these three countries will go to the default group so that it will automatically create if you look at here this is the input group and you can drag and drag below and then you can see you can drag and drop the columns so this will not have any input group you can see here so this is what you cannot see input to this particular group you can see the output only you can see the output port has been enabled you can drag and drop all this column to the first target table so that is your t underscore employees for india then you can minimize it so why because there is no space much right you can keep this target this target here uk you can little bit down uh go down and then you can see here you can drag and drop the uk also then you can have the usa you say here so that target table you can ask questions should i have always target table no need you can have different pipeline also here you can put send it to some different logic and then do some logic again you want to combine you have to do union operation that we will make after some time that we will learn after some time how to combine the segregated pipeline into single pipeline that will be our union operation so here we are the default group right even the default group also you can take it out i am not sure where it is right you can have this toggle view you can see this toggle over here right you can keep like this if i have more number of transformations are available you can toggle like this the window then i can see this this is one target table available you can drag and drop here and they can put the default group also here the default columns also here ctrl s it should be valid now even you can make arrange on right click arrange all it will be arranged on you can see that will be arranged on you can have all these four target tables and even it can put arrange all iconic so it will be showing an iconic way also you can see this way also so this is the single input group and multiple output group which is converting single group into multiple single source pipeline into multiple groups so this is the router transformation now look at here we will create one workflow then we will we will run and then we will see the result how it is segregating the data here in this folder i am going to create the workflow go to workflows create wf underscore router underscore transformation so that i&#39;m just creating router transformation i have this session here you can select the associated mapping click on ok connect this by using link task double click on the session go to mapping check whether source is pointing to source and target is pointing to target but target is pointing to source itself you can change it to target connection i want to make all this for all this target table i want to make this connection so right click so this is the connection value right apply connection value to all instance if you are making that all instance will be passed to the same connection okay you want to do truncate and load yes the same way you can put truncate target table option what is that before loading the data it will this option will truncate the target table then load so same way you can put apply attribute to all instance instance in the sense this is a target instance right you can have the same and you can go to the config object you can make the stop on errors equal to one if any one error occurs it will stop the execution you can session execution and workflow it will fail you have one more option called fail parent if this task fails if this task fails it will fail the parent so that also you can enable it this is also normally we will enable see that these are all the some basic uh checks we will do control yes it should be valid now workflow should be valid right click here you can go to the start workflow if you are clicking on start workflow the monitor will be open you can see here one input group and multiple target if i have 100 records it will split up 100 records into four different groups now look at here i will show you the job is workflow is running now and go to get run properties you can have this it&#39;s it got completed you can go to this one and then you can see here i have one source 107 record and that default is going with 19 other than this india usa uk we have 19 india it has been splitted into 36 uk 36 again usa we have 16 records and if you add all this 19 plus 36 plus 36 plus 16 it will become or not 7. this is what the applied rows and the affected rows you can have there is no rejected rows there is no error also in this particular session ok so now if you go and check the target table this is the target table right india so that&#39;s india if you click on you can see all the india related data will be there even if you have if you see here since i have made it like upper of lower up right so that is why i got this india to the group if i&#39;m going to remove that you will not get it i will show you so that i&#39;m making it so usa yes india underscore usa in in oracle dot datas are case sensitive the same way informatica also case sensitive the group conditions usa and uk you can see here this is what then default group default okay so i&#39;m going to show you this default group you can see another the other than india usa uk all other remaining countries have been loaded into the default group so this is what the router okay now if you look at here so this is a group condition right i&#39;m going to change the group condition instead of upper-half country called india just i&#39;m making country equal to india what will happen here if i go if i if i&#39;m making that country equal to india then it will pass only the the upper case the lower case record right it will go to the default group you can see after creating the session i have i have changed the mapping level so you have to refresh it refresh mapping it has been refreshed that is a symbol here save it and execute again i am going to start workflow since we have given truncate and load it will truncate the existing data and load it again we are seeing here previously we were getting 36 record to india two is coming from india like why because other countries are like lower case money you can see other than this everything it&#39;s like lower case whatever the lower case we have so that will become now in in this default group that will go to the default group right you can see the india india here this group will have only two day two records why because we have written in that way so that is why to say first side we are writing in this way okay upper of country that we are writing since we have written the upper case right so only two record has been loaded into the target table okay in real time this is what we will do single pipeline to multiple pipeline so this is what i have given x is one the router transformation you have to split up this employee into odd number your number this is very simple exercises and then you have to make the higher date is one of the column implies you have to take leap year who has joined on leap year who has joined our non-dpi so that is an exercise another exercise you have you have to take the first off record and second half record to to target table and you have you can split up this into like a duplicate record non-duplicate record also so that you can send it so this is what you can do the exercise if you look at here so this is the properties of the router transformation i hope you are clear about router transformation kindly practice the session and we will meet in the next session with another topic thank you for watching [Music] welcome in this session we will see uh the next transformation is rank transformation so rank transformation is an active transformation and connected one so we know already what is a rank rank it will do the ranking if i want to take top or bottom rank you can go with the rank transformation we have seen already that our aggregator transformation aggregator will give you the maximum minimum sum if i want to know i who is getting this maximum set of vr who is getting this minimum salary or i want to take some latest transaction data for each account something like that i want to take highest salaried employee in each department highest salaried employee in each country okay if i want to do like this ranking then i can go with rank transformation so before that we will see one example then we will go with rank if you consider this is the salary of input data all the employees one employee is getting fifty thousand another employee is getting forty thousand twenty thousand so if i have salary like this for employees selling like this if i want to do ranking based on the employee salary highest to least i will order right i will do the order by on this column normally i will write in sql we will write in this way like rank off over order by order by which column you want to do ranking then you will do the ranking either descending descending means highest to least so this is what rank column so r and k normally this is what we will do in our real time in our sql so something like if i want to do ranking i will use this way rank of over order by which column i want to do the ranking then i will make the order by class the same way here also i will do the ranking i will make order by on the salary from highest to least i can take the salary based on like this highest to least like 70 000 then next to highest will be 60 000 next highest will be 50 000 like this i&#39;m making this so we know already what is this rank rank so i can take rank here if you look at here this is the data if i want to go for rank so this is the highest salaried employee i can make it like a rank spawn next highest will be 2 next highest will be 3 rank next will be 4 5 and 5 why because two employees are getting same salary then i will make five and five next employee will be ranked as seven only why because i have assigned two same rank right next will be seven the next will be eight here and this is also eighth rank employee the next will be ten here eight nine this is ten ten eleven then twelve twelve and twelve so three employees are getting same salary i will make 12 12 12 and next one will be 15 only 15 16 and 17. so how many employees are there so those money rank i will get it so this is called rank this is called rank dense rank what is this dense rank normally dense rank consecutive rank will be assigned for an example next round will be six only six next will be seven and seven the next will be eight nine then i will have ten ten 10 10 11 12 13. so consecutive rank will be assigned this is called dense rank row number is nothing but so serial number will be assigned for row number this is the row number right so row number if i want to take something like unique unique record based on the rank i can go for the row number so this is called rank dense rank and row number by using informatica rank transformation we can do only rank we cannot do directly dense rank by using rank transmission if i want to do dense rank then we have to go with some other logic we can use expression transformation to create a dense rank this one i will explain after some time or you can take it as an exercise you can complete it and also you can do rank on only one column but here it is very simple so no need to write any queries uh it&#39;s very simple it&#39;s for an example i&#39;m getting the salary column it&#39;s coming like input and output if i want to do ranking on the salary just to make it like a rank just you can enable this rank port just to enable it if i want to do group by on the department id the department idea column you can group by it just you can make group height and also you can see the properties top or bottom if i want to do descending order so descending order highest to least then i have to make it like top here so top means highest to least number of rank you have to define how many number so i want to make top five employees in each department then this is the way i can make it as i told earlier we cannot do dense rank by using rank transformation directly instead of rank transformation here we have to use expression transformation to do the dense rank so that i will explain after some time okay now i will take the employees table from the hr schema and load it based on the salary to the target core schema i will take employees table in the source hr schema and i will take t underscore employees table in the target based on the salary i will load it the rank okay we&#39;ll go to the source here source we have already implies table target we have already underscore employees table i can go to the mapping mapping create a mapping m underscore rank underscore transformation so i can have a rank transformation drag and drop implies here drag and drop tms for implies here so this is the rank transformation you can see something like rank right you can keep the rank transformation here so by default you&#39;ll be having the rank index so what is this rank index it&#39;s output port by default will be available rank index is nothing but so this is called rank index okay this number if you are taking this column to the next level you&#39;ll be getting this number okay so if you want you can take it i can take all the columns to the rank double click on the rank transformation it&#39;s very simple i want to do ranking based on the salary i can make rank on only one column if you are enabling any other column based on higher date i want to rank then only one column will be ranked so i can just type i&#39;m doing the rank based on the salary as of now i&#39;m not doing any group by go to the properties you can see it will use cache memory why because it is doing the rank right top or bottom top means highest to the list number of rank i want to make rank for 100 records i want to do the ranking case sensitive string comparison we have already seen that what is case sensitive right so if i&#39;m doing some rank on some string column then case sensitive it will compare okay tracing level data cache is nothing but so this is the data cache so informatica will have all these data into a data cache and this is the index cache so first of all it will arrange it then it will assign the index so that is why it will take data cache and index cache so it will take some time to do the ranking apply here okay click on ok so go to this so i&#39;m taking all the employee to department id here if i want to know the rank index then i can go with this manager id just i&#39;m removing this manager id i&#39;m taking this rank index to the manager id you can ask question okay can i have this rank index to the manager id the column name is different why you are keeping this manager id column with rank index i just want to show you that how the rank index will be populated so that is why i have taken this rank index to manager id is a numeric column it is not necessary to have this column to always output you can leave it blank also no issues it is a valid now you can go to the workflow create an overflow so wf underscore rank underscore transformation you can create one transformation rank transformation click on ok you can click on the session icon keep the session here you can take rank transformation just drag and drop connected double click go to mapping source should be pointing to source target should be pointing to target so just we have to provide all the connections you can truncate target table and you can config object you can go to this stop on errors you can make it like if one error comes you can stop this workflow execution you can fail parent with this task fails if this task fails fail the parent you can enable all these properties you can start workflow so it will be started now you can see the rank transformation is running it will load only 100 records why because we have given top hundred number of rank is hundred only hundred records has been loaded there is no rejected record so throughput is 100 record per second so there is no record no there is no error can go to the target you can see here select star from t underscore employees table you can see all the target data has been loaded now right so based on the salary it is loading you can see from top to bottom it is loading i can make it&#39;s a rank you can see here so one two three four five six seven and seven y seven and seven so here we are getting the same salary seven 7 the next rank will be 9 only rank will get skipped so this is called ranking i want to take only top 10 employees then i can go with top 10 only so i can make only top 10 it will load only top 10 employees for all the records so even if you have millions of record it will not have only top 10 records you can you can start workflow you can see here now it will have only top 10 records so it has been loaded you can see 107 recorded from the source and 10 records only loaded into the target table why because we are given top 10 only you can see the target table now only top 10 records loaded based on the salary right top 10 you can see here the rank index here top 10 so this is the rank i want to make each department i want to take top one so highest salaried employee in each department just to take it like department id group by then go to top one here so each department it will take top one employee it&#39;s very simple so normally in sql you will write some sub queries or we will write some partition by department id order by right so but in informatica it&#39;s very simple you just enable the port so if i have 12 departments you&#39;ll be getting 12 records so this is what you have 12 records so this is the rank transformation you can see here so all the rank index will be now it will be one why because on the top one rank we are loading right so all the top one rank we are loading so each and every department we are loading the top salaried employees okay so this is the rank transformation so how it can be used in real time okay real time project how to be used uh so for an example if i have something data like this i have a transaction date some transaction table i have the transaction date transaction id of the each transaction then account number for the transaction name of the account holder credit or debit so amount balance amount if i want to take top latest file transaction for each account then how will you do latest file transaction for each account like mini statement i have to take our latest one transaction for each account since i am saying each account you have to take group by on the account number and rank on transaction id why because transaction id is unique id will be created so all the numbers are unique right rank on transaction id highest transaction id top one group by account number so i&#39;ll be getting that okay latest transaction for each account so something like this you&#39;ll be getting a queries like this for any interviews are in in real time scenario also so this is what rank will be used so as i exercise you can take dense rank so you can do the dense rank by using the expression transformation you have to create one variable port you have to do a dense rank you can practice that one and you can complete it so kindly practice this if you have any questions you can post so i will answer it uh welcome to the session uh the next session is like sequence generator uh this is the ninth transformation uh sequence generator is like a passive uh transformation okay so far we have seen all the most of the transformations like our active transformation we have seen only one passive transformation we have seen so that is what our expression so now we have another transformation is called sequence generator this is also passive one okay so whenever the sequence generator is nothing but whenever we need auto increment numbers so we can go for this sequence uh generator okay so we can go go with the sequence of numbers like one two three four like this sequence of number you&#39;re going to generate then i can go with the sequence editor okay so auto increment numbers so wherever i need uh the sequence number for like a primary key or any surrogate key or anything then you can go with this sequence generator the sequence generator we have two output port so we have the two output port so one output port will be like one output port will be are the next valve the other one is kerval so this is next well port so this is kernel port so next value and current value okay so this is what the sequence generator will do sequence generator will create a sequence of numbers okay sequence of numbers uh see for an example like one two three four five like this whatever the number you want you can make it like it will incrementally it will generate the sequence number wherever it is uh needed okay you have many properties like uh you have many properties uh in sequence generator these are the properties okay so what is the first one it&#39;s called start value so what is the start value from which value i have to start so that i have to give it in the start value okay start value i&#39;m starting with some eight digit number okay you can you can start with any number like start with one also start to also like four digit number some any digit numbers you can make it that is what start value so you can make it like start value here so just i&#39;m starting with eight digit number okay it will start from this number increment by so it will increment by one one one so one by one it will increment end value it will go up to this number okay one big number like 19 digit number it will go okay then current value so current value is nothing but what is the current value so while starting the value this is the current value you have to mention cycle option okay cycle option means what is the cycle you want to cycle it once it reaches this value yes so that is a cycle option okay reset means you can enable or disable the reset value so that means you are you are enabling the reset value or disabling the reset value reset means for an example today it is starting with number okay starting this number it will go by one by one right it will go by our next number will be uh next to one next number will be two so like this like this it will generate so this is a sequence of numbers it will go with one by one right so sequence of numbers it will go one by one then you can so for an example if you are making like reset option what will happen if you enable this reset option next time also it will start from this number only so this number only okay next time also it will start from this number only right it will generate like this number so it is resetting again tomorrow it&#39;s again next run is running again it will start from this number only this number like this it will generate this is called resetting the option size cycle means if you are not enabling the reset and then you are enabling only cycle then next time it will start with six only okay six seven eight cut it in that way it will go the number of cached values what is the number of cached values for example number of cast values normally zero if you are making like 1000 year next thousand sequence of number will be generated and it will be kept the same catch memory so that it will assign very quickly so that is called a sequence of numbers here okay so wherever you need sequence of number you can generate it so that is what the sql generator to generate the surrogate key or unique key you can use sequence generator what is the surrogate key i will explain unique you know right already unique how the unique value will be generated so that you know i will tell but uh look at here one table i will connect it okay so if you look at here here you have the employee key right employee id and employee first name all these are like from employee id employee to department id i&#39;m going to get from source in order to maintain the history in this table i&#39;m going to use employee key as unique okay why because employee id i cannot make it like a history right if i want to make employee id as a primary key i cannot make it a history i want to i want to place some duplicate value so for that employee key i&#39;m going to use it so that employee key i&#39;m going to generate like something like a primary key values okay so look at here so mapping create so m underscore sequence underscore generator so i am going to take employees table from source but in target i will take employees underscore history table so look at here employees underscore history so i have one more column extra that&#39;s all okay so employees underscore distributed so this is a table i have so i&#39;m going with uh so employees underscore this one source qualifier then i will have this table employees called history from employee id to department id i will have it from here employee key i will make it like sequence generator okay this is the one so a to this one two three right this is a sequence generator i can keep here so this next valve i can make it here next valve okay so i told right sequence generator will have the two values one is next well another one is current valve okay so curve next value next next value to be generated so carver i have the current value click on it go to properties in the property start value will be so i&#39;m going to have start value like four digit number okay increment by one by one end value like this current value should be if the start value is thousand current value or so thousand cycle option is just if i enabled i cannot give if if i&#39;m going to enable the cycle option then i cannot give it for primary key why because primary key will not have repeated value right i should not give and i am not going to reset also number of casuallies i am not catching any values just keeping like 0 apply here ok okay save it now what is this meaning okay so this employee id to everything i&#39;m passing from source this employee key i&#39;m generating by using the sequential like next next sequence number will be generated from here okay so if you look at here i&#39;m going to create some workflow so create wf underscore i am not going to do truncate and load okay i will remove the trinket and load why because since i am going to make a history just i have removed the trunk it unload first i am going to start it it will load all the 107 record right with the four digit number it will generate sequence of number it will generate so it has been generated now you can see so value is generating from here right employee key from thousand it is generating and it is ended with this number one one zero six so if i&#39;m going to again i&#39;m going to run it start workflow it will get succeeded so we are not going to do truncate and load just we are incrementally we are adding the data you can run it it is incrementing right the value is getting incremented the sequence employee key is getting incremented but i&#39;m maintaining the history of the record right if the record is getting changed in the source so here also just i&#39;m inserting it so the same record i can maintain here in this table with the new key okay so this is what in real time would be used to maintain the history and up but we will maintain some history we will use some uh slowly changing dimensions say cdn on that and we will see after some time okay so as of now this is the one thing can we have some another example for the sequence generator yes we can have okay for an example say for an example this is my first record second record from the source this is my record right it&#39;s all the record so these records i have to load every fourth record to target i have to load every fourth record to target so how can i load every fourth record to target so this is a target table so this is one of the interview question they will ask in interview say for an example i have to load every fourth record right so here i have to this record so tomorrow the numbers might be different this number might be different the source incoming employee id might be different so i cannot go with some hardcoded value so for this i&#39;m just generating some sequence of number like this one two three four okay so one two three four one two three four i&#39;m generating for this id i am generating like this and i have something like fourth every fourth record i have to load it right so wherever the next value is coming up like fourth value i have to filter out i have to load it okay so this is something like next val port i&#39;m going to generate this is next val port so this port is next val port i&#39;m going to give from the sequence generator i can give right so what is the start value i can give start value start value is 1 correct increment by 1 end value end value will be 4 should i give cycle option yes i have to give cycle enable ok should i give preset yes you have to reset why why because if it is ending with something like one year okay one year then tomorrow it should not start with two here right so why because we are if you haven&#39;t enabled this reset option tomorrow it ended with again starts with two only okay so don&#39;t do that so like this you can go with so this is start value increment by end value all this okay now look at here i&#39;m going to create a mapping m underscore load every fourth record okay i can take employees table i can take p underscore employees table i have to load every fourth record right so for that i&#39;m i&#39;m placing one&#39;s filter transformation okay i&#39;m placing one filter transformation so this record i have to pass at every fourth record so what i can do i can generate sequence of numbers so you can see here sequence generator right you can place a sequence generator you can drag and drop the next value to here double click on it go to properties the properties you can make start value equal to one increment by one end value will be four okay end value will be four current value and cycle option okay cycle option and reset option that you have to give that&#39;s all so apply okay and this this will have one two three four one two three four right so wherever four comes i have to pass it to the next level only to take this one in the filter transformation you have to take properties you have to take only next well equal to four so next value equal to four you can ask question can i can i take next value to next level if you are taking this next value to target right what will happen here you you will have always four number four why because you are filtering that number only right so that is why no need to take this just along with this source data you are preparing one more column like next value equal to four so just i&#39;m sending it i&#39;m taking to the next level okay now you could look at here i&#39;m going to the workflow so wf underscore the load every fourth record so you can take the load every fourth record target go down truncate target table option okay so look at now this will load every fourth record to target okay so sometimes interview they will ask this question if i want to load every fourth record how can i load look at here now it has been loaded right so every fourth record only it will be loaded okay so control enter you can see this one not three or not seven one eleven right so every fourth record is getting loaded it got completed right now it will be correct correct only alexander diana only look at now second time i&#39;m running it&#39;ll be different now this time it might be different [Music] so next transformation is union transformation okay union is a is an active transformation so this is also an and the connected one okay so it is exactly opposite to router so what router will do router will do router single group to many single too many right router will do single to meeting but this union will do many to single so that means multiple input to single output so that is what the this will do so union is nothing but you are having multiple input single output so single output you are taking multiple group input single group output so this is one group this is another group three groups i&#39;m giving like input so one group first group input second group input so third group input okay the same structured table union right union set operation you should be having same structured one okay set operation should be same structured one so this is union transformation so same structure only have to give so union will have will give you union all in informatica okay so union will act as union all in informatica remember this that means union transformation will not remove duplicate you will get duplicate if i want to remove the duplicate then here you have to put one sorter transformation to remove the duplicate okay i want to remove the duplicate so union will act as a union all right so you have to if i want to remove the duplicate then you have to go with union all so why they are they have given like union all none then you have the starter transformation to create a union but if i want to have union r then i have to go with union transformation only so that is why they have given like union all here by using here you can remove the duplicate by using distinct option you know right in sorter you have distinct so it will come with union or duplicate also it will remove the duplicate so by this combination of union plus sorter will act as a union only union will act as union okay clear what router will do router is exactly opposite to union so router will do like this is router will do single too many right single too many single input you are giving so many output it will it will take so one two three four so again you can take again one two three four okay so like this this is default one if you want to take you can take so this is router right so router is nothing but single too many union is many to single so that is the difference between router and union remember this okay so i will give you this sheet so you can remember this union how the union will work so remember in in duplicate if i want to make remove the duplicate you have to put union plus sorter this is exactly opposite router multiple pipeline input single pipeline output multiple pipeline input single pipeline output okay for an example here i will explain what is this union will do so remember these two structures are same customer table and customer table so joins is different okay two different table you can do the joins same structure table you can do the union okay so same structure table this is union right so this is another table this is another table this is another table but same structure table so you can ask question here this mobile number here it is phone number can i do union yes column name might be different but data type should be same if you have two different data then you can combine it into a single resultant set that will be a union okay that will be a union so you have the customer table you have the location table two different table then you have to go for join but this is union union means it will remove duplicate it will not allow duplicate record for example here i have arun arun here so here also [Music] it will not allow this around this url will not allow okay so only one around this bottle also will not be allowed okay why because already bala is available so arun balarakis will be there then which is from here it will be there that is called union union all means so everything like six plus two everything will be there okay so that is you know okay so i will take one table here yes underscore employees underscore union underscore one so this table i will take look at here this has three records another table is union two same structure only but different records i have okay union two same structure only okay so union two i have you can take this data as flat file also more issues you have these records how many cuts are there you have three so four five six seven eight nine ten if you do union conservation you&#39;ll get all the 10 records okay so 3 plus 7 you&#39;ll get it in informatica also you have the target table t underscore union underscore union this is another table so you can have this source [Music] implies underscore union underscore one so i&#39;m taking first table so second table okay two pipeline you can take are two sources also you can take no issues so instead of two sources we can take two pipeline okay then this is my target table i can bring the union transformation here so look at here union transformation if you drag and drop one input right it will automatically create the output group look at here it is automatically creating the output group right so we are not going to create automatically create i have taken just input group it has already taken that output group so you can take from here to here output but i have to take this also here double click on the union go to groups how many input group are there two two sources are there just click on this add new group that sound click on the add new group apply here click on ok you can if you go down so one more group will be created you can drag and drop all this you can you can connect it okay so control yes it is a valid one okay it&#39;s a valid one you can see here the this is union okay we&#39;re getting the target so this is the one ctrl s you can go to this workflow manager workflows create so wf underscore union underscore transformation okay we can go down trunk a target table option now we&#39;re just giving so first it will do all the 10 records it will not remove duplicate see here union will act as union all now it will not remove duplicate okay see here did not remove duplicate right you will get all the to duplicate record also look at from making that ascending order we are seeing some duplicate record right duplicate means full draw duplicate okay so that is a duplicate it will take okay i if i want to interview they are asking how will you make this unionist remove the duplicate after the union transformation you place sorter transformation sorter here so you can keep double click so go to properties you can enable distinct apply okay now if you look at here so since i made it like sorter right you know already how the sorter will work distinct sorter distinct will remove the duplicator right so anywhere in the pipeline you can use it that is why sort of distinct we have given they have given just you can make now look at here i don&#39;t have any duplicate records see here clear how the union will act as so and also they will ask question so why union is active is an active okay so you are giving some 10 records here 10 records here it is giving you 20 records right it is not removing the duplicate how this will be union will act as active so why because union transformation will change the row idea of the record okay so if any transformation changing the row number or row id of the record so that transformation we will call it as active so that is the reason union is an active transformation so interview they will ask definitely this question why union is inactive so why because union transformation will change the row id so that is why union is an active transformation hello friends welcome to nikeit academy in this session we are going to learn about lookup transformation so lookup transformation is very important transformation in our real time project so most of the mapping we will use this lookup transformation so what is lookup transformation so lookup transformation is to look up some values from other another table so whenever we are creating a logic a real-time project so in the pipeline if i want to look up some value then i can go for the lookup transformation we have two types of lookup transformation one is connected lookup other one is called unconnected lookup in this session we are going to learn about connected lookup in our next session we will learn about uncorrected lookup after that we will learn about when do we go for connected lookup and we go for unconnected lookup so what are the difference between connected and unconnected we will see in later consider this particular example this is the connected lookup example if you look at source we have that table called product i have product id supplier id product information all that related to product information so i have only product id but in target our next level transformation i need to have the product name by passing the product id to the table called product description i will take the product name and description okay to the next level transformation are in the target so in order to take this product id and product name either i can go for joiner or i can go for the lookup so if the table is there the source side i can go for the joiner if the table is there and source our target are if the table is flat file or anywhere in the pipeline if i have the table i can go for the lookup transformation so this is what i look up we can pass or look up a value and then we can join and then get the required details we will see this connector lookup in detail please watch this session without skipping it let us start our session [Music] so this is the 11th transformation so in lookup we have two types of lookup so one is connected lookup another one is unconnected local okay so what is uh lookup this is a lookup is an active transformation and we have connected as well as unconnected lookup transformation we will see the connected lookup first then we will see the uncorrected lookup what is lookup so we know already we have seen that join a transformation the joiner transformation will do the joining right so whenever i need to have the joining column i can go for the uh java transformation the same way the china but joiner need to have the joining column only in the sourcing but in the lookup we can have the the lookup table either it can be might be in the source or target or you can import it newly so that is what the advantage of lookup transformation connected lookup is nothing but it will be on the pipeline okay maybe on the pipeline so you&#39;ll be having something like uh many uh mini port so in lookup we&#39;ll be having input port output port lookup for tundra so we will see one by one and then if you look at here say for an example i have the employees table you are the employees table the same like joiner how we will do the same way look up also we will do think about via the employees table it has employee id so all the columns are there underscore id all the columns are there like up to department id okay so up to department id you have all the columns then here you have the t underscore employees underscore department is another table we have is called a t underscore employees underscore department another table okay so this has like so employee id all this data like employee to department id so then department name you have the department name then you have the location right location all the columns are there i need department name and location agree so what i can do either i can go for the joiner or i can go the loop so this time i&#39;m going with a lookup so this is my source qualifier you know already so how the source qualifier will be working so this is my source qualifier okay this is my source qualifier then you are there you can connect from employee id to department id you can connect so from there you can have all these columns you can have all these columns like all the columns you can have so like this you can have these columns so you can have all these columns by one by one then you can you can keep this like if you want the department name i want i can go with the i have to look up the table right i have to look up the table so this table i can go with the lookup table so this lookup table we will call it as lkp trans okay lkp trans so this table we will make it like departments table we will look up the department&#39;s table okay remember this same like joining same like joining we are looking at the department&#39;s table in this table we have some columns like yeah the department id will be there right so in the lookup table you&#39;ll be having department id then you&#39;ll be having department name the lookup table department name then you&#39;ll be having location id so all these columns will be there here so these columns will be there in the department&#39;s table okay so now i&#39;m going with this column drag and drop this column from here drag and drop this column from here so this column we will take it as a loop column so this lookup column we will call it as in underscore department id department id that we will make like department id then join these two these two columns department id with the department id then you can drag and drop the columns department name and location id clear now so this is what you have to to do the lookup so this is called the connected lookup because the table is there in connection okay so i will take the same designer i will take uh this table uh i&#39;m going to create the same thing so m underscore connected underscore lookup connected underscore lookup you can see here source i&#39;m taking the table same thing i&#39;m repeating okay so whatever we have seen on the day just repeating that we can take p underscore employees underscore department this table i want to do the lookup so from employee id to department id i&#39;m loading here from the department name and location id i&#39;m doing the lookup so how to do the lookup just click on the lookup transformation here lookup table is department stable you can take either table or flat file anything is fine so you can take the flat file sorry table and from here you can pass the department id department id to department id i can i can match it these two double click on it go to condition you can place the condition like department id equal to department id here okay then you can take the department id sorry department name and location both i can drag and drop control yes okay you can arrange it all you can see here so we are passing department id from here if you do not want this manager id you can remove it you can remove it no issues so you can have this department id department name location id condition department id2 department writing okay apply okay yes save it so this is what the connected lookup and go to the workflow manager okay i&#39;m going to the workflow manager so in the workflow manager i&#39;m creating one workflow okay so wf underscore connected lookup underscore one i&#39;m making okay then take this session connected lookup drag and drop whatever we have seen on the date just repeating it okay pointing target to target if you want truncate and load targets you can go to the lookup transformation you can you can do the look upon source player okay save it and then run the start workflow okay start workflow now so it will load the data it got completed you can see here you can see here you can have this table 107 into or not so okay so the data has been loaded now so you can see been loaded you can go and check here in the table so if you have this data has been loaded okay just it has been loaded with employee department name and location so for example i want to know i want to load only purchasing department where can i apply filter so can i apply filter in the lookup so if you are playing the lookup then it will become like a leftover join only so if i want to make sometimes interview they will ask how will you make lookup like inner join you have to make flat some filter here filter apply the filter so keep the filter here drag and drop all the columns from here to here and department name and location id you can keep it here wherever the department name is coming like something like finance department right department name equal to finance purchasing or anything you can make it like purchasing and then you from the filter you have to drag and drop to the next level seem like control yes valid refresh this refresh mapping save it start workflow and see now and then it will load on only six data six record only why because i have only purchasing department six employees are there so that is why it will do the lookup sometimes interview they will ask how will you change the lookup as inner join so this is what see here now six record only it&#39;s loading you can see here now six record is loading right so this is what you can see this value so this is what the connector lookup with single table say for an example i want to do the connector lookup on some two or three tables for an example i have one more table called department underscore lvoc another table okay this table i have to load it i want to load all the department as well as location details also in the table i will go with one more loop lookup okay so connected lookup only connected lookup two tables two lookup tables okay first table i will take it like employees table only source table here i will take employees department and location targeting so normally i will give this as an exercise but here i&#39;m i will do it okay so that you will get idea so how would it be so this table i&#39;m taking so this table you should have created for your exercises like uh source qualifier exercise our joiner exercise you should have created in source i&#39;m going to take locations or you can take it in the runtime also say for an example here first i want to drag and drop this table target table so what are the column needed from employee 82 department id i need all the column name location id state address city and the country so these three column is coming from locations table okay location save so what first i can do first i will do it like departments table lookup so whatever we have done it previously that we have to do so go to this first department stable i&#39;m doing the lookup department&#39;s table okay so look at now so you pass department id from here match this department id and this department id double click on it condition match it here you do not want this manager id you can remove it so you can bring this department name and location id from here you can pass it over here okay then i need state address city and country so from where i can take it this table these three columns i have to look up on the locations table right so how can i do the lookup on location table you have to go with location id so here i have the location id by passing this location id to next transformation i can get it say for an example here click on one more time lookup table click here so this time i will make import import relational table so where from the source is there so yeah the location table is there the source right so that is i&#39;m just connecting with source so i&#39;m just importing from source that locations click on ok so from here i&#39;m passing the location id from here okay you can ask question can i pass from one lookup table to another lookup table yes you can pass it this location id this location id i&#39;m going to match it i&#39;m going to get this street address and the city and country so these three column i&#39;m going to take it okay these three columns i&#39;m going to take it the postal code i will remove and i will i will take it like state i will remove it i will remove i will i will keep all these three columns okay okay look up condition is location id to location so one lookup table to another lookup table so we can go here workflow create so wf underscore connected lookup underscore two lookup tables two tables so you can keep you can make here so only thing is here is so one uh extra thing is in the the lookup you have to provide the connection lookup you have to provide the connection this lookup table is in source site don&#39;t forget this this is the lookup this is also source so this is also source this is also source okay how to give click on ok save it now if you run it will load the data into this table this table as up now we don&#39;t have any data but if you run it it will load the data so you can see here 1007 recorder has been loaded so it will load it with the record details look at here so all the department name location id street address city country everything has been loaded right so this is what connected lookup two tables i hope you are clear on the connected lookup [Music] hello friends welcome to nikkei academy in our previous session we have seen about connected lookup in our today&#39;s session we are going to learn about unconnected lookup in connected lookup the lookup transformation will be in the pipeline in unconnected lookup lookup transformation will not be in the pipeline if you haven&#39;t watched the previous session on lookup connected lookup please watch the session and then continue this session so in our today&#39;s session we are going to have a employee stabilized source department tables lookup table we are going to load department name and location id in the target please watch the session without skipping it let us start our session so connector lookup and current lookup unconnected lookup means it will not be there in the pipeline okay look up table will not be there in the pipeline same thing we will use but we will not put the connection so like this for the table but you need the these details right so definitely you need to bring you need to bring expression transformation so expression transformation is mandatory in this unconnected lookup okay in unconnected lookup expression transformation is mandatory one so i can take the expression transformation you take all the column from sports so up to m up to department id will be having it so department id will be having all the columns so this is what that all right you can go for expression transformation expression so what column you need department name and location id so that column you have to create from here so i&#39;m creating go underscore department name output port you have to create s underscore output so that particular column so here also you need pro underscore location id so that column you have to create it from here location underscore id so here we have write some formula so you can take from employee id to department id right so you can take from here it&#39;s from your employee id up to department id you&#39;ll be having all the columns right up to department underscore id you have all the columns but i need the department name and location so that is what we are doing there so department name and location we will take all the columns okay so now if you look at here i need department name right so i have to write the formula so this is the formula remember so this is the formula what is the formula right colon transformation name then the joining field passing field which field you are passing so that particular field you heard right passing field means the joining field which field you want to join so that particular field you got right so whenever you are writing colon lkp in expression transformation informatica assume that okay you are going to do account so here you have to do the uncorrected lookup for department name colon lkp dot ltp trans the lookup table name from here you are passing the department id from here you are passing the department id so remember for the uncurrent lookup you can you can return only one port you can return only one port so to return a location id i&#39;m going for the one more one more column call on lkp dot lkp trans one department id so one more lookup only i have to go with not on the same lookup so i&#39;m taking the first look up here i am taking the first look up here so first look up will be your lkp trans second lookup will be lkp trans1 so i assume that this is my lkp trans so this is my lkp transform in the lkp trans we have to take same so which table is lkp trans which table i have to look up it&#39;s a department&#39;s table right department stable that is what the lkp transits department department stable only i have to look up so this is same department stable but lkp trans one we have to take same department table lkp transform in this table we can take department id department name then here we have to create our own column okay we have to create our own department id and then location so from here so first table we have to take from here a column from here as department id in the underscore department id so just we have to take this this column we have to create as input port input port we have to create it okay in underscore department so these two we have to join it make the condition of these two and take this fortress return port take this port as little board so here same way you have to pass in underscore okay partners for id you have to take these two columns to join again this time you have to take location ideas written book so for an example here i want the department name so what informatica builder call on lkp assume that it&#39;s unconnected lookup dot lkp trans transmits it will go here it will pass this department id then this department id and the department table department id will match department name it will bring us return and leave it here so only one written boot you can take if you want to take multiple return port then you have to go for the concatenate function okay you can write so if i have only one table you want to take multiple return vote can i do it in the single lookup yes we can do by using the concatenate function so that&#39;s one column we have to take from source and then so in expression we have to split it this is what we will do the unconnected lookup same we will do it in the informatica like m underscore and connected lookup okay so i&#39;m just taking so look at here employees table t underscore employees underscore department table for uncut lookup you need to have expression transformation take the expression so drag and drop the column to here so double click on the expression you create whatever the column you want for next level i want a department name and location id you can make it like output also go underscore department name string and go for some 30 to 40 characters location id also number decimal so make this output port department name it should be old lkp dot lkp trans trans bracket but to pass from here department id okay department same thing you have to take same subnet will not be validated by because we do not have any bitcoins so here also you gotta take this but its lkp transform okay so we are passing the same department id and so this is for lkp trans this is for lkp transform that&#39;s all you can connect it save it it is not valid now you can see here invariant so first lookup you can take the first one okay which table we are looking at departments table just you take the department&#39;s table click on ok so first one you can you can have this department name and there is one you can remove this manager id from here location id from here in underscore department in underscore department ready okay written book department name is you can return it okay that&#39;s all you can the condition you have to join department equal to department then again you have to go for the lookup one more lookup department&#39;s table same department table this time instead of department name you have to keep the location id you have to create one more column that&#39;s in underscore department okay so this is what input port whichever the port you are making like input port that port will be that value will be passed here and return what you take the location id this time so the condition you can make a condition here to part so then right click arrange all then the lookup will come into the front you come in the front so this is what it will be uncorrected on okay and current at home i will make the workflow wf underscore unconnected lookup and go for unconnected logo you try today that look up two tables so lkp trans you can go for source only so trans one also source only click on apply okay save this you can make start or two it has been loaded you can see here same thing will be loaded okay we have truncated and loaded right then same thing it will be loaded you&#39;ll have that null value now maybe because previous one we have removed the null value but here we haven&#39;t removed so if you want to remove then after this expression you can make one filter after this expression you can make one filter okay so that is what you can make the unconnected lookup hello friends welcome to nikkei academy in our today&#39;s session we are going to learn about how to return multiple columns from uncharacter lookup normally we have already seen that unconnected lookup we can return only one written port per lookup say for an example even in the same table we want multiple port we have to use multiple lookup but we have some workaround to return multiple port so that&#39;s what we are going to use please watch the session without skipping it you will be getting lot of stuff here this is the one of the question asked mostly in interview please watch it completely so that you will be getting all the stuff let us see let us see practically so we are using two look up right to look up transformation to return two different columns on the same table so first time we are returning department id second time we are returning location location id department name and location id is it possible with the only one transformation to return department name and location id it is not possible with only one transformation is not possible but we have some workaround so what i&#39;m going to do i&#39;m just copying this copy the same lookup uncorrected lookup multiple return port how to take multiple written book multiple written port is not possible directly so i am going to use 1d lkp trans i am removing this this particular one more lookup and just removing per table only one lookup we can use for this scenario also we can use only two lookup instead of five lookup we can use two lookup so 2 lookup for departments table these 2 one lookup another 3 is for locations right for location we can use only one so only 2 lookup we can complete how to do please look at here so this one same table we are looking up right same table we are getting it so we have some work around for this so what is it we can take department name underscore loc both both of the column in same port how to take see look at here i am going to use so this is i&#39;m selecting from the department&#39;s table right department table so i&#39;m going to choose department underscore id comma is it possible to select a department name and location same column yes it is possible how to take you have to take department name concatenation concatenation location underscore id so location underscore id as this one see here department okay sorry this is a department&#39;s table right department&#39;s table so just i&#39;m selecting it so this is what i&#39;m getting so in one column i&#39;m getting two values right one is uh department name another one is location we can return this but if i return like this i cannot i cannot take only administration i cannot take only location id so to have this segregation i will use some delivery here delimiter okay some delimiter i have used like look till the symbol i&#39;m using is this one so look at here this one i&#39;m using then using that select department ids department id so department id this one has location from this one so this is what i&#39;m using it so this query we are going to use in lookup okay lookup override so how to use it make whenever you are writing this query right so make it like uppercase sometimes uppercase lowercase it will pass some [Music] issues so you can use this as an uppercase controls e i have used so where can i use it so here we have to use go to properties the properties lookup sql override right so here we can use it so this particular query will be executed first column is department id that will be assigned to a department id column second column will be assigned to department name location column single column okay it has single value so if you have many columns you can put all the columns in the tilt symbol we can return it in one single port single port only we can take but that we are taking in this particular column so we can go for more than 30 characters like 100 characters i&#39;m going with clear on this okay so we have to receive this column here which port i can receive can i use this one no i will use one more port called the underscore department underscore location dept underscore name underscore location so this column i will receive it first okay whatever the record board as a variable port why i am using is variable port variable port for intermediate calculation idm passing but this time i will get department name and location okay department name i have to get it right so how to get the department name say for an example this is what you are getting as a result from this i have to get the administration from this i have to get a thousand seven hundred per location by using the tilt symbol i have to write common formula for all the sorry because i cannot use some number of characters for this alone then this is different this is different right so everything is different different that&#39;s what we are used to till the symbol how to get the administration alone from here so you have to use substring sub str substring of the given string right single quotes comma one comma how many characters i have to take 1 2 3 4 5 6 7 8 9 10 11 12 13 14 okay one comma 14 i have to use correct from dual so administration i have taken but is it this formula will work for all the all the data so for this one is correct this one is one comma two right so to identify the tilt symbol i will go for the in string ins tr in string of this one where this tilt symbol is presenting so 15 to place right 15 to place so you have this is 14 to place in order to take this one where is a tilt symbol is present so you can use single quotes tilt symbol it will return like 15 so i have to use so here it is a 14 but it is returning 15. so what i have to do remove this 14 remove this fourteen you paste it here you make minus one okay minus one see here administration has taken correctly we are not doing any hard-coded value so instead of this we are going for any other data say for an example i&#39;m using i t department right so here also i&#39;m using id it will return it correctly so it will take that&#39;s what we have written this formula is for department name okay i will write this formula for department name so instead of hard coding this what i have to use instead of hard coding this what i have to use this port name i have to use this port name you have to use clear okay so for this i have to take thousand seven hundred how can i take thousand seven hundred how can you take thousand seven hundred alone from this so you have to go for sub string of this one so you have to use instead of one you have to find out the tilt symbol correct from each character so that three symbol plus one i have to use not minus one this time plus one that&#39;s all see you have to learn first substring and in string so i have already posted some video on substring and instinct please watch it okay please watch the substring and instrument so you can get it so this is what you can go for the substring and instrument options so this one you have to use it here i will take this one next this one i will use so instead of this i&#39;ll go for this one for this one click on ok now play okay here save see we are used to only one lookup now okay so now i will create a workflow i will show you whether it is loading correctly or not unconnected lookup unconnected lookup multiple return port multiple return vote so i have given here you can see the target i&#39;m just using this target look at now so underscore dept right this table so i&#39;m going to start it will load the data okay it got loaded now you can check it whether it has loaded correctly or not yes it has loaded correctly for all the records right so this is what you have to use multiple written port in single unconnected lookup clear we have to use concatenate function to take that [Music] hello friends welcome to nikeite academy in this session we are going to learn about lookup policy on multiple match we have already seen connected lookup unconnected lookup how to create a lookup and how this lookup will be work everything we have seen now this is one of the interview question what is lookup policy or multiple match so because of this property right informatica look up transformation is called as active transformation it is an active transformation lookup why look up transformation is an active transformation because of this particular property the lookup transformation is called as active transformation so we will see what is that lookup policy or multiple match whenever we are creating a lookup table say for an example from the source we are the department id 30. from the source we are the department id 30 in the lookup table that means our departments table normally we should not have any duplicate value in real time project sometimes we will have some duplicate records it doesn&#39;t have any primary key so that you&#39;ll be having the duplicate value so if you have duplicate value our lookup table will go and check the data corresponding data for 20 we are getting the match and we have one more 20 on the same table so if you are getting multiple match which one i have to consider either i have to take admin or i have to take sales so based on the lookup policy we are to choose from here we do have four different options in the lookup what is that option use any value that&#39;s a default condition use first value so first value means it will take from the last so lookup will create in this way so that&#39;s what it will take sales use last value it will take admin and report error that means session will fail saying that multiple match found so this is the lookup policy and multiple match in addition to this we have one more option is called return all values on multiple match what is this so for one record in the source how many matches we are getting it all the record we will get it so 20 will get admin same 20 you will get sales department so one record in source we are getting multiple record in target so because of this property the lookup transformation is called as active transmission so now let us start our practical session how it will work kindly practice this session if you are getting any doubt or any clarification kindly raise your questions to comments i will respond to it and if you haven&#39;t subscribed my channel so please subscribe the channel so that you&#39;ll be getting all the update so we will see practically now so look up policy on multiple match so i will take this one m underscore lookup policy lookup policy okay i will take so i will take the employee stabilis source p underscore employees department as target so we are going to look up the department id from employees table department id from employee table i have to take department name and location id we know already that okay here we are the employees table so from the employees table select start from employees table we are going to have this department id so department id here okay so for an example department 800 you take so 100 you are having only one record that&#39;s okay it will take 100 means it will take select start from departments departments table so if you are getting only 100 the department that&#39;s working it will take finance department right if you are getting 200 here is it possible first of all in department stable it is not possible departments table is a primary key column it will not have any duplicate value but some table in real time project some table we will have a duplicate value also without primary key so that time so we have to take which one we have to take either finance department or any other different that&#39;s why look up policy are multiple match so this time we are going to look up the flat file why flat file right in the flat file i have added some duplicate data so go here informatica 10.2.0 server infrastructure lkp files local files here i have i kept some departments same thing in the departments i the last record i kept it like duplicate value so 30 here purchasing department correct the same 30 i have made it like insurance department there is multiple match so informatica will confuse now whether i have to take purchasing or i have to take insurance per department name for same 30 right so look at now how can we do with the lookup so from here i will be passing all the columns i&#39;m going to look up so this is lookup here source site so look the flat file is not there so just import import from flat file so you can make this one so this is what you can go for departments click on okay import field name from first line comma this is what so i can go for next column numeric text this is manager id numeric only this is also numeric only listed so connected department id department id match it so since it is a flat file don&#39;t delete any columns okay since it is a platform don&#39;t delete this column if you are deleting then location id the manager id will be placed to location okay don&#39;t do that so which port i can take you go to the properties lookup policy and multiple match so what is on what is the one i&#39;m using now use any value either it will take purchasing or it will take insurance for that department id 30. so i&#39;m taking department name and location id so connect okay so connected we are connected to control yes it&#39;s a valid node okay we have and click valid and go to workflow create a new workflow wf underscore lookup policy lookup policy on multiple match okay lookup policy i&#39;m just creating the session lookup policy connected double click go to mapping so should be pointing to source target should be pointing to target and you can go little bit down you can see lkp translate lookup transformation since it is a flat file we will not have any connections so here one thing we have to make sure that we have to look up source file name we have to make that source file name this is a file name so copy this file name you have to give it here so you don&#39;t forget to mention the extension file extension dot csp comma separated value file if you are not giving dot csv it will not identify the file name so we are given the connections and then truncate target table options just i&#39;m giving click on okay so here i&#39;m going to execute this particular workflow it will truncate the existing data and then it will load it we will see what is the value it has been loaded for 30 what is the value it has been loaded for 30 whether it is taking a chasing department or it is taking insurance department so we will see now go to the core tables you can see here so what is the department id is taking department id 30 is taking purchasing only it is not taking any insurance department correct if i want to take insurance department what i have to use look up policy or multiple match go here i am using use first value use first value means from the last it will take so it will take this time it will take insurance department for same 30 it will take insurance department it will consider us okay these employees from insurance department not from the purchasing department has completed now you can check the same same 30 it is taking for 7 30 it is taking insurance department now right insurance department so that&#39;s what use first value you can go for use last value so that will take the first value first value in the file we have mentioned like purchasing department so now it will go with purchasing use any value means informatica will randomly choose the runtime either purchasing our insurance department if you are using first value it will take last value last well means first value in the order report error it will report it will fail look at here now it&#39;s insurance right since we have taken last value it will take first value that&#39;s a purchasing department it is taking for department id 30. okay i&#39;m going to use report error if multiple matches found it&#39;s a wrong scenario you you fail it you fail the session okay so that we have to do it so what i&#39;m going to do so i&#39;m going to make config object stop another equal to 1 so normally this will be taken for all the session right what is stop another equal to one if any one error will stop the execution control yes i am going to start it normally stop on error and this one we will give it for all the session it will fail now the session will fail saying that multiple match found report error see here it got failed saying that multiple match found in lookup that&#39;s what it is failing then it will not load any data the target okay if multiple match found is failing right so another option is while multiple matches you take all the multiple matches like this so 20 you take admin also 20 you take sales also both you can take it so one record in the source two record in the target that that is what uh active transformation right so how can i take multiple all values is that any values for all values no so where can i find all values while importing the lookup you have to give all wells look at here while importing the lookup so while we are taking the lookup right that time we have written all values on multiple match this option we have to enable return all values on multiple match from flat file so we have to use same flat file departments click on ok same way we have to give all the values and everything click on finish department id department ready you can take this department id this department id match it if you go and check the properties now this properties is on disabled state look at your disabled state so once you have made it like active transformation then after that you cannot change it like a passive that&#39;s why they have disabled it use all values okay this option we cannot select any other option here so once you have used all values that&#39;s all so all the properties other properties are disabled because of source is a flat file lookup is a flat file okay apply here okay here then we can give here see here there is some twist what is that can i use directly from active transformation to target is it possible look at here the blue color you are saying concatenation disallowed on transformation this one is not possible we cannot take it here then how can i take it so we have to take these two columns right can i use expression no expression also we cannot take it so if i&#39;m using expression see here i&#39;m taking the expression here expression also same thing so we cannot use this is a this expression is a passive and this is a one we are getting it see here it is not connecting the department name and location it is not connecting so we cannot use expression also what i can do one workaround is i&#39;m using as jaina transformation you can go for any other different logic i&#39;m using as jaina transformation these column i&#39;m taking from the source these two columns along with the department id i am taking from the from this by using this department id this department id i will join it so department id from source department id from here i will join it but still both are like same only see this department id this department is same only but the giant condition i&#39;m giving it so whenever we are giving like this right whenever you are giving like this double click go to condition so here i&#39;m using department id here i&#39;m using department id whenever we are using like this look at now one changes we have to make so department id location video everything i have given so if i&#39;m going to save this it will not be valid one see here it is not valid on mapping it&#39;s very invalid so why it is invalid we have to check here okay each input of china transformation must originate from different source qualifier joiner or vsam [Music] since we are taking the columns from the same one you cannot take it take it directly so for that i can go and use sorted input option if you are enabling the sorted input then this will become a valid one but while in the runtime it will fail because of we have enabled the sorted input we haven&#39;t used sorter right so that will fail so remove this use one circle transformation use one starter transformation to sort it out so what is that department name location you sorted based on the department id here so you give all the three columns to this three columns same way from the source also we have to give sorted input so remove all the columns use sorter here the source double click here you can go to keyboard you can use the same sorting columns so now you can use all the values from here okay save save this control yes it&#39;s a valid one okay since we are taking this one since we are getting the same department id here right department id column from here department id column from here so you will get all the matches am i correct this department id we haven&#39;t taken from that location sorry from the department&#39;s file we have taken this department id from same column same table this department id we are taken from same table right same table that&#39;s why it will go for cartesian product that&#39;s why we&#39;ll go for cartesian product i will show you so refresh it double click go to mapping since we have removed the lookup the lookup file name has been removed so you can take this dot csv so look at here i have given that file name and starting the workflow since we are taking the department id from the same table it will go for cartesian product so we have to remove the cartesian product how to remove that i will tell you this is one of the interview question they will ask definitely an interview please make sure that see how many record you are getting one not seven or not seven you&#39;re getting three three one six so what we have to do you have to check here so for one record you are getting multiple records this time we are getting 30 insurance 30 purchasing for same record but multiple times we are getting correct multiple times we are getting to remove this multiple duplicate record how can i remove the duplicate record which transformation i can use after this joiner so can we use sorter to remove the duplicate so use sorter distinct to remove the full draw duplicate right so we can use this select see still one more workaround is there that i will tell you we will validate the data and then we&#39;ll see now instead of three three one six it will reduce all the full draw duplicate it will delete since it is going for cartesian product we are getting like this full duplicate i will remove it look up policy on multiple match has been removed so you look at here so for each and every record we are getting only one record so what is this record we are getting for 30th department for one record in source two recording target correct so only two recorded one is for insurance one is for purchasing this is what we are getting it clear but still we have one uh logical error so this is six records right this is six records am i correct for 30 years department we have six six records for this six record we are getting 12 records output 12 records we are getting so source via 107 plus 6 record extra we have to give 1 1 3 but here 1 1 2 only we are getting why 1 1 2 why because it&#39;s a due to we are making in joiner we are making like normal join if you are using likes master outer join master outreach left outer child if you are using master router whatever the record it has been filtered in the inner join that look at now you&#39;ll get that null value record also see this 178 you are not getting now right 178 you didn&#39;t get it see look at your 177 179 that null value record we didn&#39;t get it but now we will get it one one three record see these are all the transformation you can check it and then you can see here now one one three and you can go here and check see based on the scenario we can use it the transformation and we can do the work around clear see for this you should know what is sorter what is joiner what is sorted and put in the joiner everything i hope you are clear on this kindly practice this look up policy on multiple matches kindly practice this then you will get all the stuff very clearly so thank you if you are clear then we will meet our next session [Music] in our previous session we have seen 11 transformation this is a 12th transformation these 12 transformation only frequently used transformation in our real time okay these 12 transformations are frequently used to transformations after that we are going to see five more transformation that is based on the project we will use like a java sql transformation normalizer transaction control stored procedure and all we will use based on the xml we will use these transformations are based on the project the requirement okay here we are the update strategy this update strategy is nothing but so so far we have seen only insert so we are going to insert the data right just we are inserting the data in the target we have truncated and loaded right but in our real time see remember we will for all the tables we will not do truncate and load only uh only staging table we will do the truncated node remember this in real time don&#39;t enable truncated node for all the tables it is not advisable only staging table we have to do that truncate and load what is staging table i will explain the project explanation okay so now we will go for the update strategy update strategy it is an active transformation under connected one is an active under connected one and we will go for the data driven so dd is nothing but the data driven data driven means based on the data it will drive so whether it has to go for insert or update see look at here by using update strategy we can do four different operations one is dd insert okay one is dd insert dd update dd delete dd reject these four operations you can do what is dd dd is nothing but data driven so based on the data it will be drive uh whether it has to go for insert or update or delete or reject either you can use dd insert or you can give zero or one for dd update you can use one you can go for two or three all this so instead of deleting normally we will use these two very rare case very very rare scenario 90 percentage 95 percentage we will use this one only very very rare scenario we will go for dd delete and reject even instead of deleting the hot hot delete it will go for soft delete soft delete means instead of deleting the particular record we will go for the we will inactivate the record by marking some plugin that is what we will go for the update strategy update strategy okay so in order to use update strategy remember you should have a primary key in target table you should have a primary key in target table clear on this okay so now we will i will explain one scenario where we will use the update strategy uh look at this particular table see this is very important uh session in real time whenever you are going for any interviews or real time project so definitely this scenario will come into picture okay so definitely be using this so please watch carefully if you have any questions you can ask me you have source via the customer table you assume that source we are the customer table uh each and every time this is like your oltp system so oltp system each and every uh time you will be getting the data this oltp right oltp you&#39;ll be getting the every day you&#39;ll be getting the customer data in oil ap we have table called dimcast okay dim underscore customer is nothing but dimension table it is nothing but a dimension table okay this dimension table will have okay we have a primary key dimension table will have a primary key and attributes these are all the attributes customer name email mobile number city country all these attributes we have okay what is the scenario see if i have the record say for example very fast today you are running this workflow in our real time project okay very first time you are running this upload the project then we are we want to insert the record into the target we want to insert the record into the target we can insert it see look at here these are our new record today it has been newly customized has been added so what i have to do i have to insert this record into the target table your your data values okay how can i find these are the new record see this customer id is available in the old tv that&#39;s your source table it is available but the same customer id is not available here okay you compare this customer id and this customer id by looking up the column okay you take source customer id and target customer id by lookup okay compare you&#39;ll be getting thousand one in the source you&#39;ll be getting null value against this thousand one if you are not getting any thousand one here that means that is a new record okay that record is not available in target so then you can mark this insert flag is true just we are marking is sorry this one is true we are marking this particular flag is true very first time see all the flag is very first time we are making like true so if it is insert flag is true obviously the update flag is false false means the record is newly going to the target we are not going to do anything okay if it is true then we are inserting the record into the target table just we are inserting it very first time all the record will get inserted into the target table very first time all the record it will get inserted into the target table okay this is the insert plug so what is the formula for insert flag if it&#39;s if it&#39;s not lookup customer id then we are setting up this just true if this flag is true we have to insert the record into the target just we are informing informatica okay these records are available in your source it is not available in target go and insert it okay so next time you are running it next time you are running it what could be the insert flag for all the record you have to set up whether i have to set up the engine plug if i&#39;m going to next time if i&#39;m going to run next time i&#39;m going to run this record is already available here right should i make the insert plug is true if you are making then it will go for insert you&#39;ll get primary key violation to fail okay don&#39;t do that you make it like a false okay it also falls see all the record also you you have to make it like a false if it is false false then the record is matching with this record target record okay source record is matching with target record do i need to do anything if the source and target is matching do i need to insert the record or update the record no okay you assume that next day next day the session is running one particular customer has changed the mobile number next today are some sometimes later okay one particular customer has changed the mobile number you assume that what i have to do i have to compare each and every record one more time this record is record is equal then any of the column is not equal if any of the column is if any of the column is not equal then i have to any one of the column is not equal i have to update it okay i have to update it how can i update if any one of the column is not equal you adapt it so you can go for this record you take this record you take both are equal right both are equal then you are checking whether this is equal yes it is equal this is equal yes this is equal not equal if it is not equal set this update flag is true that means we have to make it like a we have to update the record that we have to make this particular flag is true if the update flag is true we are to update this record based on the customer id keep the customer id and update this mobile number not only this mobile number all the columns will get updated here simply update okay if i&#39;m asking what is the mobile number of previous mobile number of vijay i do not have information right it will simply update the record is called a simple update simply it will update the record this is called a simple update so it will be simply it will get updated i have given this table just to create this table target also you create insert the record into the table to a target don&#39;t insert it okay if the record here it is deleting if you want to delete it here then we have to use dd delete operation that i will tell you okay okay i hope you are clear then we&#39;ll go for the table see here i have the table here source and target table customer update the other table called select star from yes underscore underscore update update strategy i have made it like this is what we are the data so one two three four okay we have four records i will go to the target table target table first of all i will manually trunk it manually trunk it okay if i have any data here i will truncate this table but remember target should have a primary key target should have a primary key okay so look at here select star from t underscore customer underscore update okay this is what we have now we can go for folder and we will create mapping m underscore update underscore strategy update strategy this is called a cd type one what is type one i will tell you a cd type one i will tell you what is this is whatever i have explained so far that is a type one so i will take the source table hr okay you go for hr table and then i will make employees underscore sorry customer underscore update source table source you may have primary key or you may not have okay but in target we should have a primary key target we should have a primary key look at here target we should have a primary key customer id should be a primary key okay so i will go to the mapping here i will take the source table source table i am taking here i will go for the target table t underscore customer underscore update target table i will go for okay so now i will take first of all i will look up the target whether this record should go for insert or update i have to look up for this i will take on expression transformation i will take all the columns i will look up the target now and look up the target again source customer id take the lookup here so far we have looked up only source table right so this is the difference between china and lookup one difference in lookup you can look up anywhere source or target anywhere so you can look up this customer underscore update it has been created against this customer id okay this customer id this customer id we are checking and then we are taking all the remaining say these columns you are taking from customer id to all the columns see these columns from lookup double click go to condition you make customer id equal to customerid clear lookup condition so we are comparing this customer id and this customer id if you are getting null value on this customer id we are making this flag as true if it is equal both are equal then other columns we are checking any of the columns is not equal we are setting up this flag so we have to create two flag one is insert flag other one is update flag so just i&#39;m creating a flag here see these columns from lookup these columns from lookup so i will make lkp underscore so i&#39;ll make lookup columns to just naming sake i will take lkp underscore to denote these columns from lookup is it is it compulsory to make lkp underscore no it&#39;s up to you if you want you can make it lkp underscore okay you can keep the cursor here you can go for add a new port here i can make insert flag or update flag so insert plug is this one we&#39;ll make this one you can go for insert flag [Music] customer id then it is true otherwise it&#39;s false clear any doubt this logic is clear if condition the condition is true what i have to make just i&#39;m making this flag is true okay otherwise false clear validated should be valid okay update flag i&#39;ll go for update flag what is this update flag we go for if same way if this customer id your source customer id equal to target customer id if both are equal and under condition you have to make and if any other column is not equal okay and any of the column what is that column customer name not equal to lookup customer name or anyone right that&#39;s what our email id not equal to this email id are this mobile number not equal to this mobile number are the ct not equal to the city are this country not equal to this country if any one of the column is not equal then i can make it like true otherwise make it like false clear see look at here if if these two are equal if any of the column is not equal any one of the column is not equal then we are making this column is true or false click on ok now click on apply ok so here we are the two flags based on the flag we will route the record whether we have to go for insert or update so we will take the router here we will route the record so insert one group update one group see these this blue color mentioned right these record only should go for insert or update by keeping this flag why we are not taking these columns these columns are not needed only these two flags is enough this flag will route the record from the source whether you should go for insert or update okay we are taking it how many groups should be created here two groups go to group the router create two group one is insert group update group see one group you are creating by default another group will be created first you can go for insert plug okay insert group okay insert group this is for update group you can go for update group okay either should go for insert or update otherwise it will be blocked here the router set will be blocked so which one will be true here you have to make insert flag that should be a true here see whichever the record is flagged with the true here okay it will be flowing to this particular group if it is all the record is false it will be blocked here itself the insert flag itself okay whichever the record is coming like update flag as true it will grow it will go to this particular group clear two groups will be created one group is inside flag other one is update group see insert flag you can directly go and insert in target insert flag you can go and insert directly but update flag update how to use the update strategy in order to update the record you have to use update setters this is what the update strategy this is what the update strategy you can take the update strategy here you can drag and drop all the columns from the update group from the update group then in the update group you can write the logic what is the logic either you can go for 0 sorry 1 or you can go for td update dd update can i write expression here yes you can write it through expression see for an example if the flag equal to something 1 you can go for a dd insert something like that you can write like this r you can go for dd insert so you can write some expression also here in the update strategy but i am not using any expression just i will be removing it okay we are we are validating only dd update dd update r1 i can use so i can take their target table see here i have taken the target table how can i take one more target table should i take one more time yes r you can copy this paste here ctrl c ctrl v ctrl c this target ctrl v here the name will be different if you look at here the name of the target instance will be different here t underscore customer underscore update here t underscore customer underscore update one but if you double click you can see the target table is tn for customer underscore update here the target table is t underscore customer underscore update both the tables are equal only okay so now you can join all the columns here it will be connected you can see it&#39;s a valid one okay it is valid one this is what you&#39;ll go for the update strategy update strategy so now you can arrange it arrange all to be arranged see one more time i will repeat first of all we are taking the source the source should be either inserted or updated how can i find by using this customer id by passing this to a lookup customer id customer addition equal and i&#39;m making that whether the customer id is present or not the customer id is not present in lookup then i will make this flag is true if any one of the column is not equal i will make this flag is true then based on these two flag i will segregate these two groups one is for insert one is for update insert directly i will insert it update through update strategy dd update i will use i will go and update the record for update strategy you need to have one primary key column remember this okay please make a note of it everything any doubt here this is very very important so whenever we are going for next scd types right so a cd type one this is what we will use okay now i will go to the workflow manager to update strategy wf underscore update underscore strategy you take the session and go for update strategy double click on the column go for mapping so should be pointing to source target should be pointing to target but remember you should not truncate target table you should not rank a target table look up look up which one i have to connect target here i can go for the target okay target here so apply okay save this it should be valid now okay it should be valid now okay so now i&#39;m going to run it as of now you will not have any record in the target so now we do not have any record so i&#39;m going to run it all the record should get inserted now all the record get inserted see regard 4 records should go for insert why here zero there is no data for update right that&#39;s what four record from source four record got inserted and zero record for update if you go and check here all the record will get inserted see here all the record has been inserted here in the target okay clear on this okay if i&#39;m going to again i&#39;m going to run the same workflow what should happen what is the statistics here four zero zero why there is no data for insert and update that&#39;s why zero zero so why because source and target is equal that&#39;s what it will not go for any insert update okay now i am going for something like some update i will go for i will source record i&#39;ll go and update say for example this particular record i will go and update this email so manually i&#39;m updating it you can update see who will update in real time the date the customer if the front end tool they are going to update it here it will get updated automatically okay gmail.com so the mobile number i&#39;m writing here okay the mobile number so i&#39;m writing the mobile number here okay i have updated the record click on concatenate sorry commit will be committed so make sure that you are committing the right times after updating it if you don&#39;t commit it it will not be updated so if i&#39;m going to run what could be the statistics here if i&#39;m going to run again what could be the statistics 4 0 1 okay 0 means nothing is nothing is inserted one means one record is getting per update four zero one if you go and check here okay so this is what got updated in the target okay it got updated in the target [Music] welcome on second in our today&#39;s session we are going to learn about java transformation this java transformation is used to implement the java code in our informatica project for some of the real-time applications right real-time scenarios where we will use this java for easy processing the data okay like encryption decryption encoding the data decoding the data like we need to generate some numbers okay transaction even number some numbers some specified to a particular number for this particular identify the each transaction in order to hide that numbers how the numbers are generated so we will go for something like a java transformation in order to create some kind of business numbers some encryption decryption all these numbers right you will go for this java transformation so either java will be used as active or password so what is this active right so whenever you are going to take the java transformation you can say it&#39;s a active transmission or passive so you know active what is active if the java transformation is going to have multiple records in target that will be your active okay so single record multiple records that are the number of record is getting changed the target in the output then it will become active right so passive so we are going to give some passive uh transformation logic then we can take that java is passive so this is what the java so in order to use the java transformation we should know the java concept very well okay we should create at least we should take the java code from the online resources and we are to make use of that java code here just we will take one table so if you look at here i will take this particular i will connect to this repository i will take this particular table so i have that a data like this okay i will take this particular table the source select start from s underscore customer table customer underscore java so i kept for naming convention for to understand easily for java so you look at your customer id customer name customer data birth i have like this and mobile numbers so i have mobile number like this we have for single customer we have different mobile numbers okay so the use case is here i have to split this mobile number into three mobile numbers for this particular customer so one record in source three record in target one record in source four record in target so based on the mobile number i have to split so this is some use case so this can be achieved in expression transformation also okay you can take the expression variable port you can encounter this this particular tilt symbol and then you can go for some looping and then you can get it but it&#39;s very difficult right looping concept in expressions lot of variable port it will take lot of memory also it will take a lot of time but in java we can implement easily so even i have given an expression i have given one logic for a looping concept right i used to call a new sticker the same scenario you can use this java java transformation so this particular data i&#39;m going to load in this bit so this way i&#39;m going to load it so how to load so for single source so like this i&#39;m going to split up based on the mobile number how many mobile numbers are there three mobile numbers i&#39;m going to make it like a three four mobile numbers four records so this is the use case so how to do it so we will do it and then i will truncate the target table so we will load by using the java transformation so if you look at here select start from now the table doesn&#39;t have any data so i will take these two tables as source and target go to source take the import from database hr then click on connect and take the hr the source table i will take this table so target also we have already i have imported this particular table i will go to this mapping i&#39;ll go to this create mapping m underscore java underscore transformation i will create this java transformation okay i will take the source table first take the source table first customer table target table here i can keep this is my java so click on java here so it will ask okay please choose the java transformation type either passive or active so once you have chosen after that you cannot change so for this scenario we have the active so i pick a single record in source multiple recording target so we choose our active so this is an active transformation now so normally what will you do you will drag and drop the port to here right so instead of that double click on this normally you can drag and drop also no issues see this is what the input this is what your output so you can keep the cursor here you can create so how many portions you have four port or you can drag and drop that is also fine so keep the cursor here on the output so create four four targets so what is this three targets are four targets we have to create here i&#39;ll go to the customer id first one is customer id second column is customer name second column is customer name third column is date of birth so you can take data the fourth column is mobile numbers in source we have mobile numbers mobile numbers then what is the data type customer id i can have a as decimal customer name i can have it like a string under characters data birth date and time mobile numbers second string so source via the string you can have it like a string and customer id here so we can have this one is output right so i can have o underscore customer underscore id this is a decimal only you can keep this as decimal then this is like oh underscore customer name customer underscore name who underscore data birth underscore mobile so mobile only not mobile numbers just i will keep us mobile to understand okay this is a single number so this hundred characters data but this date and time i can have the mobile number also string okay 100 characters after that we will convert string into numbers so this is what we will create this particular columns and go to the java code so this is the java code actually you can remove this code these are all comments so based on the column right you will have the full code if you look at here based on the columns you have the full code so all the code will be created like this okay so all the columns will be have created like this we will create a code here so we have to create a code here so that code we should know something like i will give you one exercise for you like encryption decryption you can try that okay here i am going to create so first of all i am going to have a variable called string string variable i will take a strs variable i will store whatever the mobile numbers i&#39;m going to give see whatever the mobile number i&#39;m going to give right so that mobile number i will store it here mobile underscore numbers it&#39;s a string okay that will be stored in str the variable okay it has some tilt symbol right so to take the tilt symbol i will take one temporary variable in the temporary variable i will store this value string null value and i will take that temporary variable called it m so this is temp then i will take a delimiter string to store this delimiter so what is the delimiter i have to store this is the delimiter right so this is the delivery so i have taken all the variables you can take some java code online also so whatever the code you want you can take it say for example i want to store the temp equal to these are all methods str dot string dot split i&#39;m going to split based on the delimiter based on the speed delimiter i&#39;m just splitting that string and storing the value into the temp variable but i have assigned the same variable as null i have split it based on the delimiter you can assign it to delimiter and then you can split or you can directly give the delimiter here then i&#39;m going for the looping so here i will go for for loop for i equal to 0 less than m dot length eng th length and then i plus plus so up to this number i will go and then i will press plus okay then you can open those open curly braces close curly braces okay so you can open and close java we do not have any indentation but in python we have some indentation okay that we need to follow so i will map the columns o underscore customer id equal to customer id from source customer id so that we will use o underscore customer name equal to cust name o underscore dob equal to dob these are all the columns direct uh load so underscore mobile that is a target right or underscore mobile equal to what is that mobile we are splitting the temp variable we are splitted right so temp i i can store so semicolon you have to make this as a loop for this you have to use one method called generate row so generate row is nothing but for each and every record it will create a rose will be generated okay so all the semicolon and everything should be fine so once you are done with this you have to click on compile so compile it it should be compiled okay we are getting some error for i so what is this i whether it is integer or variable cannot find symbol right either you can define here integer i or you can initialize here itself teacher right okay so whenever you&#39;re going to compile so here itself it will compile and then it will say like the compilation should be successful then only it will be valid so if you want to see the full code you can go to the full code the java transformation like this it will import all the packages okay here what are needed so public class so within this what is the variables private public all the methods it will be taken classes and it will create for each and every record they create like this this is the entire code so entire code will be created by informatica itself we are not going to create so exception model also apply okay then take all the four columns into this one target also you can take from here control yes it should be a valid one okay you can go to workflow manager so just i&#39;m creating the workflow to run the java so workflow java underscore transformation java transformation one i will use i will take this particular session java transformation okay so link it so java transformation this is the one right we have created double click on this same thing we have to give the connections if you have any jar file in the project if you have any jar file the java executable file then we have to mention this the class name okay so here we have to mention the jar file so it is taking the default file no issues and apply but in real time project we have to give the java hr file here okay it should be a valid one now i&#39;m going to run it i&#39;m going to start workflow it will load the data into this particular table as of now no data is available it&#39;s getting loaded succeeded two records in source seven records in target so three records for first record four records for uh second record so seven records has been loaded into the target so for this logic just i have created like this say i want to do some encryption decryption for our customer table or any tables so for employee tables i have to create some encryption id for product table or any tables accounts table account number the card number i have to make the card number as encrypted key so make this particular this scenario some exercise say encryption decryption by using java this something like you&#39;ll have you&#39;ll have a encryption decryption logic okay some java code will be there so this import and all informatica will take care you can have some encryption decryption some different logics will be there you can take any one logic okay so this is some method so no need to go for this this method you can go for some other method some other method lot of methods will be there okay top methods will be there you can go for any one method uh you can use it this method okay you can use any any one logic whatever the logic you need okay just to try this if you have any questions you can ask me i will help you out i will give you some code example code for encryption so that you can use that code you can give some account table you can go for the encryption decryption so if somebody else asking that question right you can tell them like for encryption decryption or encoding decoding some something like a conversion or this whatever we have seen for the looping and everything so here we can input take input all the input as one single string then each and every string will have some new line characters right so if new line character says encounter you can put comma here that&#39;s all that&#39;s very simple right so you can use this java transformation to implement this particular logic also either java or expression hello friends welcome to nik iit academy in this session we are going to learn about parameter and variables i have splitted this session into two different parts first part we are going to learn about parameters second part we are going to learn about variables so this is the agenda for today&#39;s session so what is parameter what is variable what is the difference between parameter and variables how to pass a value to a informatica workflow or session through a parameter file how to pass a different value in the next run how to configure the parameter file in the runtime so that we are going to learn with example whatever the concept i have explained in our session i have given in the document it is available in the descriptions you can download the document for more details if you haven&#39;t subscribed my channel so please subscribe the channel you&#39;ll be getting all the updates whenever i&#39;m uploading the new videos let us start our session so parameter and variable is nothing but if you want to pass any value dynamically to the mapping or dynamically to the session we can give us parameter or variable okay so what is parameter and what is variable so everything will be maintained like a dollar symbol if two dollar is there then this is like a mapping level parameter okay this is mapping level uh parameter single dollar is there this is a session level parameter okay uh session level parameter so you you have seen something like dollar pm src file directory right download directory it&#39;s nothing but file directory is nothing but it&#39;s a source file directory you can see here everything we have here we should have if you go here the source is a file then you can see something at dollar pm source file directory this is a single single param single value is there right nothing but dollar pm source file directory equal to the c drive informatica that path will have why they are giving like this it&#39;s a parameterized value instead of particular value so if you are giving a hard coded value what will happen if you are migrating your workflow into other environment then that environment will will not be will not have this particular path okay and also some path some some particular path if you are if you are changing over the period of time you have to change it all over the workflows so instead of that if you are giving us parameters then definitely can able to give so whenever it&#39;s required all right so this is the lpm level it&#39;s a session level properties and so what is the difference between parameter and variables so for example you have the parameter the parameter is nothing but you&#39;ll be having dollar dollar okay dollar is the mapping level parameter right so parameter it will not change during the runtime okay it will not change value during runtime and the session level parameters okay the session level parameters are the parameter if i&#39;m taking for an example dollar dollar on value i&#39;m taking okay so something like i&#39;m taking some value any any values i can take for an example i&#39;m taking dollar dollar salary equal to 1000 or 10 000 or anything so during the run time it will not change okay during the runtime it will not change so this is like parameter so once it has taken the value throughout the decision that value will not change but what is variable during the runtime it will change okay so variable value will change the value will change during the session right may or may not change okay so may change during the session runtime so that means if you are giving some dollar dollar count that value will be changed during the runtime also so this is called uh parameters and variables okay so like if you want to give like something like parameter and variable you can go with so dynamically if you want to pass a value to the mapping then you can go for either the parameter or variable variable is value will change during the runtime the parameter will not change so you can see the variable port also right you can see here i&#39;m going with something like parameter so here i&#39;m going to create a mapping called m underscore parameters okay so m underscore parameter there are two types of parameter values are there one is global parameter global and one is local parameter okay so what two types of parameters what is global parameter global means any session any workflow can read that okay that is a global parameters so any session or any parameter can read that value but the local one is nothing but it&#39;s uh particular to the local to the that particular session you have to mention that folder name session name and everything then that value should be applicable for that session only it is not applicable for other session okay but you are defining a parameter or variable as global then others can use it so that is what the global variable so it can go with something like so this is the local variable and you can see here so look at here i&#39;m going to take one uh so what it will do will automatically take select all the records from the source but i want to dynamically select the values of the source then i can go for something like parameters uh you can first you have to define that what parameter you want you can go here the properties i&#39;m just making that something like even the scale query also you can give it like a parameter but just i&#39;m giving var class only in the parameter just is generating the test here okay generate it and give this where class as dollar dollar something like dollar dollar where i&#39;m just giving us this class is the dollar dollar if i&#39;m not giving any value for this dollar data where the null value will be passed here so click on ok what is downloaded first of all we have to declare it go to mapping to parameter variables add a new parameter or variable it&#39;ll ask you want to have download under where that that one we have used right so just i&#39;m making the data whether it is a variable or parameter so i can take it as a parameter so i do not want to change the value during the runtime mostly we will create a parameter only here so string dollar dollar bar will be like i can give some hundred characters a thousand characters so it&#39;s expression you can make like false okay you can give okay so you can make control yes it should be valid now then go to the workflow so you can create one workflow new workflow so wf underscore parameter okay parameters so i can give session so m underscore parameter click on ok then click on link task double click on the task go to mapping so should be pointing to source target should be pointing to target you can go little bit down you can see trunky target table option right and provide it and then you can apply click ok but we have provided a parameter right you can go to the c drive where we can keep the parameter file we can keep c drive informatica 10.2.0 server and you can go to the param file this is the folder and go inside you can create one file here param so any file name so that file name we have to give here so this is the path right you can see here this is the path in this path we have the param.txt right if you are using any parameter you have to give the properties the properties you can give parameter value right it has one option called the parameter file there you have to mention the parameter file name so click on parameter file name here just to click on ok so what is the file name param dot pht so that file name we have to give here so click on apply okay that&#39;s all so dollar dollar where we have to define here go to parameter file double click on it so you can mention like it&#39;s a global parameter if it is a global then you have to mention like this and dollar dollar value it&#39;s a case sensitive so dollar dollar you have to give like this and you are giving where salary greater than fifteen thousand just making that class then apply ok so what informatica will do informatica while starting it right so if any parameter has been used it will check the parameter file it will go and read the parameter value since we have given us parameters dollar dollar where it will read the data now look at here so this is running informatica is running today i&#39;m giving like dollar dollar where equal to where salary greater than something but tomorrow i may give some different one it got succeeded now you can see it should have taken three records from it if you go to the file the session log you can see so dollar dollar value will be presented here from employees where salary greater than fifteen thousand sorry the parameter value has been passed okay so now you can see go to this workflow parameter file i&#39;m not changing anything in the session level or informatica level just i&#39;m changing the parameter value okay so for example where job underscore id equal to sa underscore so your rf is nothing but sales representative i&#39;m just giving it job id equal to this one so i&#39;m just making like this is the parameter value i haven&#39;t changed anything in the informatica level just i have changed that parameter file here start off below you can see now the characteristics changes right now it is reading 30. in most of the cases in real time the connection value also parameterized the connection value also parameterized so this is like where java id equal to this one okay it&#39;s passing the runtime it is passing the parameter value okay so this is something like parameter global parameter you have to mention like global if you want to give local parameter you have to give this formula for example so global is nothing but all integration service integration service process and able to read that global parameter so you can give this uh this is the one normally we will follow okay something like folder name okay folder name this is entire workflow level folder name dot wf colon or floating are if we will give in this format folder name dot wf underscore name worklet name or session name or you will give this this this formula also the session names if you are giving like this this is the named session right for that particular session you have to give a value so we have to give for an example you want to give for this folder for this folder this session alone you have to something you have to give like this stuff global you have to give folder name will be so this is a folder name right training underscore file so this is a folder name that workflow name so this is our cloning this is session name for this session we are giving right so we have to give session name this session name here like this we have to give and so global like i will remove it now so training underscore five dot this workflow dot session name equal to this one for this we are passing so now you can check whether it is able to read the parameter file okay so 30 records is getting right so this is the this is the way you have to cue for the parameters for some other parameter value you have to give some other name you can give some other name also any path you can mention but you have to tell that information that okay this is the path this is a parameter file but the parameter name the file names vary correctly okay so if you are giving some different name it will get failed even some real time the parameter file also we will create runtime connection value we will create runtime so that is also possible here we are giving like this right but we will create the parameter file runtime also that script will create the file subscript will be there so that script based on the value it will create the file okay use some script to run this particular workflow so look at this is this is one of the wrapper script something like bash script right that&#39;s what you have and this is comment commented out pm home is nothing but infra path so info path and info server info server name you have to give either w uid or prod informatica domain the domain name you have to provide workflow name the you have to give the name formatting a user so what is the user you have to give password you have to give the password or what flow flow name have to give what is the workflow name here so start then dollar pm home that is nothing but so this folder right we you remember that informatica folder so informatica 10.2.0 server infrastruct so here you&#39;ll be having that this is the dollar home path so it will be mentioned in this admin level then you are passing this one pmcmd then start workflow these are all something like variable parameter passing you have to give all this informatica user particular password domain and everything folder wait time or pro name everything you have to give so it will fetch the workflow state uh status so it will take the status whether it is running or succeeded or failed it will take and then if the workflow status equal to zero then it should show you like execution successful alif we are making that execution has failed something like other than 0 we are getting some status like 1 or 0 or something like that 1 or some other value then you may assume that ok this workflow got failed right so this is what you can take this is one of the pmcmd command wrapper script to start the workflow is not to create a parameter file so this is the way we will create normally the parameters and the variables [Music] so debugger is nothing but step by step execution of the flow if i want to know the step by step execution of the flow then i can debug it session say for an example you are a developer you are developing the code and then you are you are completing the mapping and then you want to test it so you are running your session and then reading the data and then loading the data into target table but the session got succeeded but you are not getting expected result then what will you do so normally if the session got failed the session is failed you assume that session is failed if the session is failed you can check the session log so normally you will be checking the session log why it got failed and you&#39;ll be troubleshooting it okay so white card failed and then you will be troubleshooting it if the session got succeeded the session is succeeded and you are you are not getting expected output okay i&#39;m not getting expected output say for example your expected output should be so you&#39;re expecting what a particular column expected output for a particular scenario like some value you are expecting 1000 you assume that one value okay but actual output you are getting is none say actual sorry output you are getting like a null value for this particular column it&#39;s not a error it&#39;s a logical so somewhere you are missing your logic then you have to find out right where you are missing the logic and everything so that you have to go for a debugging so you have to debug it so debugger is nothing but step by step execution of the flow okay step by step execution of the flow of the transformation flow step by step execution of transformation flow so pipeline flow you can say if you look at here so i can debug any session say for an example i want to debug this a cd a cd type too i want to debug it yes i can go for debugging say for an example if you are not getting correct output you can go here go to mapping debugger you can use start debugger so start debugger you can go for next so you you&#39;ll be choosing which session i want to go for like you use an existing session instance use an existing reusable session create a debug session so i want to go for use an existing session instance so that means it is a non-reasonable session this is reusable session so you know already what is non reusable reusable session right you can go to next and then it will take that particular session so this is the session right you can go for next discard target data while running the debugger it will not physically write the data into target table okay while we are running the debugger it will show you what data is populated in that debugger but it will not basically go on right into the target table this database table it will not write okay this is what the debugger okay i can click on finish it will open two window so one is current instance this is what your current instance current instance means so this is one instance this is another instance this is another instance instance by instance instance means transformation once your debugger start running all these are disabled transformations everything is disabled state okay so now i will go with the mapping go to debugger so you can go for next instance that means next instance means next transformation it will move so first time using the next instance you can see here f10 right so next time you can press f10 so you look at here where is the session now session is there in source qualifier so here this is the session source qualifier okay if you look at your row number row id 1 that means first record so first record from source it is taking so you can see your customer id first name mobile number all this you are saying here and then you can this instance is in source qualifier so keep the cursor here you can press f10 here then it will take second first record to a lookup so look up also you are getting lookup key and then this is the customer id one from look up right so we are getting the matching then if it is matching an expression we are getting that new record flag is zero update record flag accounts zero so effectively today&#39;s date end date is 31st so go for next one so it is getting filtered out okay it is getting filter out the row type if you check filtered out then this record is not going for the target okay if you are seeing anything like uh filtered out it the record is not going for the target okay how many targets are there two target right you can check two targets or three target what are the target here see for all the target it will go like a filtered one only so you can check here okay next instance the same way you can go for next next instance so how the data is flowing you can check one by one see here one by one all the record is getting filtered out it&#39;s not moving on to the target so this is what you have to debug it so once everything completed automatically it will come out of the debugging debugger start a shutdown complete okay so this is what it will go for okay now i will go to the another mapping here it will i will show you what is the debugger and how it will be so you remember we have seen transaction control transformation right transaction control to generate the dynamic target right target dynamic target we have used transaction control say for an example this scenario i&#39;m going to take how many transformation areas you&#39;re using source qualifier is one starter is two expression is three transaction control is four so go here the same way debugger so i&#39;ll go for the start debugger go to next you can go for use an existing session instance click on next it will go and bring the sessions so which session you want to use i want to use this transaction control this first session so i will take the first session click on ok discard target data you can click on finish then the same way two window will be opened one is target instance other one is current instance so you can keep just why i have taken this scenario i will tell you since we are using sorter here right so you can see here debugger you can go for next instance you can go for next instance if you are taking next instance row id 1 wires the instance suppose qualifier you can see source qualifier first record it has taken so you you can validate one by one record here so you are clicking on next instance f10 so f10 is means next instance it is not moving on to the sorter it is still available on the is still on the source qualifier you can see here second account has taken from the source qualifier press f10 third record fourth record fifth record like this it is going why it is not moving moving forward why because next transformation is shorter since next transformation is shorter it will not move so why because here we have a sorter will take all the record and then by using all the record it will do the sorting so that&#39;s why this is not moving forward okay so here we have so this is what it will move and then you can you can go for f10 okay so if you have you can ask me so how many records are there one not seven record is there so one not seven times i&#39;m pressing that f10 it is moving you can ask me one doubt okay if i have millions of record will you press million times no so that we will use some something called concept called breakpoints i will tell you what is that so this breakpoint concept is applicable in other like a java python also we have same way here one not six right look at here one not six next to one time i&#39;m pressing f10 one not seven now the instance is available in source qualified if i&#39;m going to press one more time f10 it will move on to the next transformation that is whatever sorter should look at your starter so on what basis it is coming like one not nine it is sorting based on the department id not department you can read it country only we are using the keyboard so based on alphabetical order it is making the country so c then i&#39;m going for the f10 f10 means next instance so how it is moving to the expression okay then here transaction control it&#39;s coming here you can see here it is generating the underscore country underscore canada.csb change flag equal to one that means true and then commit before this expression is making like commit before right so you can make one by one f10 again f10 so this is insert look at your row type is insert means this record will go to the target table okay this record will go to the target so next to the card it is taking where from the starter extra also canada then you can make it like f10 so it is moving right so here it is moving so since previous record this record is saved change flag is zero only previously change flag is one now the oh change flag also zero only it will go with continued transaction previously it was commit before right but now it&#39;s a continued transaction so this is what it will proceed okay if i have millions of record should i go for like this one by one no i want to check only for india i want to check only for india how can i do it so go with debugger so in two ways you can do one is you can go further so i will go for stop debugger will stop it two things one is in the source qualifier you can apply some filter the source qualifier you can apply some filter you can make a bad country called india that that means it will pass only india only you can easily track it otherwise you can go for breakpoints if the source is flat file you can go for breakpoints see here go to mapping debugger edit breakpoints it will ask where you want to place the breakpoints i want to place the breakpoint in expression wherever you want you can keep it no issues i want to keep breakpoint in expression just time making expression transformation go and add the breakpoint so look at here add breakpoint click on condition so what is the condition so i want to make country equal to india right so i can make india country equal to india okay here so i have added on breakpoint you can see here so one stop button will be appear here whenever i&#39;m going to start the debugger stop start debugger to next go to next and use this finish it up so this time you will find one stop button look at your stop button break points it is called breakpoints so what you have to do since you have applied the breakpoint right no need to go for one by one go for directly continue so what is continue it will continue all the record it will break only that india in the expression transformation look at here india expression transformation is it making so record is india right country called india so other countries it will pass so china and england it has been passed canada china england has been passed but now india is coming it has stopped now right so now you can go for f10 so one by one you can move so you can check only for india how the india is moving this is called breakpoints so if if somebody asked when will you use debugger so if i&#39;m not getting expected output in the target if the session got succeeded then i will use the debugger in debugger we have two window one is current instance other one is target instance okay target instance and we are the breakpoints breakpoint is nothing but if i have millions of record i want to test it for a particular record i can use breakpoints i can test that particular record alone okay so this is what you can inform them if the session got failed what will you do yes check the session log okay how will you identify various errors if the while reading itself is failing then you have to check so why this has failed right that&#39;s what you have to backtrack these are all the troubleshooting method continue means other record will continue it will stop in the whatever breakpoints you have applied okay from there you can go for next instance after see for an example one million record is there you want to particularly this particular recorder you have to check it continue all the record you break it only particular record so that&#39;s what breakpoint will be used [Music] so welcome to the session in our today&#39;s session we are going to learn about performance tuning see whenever in the real-time project right whenever we are going for any real-time project so definitely we will be handling huge volume of data so one thing is huge volume the other one is the number of columns will be there okay more number of columns huge volume and complex business logic so in informatica just here we are using maximum 5 to 10 transformations but in real time we&#39;ll be having lot of transformation logic that time it may take some time okay to implement logic for all the records it will take some time to avoid this kind of scenarios we have to do the performance tuning so if we are using normally we are using these transformations in our informatica so definitely we will face performance issue these transformations like if you use aggregator yeah definitely we will face a lot of performance issue so why because aggregator will consider all the record to aggregate the values okay it will take all the records to aggregate the values and sorter yeah this is also it will consider all the records and rank transformation so all that all these transformation right so every record it will compare and it will give you the results so aggregation lot of cache memory will be used sorter rank joiner lookup so everything the catch memory will be used so that&#39;s what if you are using these type of transformations so definitely you&#39;ll be facing performance issue okay so how do you know this this session is running like a performance issue so first of all the session is long running okay the session is running so without so one thing is huge volume you can check the throughput if the throughput is very high and also at the huge volume it is taking a lot of time no issues so let it be running okay but the volume is very less but it&#39;s taking a lot of time to complete and on its long running sessions or we might be going for the timeout issue so timeout issue is nothing but so whenever we are loading the data fetching the data from source and target we are loading so that time you maybe we might be getting the timeout issue that&#39;s what timeout issue our session failed session might be failing due to the timeout issue and cpu consumption is more if you are getting these kind of issues so definitely it&#39;s a performance issue either as a developer we have to find out our informatica admin team they will send you the mail saying that okay these sessions are getting lot of time are taking a lot of cpu so we are converting this particular session and then please take care please implement some performance shooting method okay they will inform us another way so if real-time project if you have many jobs are running okay thousands of workflows sessions are running if your if your customer is saying that okay if your manager is saying that so take any top five sessions top ten sessions and implement the performance shooting method something like a process improvement okay process improvement we so ourselves proactively we have to find with sessions that is taking a lot of time and so what is the reason it is taking a lot of time and we are to find out okay this is what it is taking a lot of time and we have to fix the issue how do i know that these out of these manipulations these sessions are taking a lot of time okay we do have metadata repositories okay so admin repositories in our system we have created right repository in admin schema or in fast schema the same way in real time the informative admin team you can get the repository queries from them so a lot of repository tables are there okay metadata tables are there informatica so we can go to the repository uh tables say for an example whichever the table we are getting like opb so this is the table starting with opp is nothing but it&#39;s a repository tables so opv subject is nothing but so these are all the metadata tables if you look at here select star from this table it will give subject is nothing but folder name see whatever the folder which is available in our repository all the folders it will be displaying here okay you can create so this is the back end query we can execute it and then we can find how many folders are there so what are the different folders and when it has been created all this you can able to see here okay like this we used to have lot of tables we used to have the opb subject task instance run and the workflow so you can take this for plural if you look at here when the workflow has been executed and that you can see here select start from this one if you look at here so this is what you are saying all the subject id is nothing but the folder now folder id then workflow id so by using that workflow id you can find the workflow name and each and every workflow we have a separate employee if you look at here workflow ids what is the server id and when the start time the end time of this workflow and where the workflow log has been created okay then you can find where the workflow log has been created and you can find different informations in this table same way we used to have lot of tables here in our informatica repository i have given one repository query this query will give you the top long running sessions okay so each and every session it will give you the session run time we can find you can take top 10 or top 20 other sessions so if you look at here subject name so by combining multiple tables here so you can take this but this queries in in the unit also we can take it no issues like from these table we have joined and then see whatever the columns needed suggest we have taken that so subject name is a folder name workflow name instance name load date session start time session end time then you can find the duration of this session okay so this is what each and every sessions duration we can calculate if your if your lead saying that okay go and find top long running sessions top 10 long running sessions and try to implement performance tuning method for the test session okay if they are giving you this particular task then you need to talk with your informatica admin team whether a posting has been created or give them this particular query they will execute or you can give the same query to a dba they will execute it and then they will give you the result set so from the result set you can find whichever the session is taking longer time you can take five sessions or ten sessions you can go to the subject and go to the folder okay this is the folder workflow and instance and you can get the session log for all the sessions first of all if i want to implement any performance tuning definitely i have to analyze the session log okay so definitely i need to analyze the session log in order to analyze the session log i need to know which session i have to analyze right so this is what you have to get the details okay by using this below repository query we can get the folder name workflow name session name number of minutes the session has taken to complete so everything we can able to find okay after getting that we have to get the session log either through monitor or we have to get the session lock from unix server okay so once you have taken the session log in the session log you can find session load summary so if the if the particular session is taking longer time to complete so definitely you&#39;ll be having the three threads this load summary you&#39;ll be having the three threads one is called a thread transformation thread and write a thread so read a thread there will be idle time and busy percentage so what is this idle time means idle time means okay it is waiting for the data so it&#39;s being idle and that busy means so it is using all the resource 90 percentage of the resources to read the data so the reader we are getting more than 50 that means a lot of resources it is using bottleneck we&#39;ll say it&#39;s a bottleneck we might be getting timeout based commit point in that target side in our right side why because here the source side it is completely busy occupied then target will be idle idle so it will be idle it is idle for 90 percentage here we see for 90 right here idle for 90 only 10 percentage right so this is what will be having the bc percentage if any if you look at here this is one sample session log in any of the session log if you are finding bc percentage is 100 that means there is a bottleneck where the bottleneck here the reader thread there is a bottleneck while reading we are finding lot of bottleneck issues if you look at here if the bc percentage is 100 then idle time will be zero that means it is not ideal completely it&#39;s busy okay this is a writer threat lighter thread there is no busy percentage zero only okay this is our total id of time why because so here completely occupied right so that&#39;s what we are finding the ideal time okay so this is what you will be finding the in session log you will be finding the bottleneck either source side or transformation side or target side this is called a bottleneck so mostly in our real time project where we will be finding the bottleneck right on the source side okay while reading the data we might be getting the bottleneck the second highest bottleneck in the target set okay it depends on the scenario which depends on the tables so which table you are loading and from which table we are fetching and databases also and server capability also it depends on these factors and the cache memory size and everything its depends first of all if they are asking an interview how do you find performance issues so i will be checking the session log where the bottleneck and finding that okay if the busy percentage is more than 70 percentage there is a bottleneck in the either in the source side or in the target side or in the transformation set okay if it is a source side it for the source side bottleneck so we have to analyze the source query source db source table everything we have to analyze it optimize the source code query so you can use some hints if it is oracle table try to use some hints or in the pre-scale post scale you can write some collect stats before reading the data you can do the collect steps on the source table and optimize the lookup queries if anything and the indexes say the source table doesn&#39;t have indexes the previously create index the post sql we can drop the index and we can go for the partitioning so these are all the techniques we will you will be using in the source side okay we&#39;ll be using in this whole set okay the target segment the target will be using the drop index so if the index is there in the target site and we have to drop the index load the data and after loading the data you have to create the index that we can use it in the pre sql post physical the next one is called partitioning so partitioning is nothing but the parallel processing that we will see now so that you will get clear idea so these are all the methods you&#39;ll be using that to improve the performance okay in the transformation side if it is aggregator then you can use sorted input or incremental aggregation so that concept incremental aggregation i will tell you but you know already what is one way started input right so before rank transformation you can place sorter before joiner you can place the sorter before lookup you can place the sorter that&#39;s what you can improve the performance of the these transformations creating the look of cache and everything okay so you are using these methods still you are not able to implement performance tuning okay you are not able to increase the session runtime okay so how to optimize them we do have two different methods to implement the performance tuning one method is called push down artificial the other method is called partitioning so definitely they&#39;ll be asking an interview so what are the two different methods are there to implement the partitions so one method is called push down optimization the other method is called partitioning what is push down optimization the push down optimization or partitioning these two methods is licensed properties okay so separate license you need to get it for these in order to use these two methods we have to have a separate license the license property so we will call it as push down automation is nothing but pdo so if you consider the push-down optimization is nothing but converting the transformation logic into sql query and pushing down to the source or to the target either to the source or to the target or you can push it to both so that&#39;s what the push down optimization if you look at here so you can take this is the left hand side is source left hand side is source right hand side is the target so here we are the target cycle this is what the source side was stable so this is the transformations we do have transformations here we have the transformations here we have the transformations a lot of transformations are there and here we have the transformations okay here we are the transformations and here this is the target so left hand side you have that source on the right hand side we have the target okay so in between we have the transformation logic see all this will be taken care by informatica so the data is coming to the transformation source qualifier after that the data will be moving one transformation to another transformation right see from one transformation to another transformation the data is moving like this and informatica will take the data from source that is source qualifier it is taking right then cache memory the based on the logic it will transfer the data and then it will load the data into target so this target it will be loading okay this is source side this is target set okay this transformation logic where it will be performed in informatica level okay so instead of doing this transformation into informatica level we can convert the transformation logic into sql queries so if it is expression our filter our aggregator rank or anything you convert that into sql query we are not going to convert informatica will convert it okay convert this logic into sql query and pushing down to the source side or target side so if you look at here so it traverse from source traverse from source and it will push it will try to push try to convert this transformation logic into a scale query so as much as possible it will traverse it will convert informatica will convert it this transformation logic into sql query under pushing down to the source side as the select statement as the select statement this is what push down optimization on the source section okay the same way from the target site it will traverse okay the transformation logic tries to convert these transformation logic into sql query under pushing down to the target section so what is the query here insert so it will convert insert or update okay so either insert or update so it will convert this is what it will go for so you can do both both means like it will try to push the logic from this side on this side so remember if i&#39;m implementing push pushdown optimization then the source or target should be a table it should not be a flat file if source side is flat file if it is a flat file right say for an example this is the source side is flat file we assume that this is the flat file so then i cannot implement piston optimization on the source side if it is a target is flat file then i cannot implement push down notification on the target side but still you can use it on the source side so either one of the source of target it should be a table relational database table so this is what we will implement so where we will implement right so definitely be having one options in the session level just you have to enable that so we should we will not have here we will not have here but in real time we will be having it okay so if you look at here source side push on optimization target push now automatization and full push on automation so if it is a source side it will convert into a select statement okay select statement the integration service generate the select statement as per the transformation logic for each and every transformation it can push into the database we are not going to write any sql query it will automatically write target side yes it will convert this sql query like an insert into insert delete update statements this is what it will pushing down to the target segment so normally daily statement you will not use insert or update will use normally you can go for both so where it will be available if you go to the any session right if you double click on the session you can go to the properties in the properties you can find push down optimization okay this is what the bootstrap optimization you can see here you&#39;ll be having option called source to a source to target to full or you can pass this value as parameter so you want to use a source or target side or full it&#39;s based on the logic bottleneck we can we can give the data the runtime okay this is what you&#39;ll be having it once you have selected so this one then if you go to the properties here so you&#39;ll be seeing the mapping pushdown optimization if you click on this you&#39;ll be having view post optimization if you click on this you can able to see the queries we are not going to write any query informatica will write the queries okay so this is what we have to implement the push down optimization methods so just i have given the separate documents already please go through the document you will get clear idea about personal authorization if somebody asked an interview so what is pushdown appreciation have you implemented in your project yes pushdown optimization is nothing but it&#39;s a converting business logic into a sql queries and pushing down to the source or target or both it&#39;s a one of the performance improvement technique i have used in my project whenever i am facing some issues on the source side or target side as a trial and error method i will implement custom optimization i will check that okay whether it is improving the runtime of the particular session if it is increased then i will use this question optimization if not i will go for the another method called partitioning normally in most of the real time project we will go for partitioning whenever we are finding performance issue so we go for partitioning so partitioning also a licensed property here in our in our informatica we do have partition we will do it practically so partitioning is nothing but we can go to the informatica then we can implement some partitioning so partitioning is nothing but it&#39;s a parallel processing this is a parallel processing so we do have normally we have only one parallel thread like we have read a thread read the data from integration service we have the three threads the handles so reader thread will read the data from source transformation thread will transform the data write a thread will write the data into a target okay this is what we&#39;ll be having the three different threads okay then we&#39;ll be having if you are going to increase the number of thread parallel processing parallel processing in the sense we are increasing the number of thread so instead of having one reader we are going to make three readers that means the same work we are splitting the work to multiple persons right so if one person is taking 30 minutes to complete if the same work given to three three members then it will be reduced to 10 minutes right the same way here also instead of one thread we are going for parallel thread how many number of thread here we are creating three threads three parallel threads first thread it will run second trend it will run parallely the third thread will run valid these three will run parallel simultaneously on the cycle based it will run so fast reader thread will read the data and transformation thread write a thread will wait for that it will read then next instance next cycle next instance the transformation thread will transform the data at the same time the reader thread will read the data second data okay this is what we&#39;ll go for the threads number of threads parallel processing sequential okay this is what the parallel processing the same way reader 2 will read the another data theta 3 will read the another data this is what number of thread so first thing is we need to consider three things whenever we are going to implement partitioning you can also explain the interview so whenever i am going to implement the partitioning i will consider three things one is number of partitions number of partitions means number of thread so here for an example i am using three so this is the default normally we will go for three threads parallel threads you can go for three or four or five so or six or eight or ten also normally we used to go up to ten ten different parallel processing okay not more than 10 if you are going for more than 10 you need to get approval from your informatica admin team why you are going for more than 10 10 parallel trade why because it will take a lot of resourcing okay resource informatica resource it will take so that&#39;s what we should not go normally we&#39;ll go for three threads or four or five okay maximum that&#39;s what we&#39;ll go with some some cases some project they will go for even eight or ten also it&#39;s based on the project and capability of the server okay then we are the partition types what types of partition we are going for and where the partition point is created number of transformation if we have more number of transformation where we are creating the partition point that is also very important okay number of partitions here three threads partition type we have different partition types look at here so one is tp level partitions so if the database table has partitioned then informatica we have to tell that okay database level partition already done okay not in informatica you have to do it on the database level you have to do the partitioning on the database level then you can do it on the informatica level and range partition so if you have the data definite range from 1 to 10 10 to 20 20 to 30. so like this you have a definite range of data some product ids or department id or anything you know only these ranges you will get then go for range partition normally you will go for pass through partitions so this is what default partitioning is called pass through partitions even one single partitions we have the pass-through partitions then we&#39;ll go for round robin hash auto key hash user key so round drafting is nothing but based on the threat availability it will use that round round-robin phases question okay then you&#39;ll be having the hash auto key hash user key so this will use hash algorithm whenever you are using this kind of transformations right so you can go for hash user key it&#39;s a hash algorithm so each and every record will generate information will generate hash key and based on that we will be doing the partitioning okay so now we&#39;ll go for so one type of partition we will see that pass through partition if somebody asked so what is the default partition you will be using in your real-time project you can tell them i will go for pass through partitions normally we are doing a lot of parts through partitions in the real time project okay so if you go here i will open this i will show you how the password partitions will be takes place just i will take one table so you can take anything this time taking the employees okay i will take employees data here so here this is what a session can go here mapping you can see the partitions here or you can go to the partitions how many partitions are there we have the one partition so only one partition is there you can go to edit partition point so this is the edit partition point you cannot delete partition point now why because it&#39;s only one partition point is there you cannot delete it see the re-partition point must be specified at this transformation we cannot delete even one partition point okay at least one partition point should be there is there it is the pass through partitions look at here the pass through partitions okay then you can go to the edit partition point so i want to edit the partition point in three partitions like three partitions are four or five so what is this partition type so based on the scenario how many partitions are there we have three partitions we have either you can go for pass through or key range or you can go for database partition here i am going for the pass through if you know the ranges then you can define the ranges also so just click on ok this is nothing but order of the partitions which partitions we have to order then this is what i can go for hundred percentage okay this is it&#39;s nothing but window size okay then you can go to the transformations the three partitions will be there here so in the sql query are in the source filter so we have to segregate the data based on the partition that we have to give so how can i segregate the data so here i&#39;m going to use some some common method if i have a unique key just i&#39;m giving the unique key here employee is the unique key right so based on the unique key i&#39;m going to do the partitioning so employee id just i&#39;m using the mod function mod of employee id since i&#39;m having three partitions i can use three mod of employee comma three if it is equal to zero then i can use this whatever the record it&#39;s coming employee id we are dividing into three if you are getting remainder is zero it will go into this partition and equal to one equal to one it will go to this particular partition equal to 2 it will go to this partition so it&#39;s mod is nothing but a modulus function i&#39;m i&#39;m making that division of employee id see for example employee is 100 if you are dividing by three so what is the remainder it&#39;s a one right so one so that one that hundred employee it will go to this particular this particular partition only two so whichever the record is coming like employee two remainder then it will go to this partition this partition like this data will be moving from one partition to other partitions so this is what we are going with the partitions okay then i&#39;m going with target no need to do any any changes just you can give all the details then apply okay if i&#39;m going to run this you can see this normally this is what the partition will go for pass through partitions if you look at here parallelly it will run three threads okay if you double click on this normally we will be finding that source and target status right okay surprise that why there is no node or apply node or anything you can see the plus symbol here if you click on the plus symbol you can see the partitions number of partitions the 107 record is equally partitioned in multiple layers right so you can see see you you will ask question here the throughput is very less instead of high so still we are having like one minute to uh one second to execute right so previously one knots on our record also one second now also once again then what is the use so if you have huge volume of data then this will increase okay while starting this is what the throughput after that it will get increased by while session running okay it will get increase automatically so that is what it will be running so even lot of sessions we are doing performance improvement method in our real-time project the session is taking a lot of time okay based on the source data target data we will go for the partitions like this okay so this is what one of the partition method so you can go for the other method is called instead of pass through we can go for some range partitions so what is range partitions here so i can go here so instead of pass through i can go for range partitions click on the partitions here edit partitions so instead of this pass through then go for key range i will delete it to go for key range i&#39;ll add the key range here apply see here if you are applying it you will be having the key here edit keys right so you can go for edit keys which key i want to go for employee id or department id or whatever it might be so you can use some keys so what is the key range you&#39;ll be getting it so employee click on ok start range so employee id from 100 starting from 100 right so you can take from 1 also from 1 to 140 okay from 140 to 180 okay from 182 so you can use this 250 so like this i&#39;m just making that we will check it how the data is flowing so our plane okay it should not get duplicated so make sure that it should not get duplicated okay so no need to mention this this logic here just we have made it like here no need to mention so you can remove this so just we have applied already right the ranges we have applied i don&#39;t need to mention anything here okay so this is what partitions you can apply save it okay then run it if you look at here so this time we are running on the range partitions we are giving some definite range but make sure that the future it should not change the range should not change that&#39;s what you have to go for the range partitions in future if the range is changing go for pass through partition that&#39;s the best method okay normally you go for best method you can see here 40 40 and 27 okay record it has been 107 record has been splitted into multiple layers so this is what you can find the range partitions and if you have any other transformations right say for example i&#39;m using sorter starter transformation sorter as i told if you are using any of this transformation definitely you can go for the hash auto key hash user key if you double click on this you can go to the mapping you can see here here you can see i can go to the partitions you can implement the partition point so partition point is nothing but so here we have one and two right we can add partition points here also just we can add it so i want to add three partitions so here i want to go for hash auto key hash auto key look at here hash auto key three partitions now here what is the partitions pass through partitions source side we have the pass through partitions in the transformation side we have hash auto key so based on that then you can go to the transformations here you can implement the source side we can implement the same type of partitions so here i will go for mod function so equal to 0 this one equal to 1 this one equal to 2 from the source this is the partition point from the source it will take based on this one only but after the source the shorter transformation it will change it based on the hash key so this is what you can improve the performance of the sorter okay this is what you can improve the performance of the sorter also so you can see here a data might be getting changed after the source after the source the reading the data right after that it will be changing so here you can see here data is like here is having that 35 36 3600 but after the hash key generation the target it is changing 29 but everything is same only the number of record will be same okay but here it is based on the hash key the number of record passing through the partition will differ because it&#39;s based on the hash key algorithm okay this is what hash auto key hash user key you can go for okay if interview if they are asking have you implemented any performance tuning method you can tell them yes i have first of all what will you do i will analyze the session log i will find the value of the bottleneck and so if there is a bottleneck on the source side or target side heavily identify so there is a busy percentage right so busy percentage i will find and then i will find vary the bottleneck based on that i will go for the the methods if it is performance degradation on the transformation level then i will go for transformation side some implement some performance tuning if it is on the source side or target side so based on the scenario what i have explained you can explain them so index creation all this you can go for or i can go for push down optimization or partitioning push down optimization is nothing but both strong optimization and the partitioning is the license property whenever i&#39;m finding some issues so trial and error method i will go for first customer notification it&#39;s a license one in not it will not be available in all the project okay some project they will not buy this pushdown automation okay some project only will buy it it&#39;s based on the company okay and here the partitioning is nothing but parallel processing parallel processing we do have different partition types we have to consider three things one is number of partitions partition type partition point just how i have explained here pass through partitions range partition high shutter key so that you can explain them so they will get convinced okay this is what you have to explain the partitioning and performance improvement method and push down optimization in the real time project and also in the interview see this is what you have to explain so just we have we have seen so that you have to explain them very clearly this is what the partitioning always they will just ask what is partitioning have you done the partitioning performance tuning what is meta categories so i will give you this document please go through the document go through the video and practice it so completely practice this this is what they will ask in interview whatever i have explained here if you are explaining them then they will not ask any questions they will get convinced [Music] okay in our today&#39;s session we are going to learn about slowly changing dimension we have seen already what is dimension table dimension table is nothing but it will have a primary key and attributes this is what the dimension table it will have a primary key and attributes we do have many dimension table in our real time project so if you look at here we have customer table product table account table user table employee regions locations time dimension store dimension virtual dimension like this we have lot of different different dimension table so these dimension table we will extract the data from source and we will load it into a target table that&#39;s whatever we will maintain into our data values how are we going to maintain that&#39;s what the slowly changing dimension will have okay in order to maintain this table in target we have different types three types we are we are going to learn about here the first one is type one no history will be maintained that means whenever there is a change in the oltp side so just we are going to change in the target side oil ap side also okay simply we will update it we have already seen this in update strategy session so if you haven&#39;t watched please watch the session practice the session then you can practice the next one okay so this is update second type type 2 type 2 is nothing but history will be maintained so this is what definitely we will use in our real time project history will be maintained that to row level history so entire row will be taken for history so we do have different methods so first method is flag method by using the flag we will maintain the history by using the version we will maintain the history then we have something like effective date and end date so date method by using that method we will maintain the list so that we will see one by one what is that then type three so here the entire row will be maintained for history so it is taking lot of space to avoid that we are going for the column level history okay only recent level history will be maintained okay reason history so last changes only will be available the other changes we will overwrite the data so that&#39;s what we will maintain so we have three different types mainly type 1 type two and type three so type one no history will be maintained type two history will be maintained the two row level so we do have three methods you will see one by one all the method and then type three only reason ht will be maintained column level history will be maintained okay the type one we have already seen just we will i will explain okay consider yes underscore customer is a source table customer table it&#39;s a source table dim underscore customer is the target table or olap side we are maintaining this table so we need to whatever the customer is inserted in the source so a new customer should be inserted into the target if the customer is updating any of the attributes any of the different dimensions that we have to update it here so initially how we will do in type one we will take this particular customer id we will take a lookup customer id we will look up the target customer id and if you are getting null value we will go and insert very first we will go and insert all the record so we will insert it very first time it will be inserted okay second time the job is running you assume that there is no insert and there is no update in this whole site if the job is running then nothing will be happen here okay the target so why because here we have source side we have customer id target side we have the same customer id if both the customer ids are same then we are going to check whether the first name is not equal to this first name mobile number not equal to this mobile number address is not equal to this address zip code and country so all the column we are going to check it if any one of the column value has been changed okay any one of the column value has been changed then we will go for update we will have some flag here instead flag update flag so based on the flag we will take say for an example this mobile number there&#39;s an update there&#39;s an update in the mobile number you assume that here there is an update okay there is an update in the mobile number so what will happen so this customer id and this customer id are equal then first name equal to first name but the mobile number not equal to target mobile number lookup mobile number okay this is what lookup okay if it is not equal we are updating this record in the target okay so we just update we just update this record in the target this is what type one we have already seen that and we have already completed the practical session also in this type okay this is called type one normally type 1 will not have a history that&#39;s why we will not use in our real time project mostly type 2 will use so what is type 2 so if you look at here if you look at the type 2 so we are the source table we are the target table so i am going to check whether this particular record is there in target or not if it is not there i am going to insert the record okay so very first time all the record will get inserted okay all the record will get inserted after that if there is a change if there is a there is a change in the there is a change in the source side mobile number has been changed you assume that this change should be taken as a history the target okay so if one customer is changing any one of the attributes either the first name or mobile number address zip code or country any one of that would be switching getting changed so we will take this particular card okay we will take this particular record if there is an update in order to maintain the history we will take the entire record okay again we will go and insert it here okay we will go and insert it here even though if it is an update here on the source side we will insert it in the target okay so here two records are present for this particular customer right we may get confused which one is the latest one so whenever there is an insert so we will maintain one column called the flag in the flag we will maintain flag equal to one okay we will maintain whenever there is an insert we will make plug equal to one okay so the existing flag right and this flag we have to maintain like as a zero we have to update like a zero so this column we cannot make it like a primary key why because there is a duplicate and i cannot update with this column i cannot update this one so we have two records right so we will maintain in order to maintain the history now to maintain it in order to go for update we will maintain one more key column called surrogate key this is called surrogate key okay this column is called surrogate key su r1 g80 is nothing but in addition to the natural key column that&#39;s a primary key column in order to maintain the history we will maintain one more key column that is called a surrogate key this key column is nothing but it&#39;s a unique column by using sequence generator we will create a surrogate key okay so here we will maintain the key column key values okay like this we will maintain the key values okay this one this record we will maintain as update 0 we will update it like 0 we are updating the flag as 0 here okay so new record we will insert it with the flag equal to 1 okay with the flag equal to 1 for update record we will insert flag equal to one okay we will insert it flag equal to one then update the existing record the flag equal to zero okay again the same record is going to change you assume that the zip code is going to change if there is a change in the record so how which record i have to compare either this record or this record there is a confusion right so always we have to compare with flag equal to one so this record we have to compare okay so that&#39;s why we will take look always look up we will take active record always we will take lookup is a active record look at here if there is a new record in the source new record in the source we will insert the record with the flag equal to one okay if there is an update the source we will insert the record with the flag equal to one and update the existing record with flag equal to zero and always look up the active record look up it&#39;s nothing but flag equal to one so this is what a cd type to flag method we will create a mapping for this then you can able to understand then type to version so how we will maintain in our uh word document all the version right in the project under we will maintain version version equal to one then if there is a change so one plus one you will make two right so the again you are going to change it then three version equal to three that&#39;s what we will maintain right the same way if there is a change very first time if the record is coming up then you will be maintaining version equal to one okay version equal to one if there is a change in the existing record we will maintain one more version one more record here version equal to version plus one existing version plus one if the same record is going to change if the same record is going to change again the previous version two plus one it&#39;s a three three plus one four so like this we will maintain the existing version plus one okay so this is what we will maintain the version so if new record we will maintain version equal to one if there is an update in the record we will maintain version equal to version plus one existing version plus one so that means we will we will never ever update any record so there is no need of update strategy in version method there is no need of update strategy simply insert only but we will check what is the existing version then that person will maintain so for an example this nega is going again the record is going to change then already version is four versus maximum version we will take the lookup we will take maximum version again against this customer id okay customer id we will take the version maximum version so that version we will take and then while inserting it version plus one we will insert it okay so this is what we will go for the version method so flag method question method i hope you are clear on the flag and version method next one is called date method what is date method right so these two method we will we may not know when the data has been changed okay when the data has been changed you you do not have any idea but in version method in the date method right we will maintain this we&#39;ll maintain the column like effective date and end date so what is that effective date for an example today the record you are changing this new record is added into the source then effectively it will be system data so this state will be effective date the end date will be defaulted 999 will maintain so we may not know when the record is going to change okay when the record is going to change we may not know right so that&#39;s why end date we will maintain like nine nine nine if there is a change in the source the source we will insert the record effective date equal to today&#39;s date say for an example today&#39;s date the state is the today&#39;s date and the end date is the 999 and the previous record this same sramanti record here okay this record effective date is same date only but the end date we will mark this system date to end date okay from this date to this date this is an attribute after that from here this is an attribute so what is the which one is the active record the active record is nothing but whichever the date is end date equal to 999 sometimes some project they will use null value some project they will use two triple line some project where they will use four names okay but anyhow it&#39;s a date method this is what we will maintain if there is a new record in the source look at here if there is a new record in the source we will insert the record with the effective date is today&#39;s date end date is defaulted 999 if there is an update if there is an update in the source we will insert the record with the effective date is while inserting this data effective date is today&#39;s date and date is 999 but while updating it we will update the end date equal to system date clear so this is what we will maintain the effective date indeed okay effective date and date look up you have to always look up on the 999 so this is what you will do so i will do this date method so you can practice this version and the flag method or you can check it and the cd type 3 so type 3 is nothing but say for an example this is what the source type 3 is nothing but reason history will be maintained that to column level so whichever the column we have to maintain for history so that column alone we have to make say for an example i want to maintain a history for mobile number history for address it refers zip code history for zip code and everything so that column alone i will maintain here so previous mobile previous address previous report all this okay so this is what i will maintain then if there is a change say for an example this customer is changing the mobile number okay it&#39;s changing the mobile number you assume that there is a change in the mobile number so what we will do we will take this mobile number already existing mobile number ctrl c and ctrl b here we will we will copy and paste it here that means we will maintain as a history here then this mobile number we will insert it here we will update it here that&#39;s what we will do there if there is an update so one record in source one record in target if there is an update also the same record we will go for the history if again the record is going to change this will go to this one previous mobile number then this one comes in the actual mobile number but here the previous to previous mobile number will get updated okay this is what we will maintain the scd type three only column level history will maintain okay so i hope you are clear on this what is std type one type two type three and type four sorry type three any doubt in this any doubt on this method flag method question date and type three okay i hope you are clear we will take so the mapping if you look at here so if you have practiced if you have already practiced the update strategy then this is very very easy okay it&#39;s very very easy the extension to that particular update strategy only this a cd type 2 history so look at here i&#39;m taking the source table and target table select star from yes underscore customer underscore cd okay so this is the table we have in our source and the target we have t underscore customer underscore svd underscore date is another another table initially we will truncate the table okay so we will manually truncate this table truncate table table name but after that you should not truncate it so this is what we will go for t underscore customer underscore a cd underscore date okay this is a table we will go for we will take these two tables for source and target so you&#39;ll be expecting lot of questions from this so if you have any questions please ask me and by today itself you have to practice all this okay we can go to mapping m underscore customer a cd type to gate method okay date method i will do it other method you can do it std type 3 i have already posted video on youtube so please watch it you can able to complete the source i&#39;m going to take the source table the source i&#39;m going to import it hr and i can take the source table yes underscore customer underscore a cd on the target we will take this target table e underscore customer underscore a cd date in this table so make sure that target has primary key okay so this is called surrogate key here and we have one more key column this is not key column in order to maintain the history we are maintaining one more column okay effective date indeed then go to mapping it&#39;s very simple only the same like update strategy only so we will take yes underscore customer underscore sct stable source table okay and then t underscore customer underscore svd date so this is the target table okay so this is a target table and i&#39;m going to take expression transformation here so i take expression transformation so we have to find whether this particular record will go for insert or update just i have taken expression transformation then look up on the target right so we have to look up on the target against this customer id so take the lookup here on the target table so this is a target table choose a target table and look up again source customer id okay this customer id and this customer id we have to match it the condition customer id customer id and just i have matched it so from the lookup i will take all the columns okay so these columns are from lookup customer customer key i need to take i do not want this effective date and date just you can take from customer key you can take all this column okay so we will we will create two port one is insert flag update flag so as we have seen in our update step update strategy transformation we will take new record flag so this flag we will set this flag whenever new record the source new record flag this is update record flag so we will set this particular flag whenever there is an update in the source update record flag then we have here the expression in the expression we will have the new record is nothing but okay if a f is null of professional love customer id okay if it&#39;s no love look up customer id these columns the one is nothing but lookup so if you want if you want you can make it like lkp underscore also so if you if you&#39;re thinking that no need then leave it if it&#39;s looked up customer key as customer if it&#39;s not look up customer id then i&#39;m making like true otherwise i&#39;m making like false so this is what we have to maintain this is what we have to maintain for new record new record flag okay validated it should be validated then update record flag so when we will update when we will set this particular flag if if this customer id okay this customer id equal to this customer look up customer id and and first name first name not equal to this first name or any one of the column we are going for r okay mobile number are not equal to lookup mobile number or address not equal to lookup address or zip code not equal to this code or country not equal to this country okay if it is any one of the column is not equal then we&#39;ll maintain like true otherwise false okay so validate it it should be valid then click on ok that&#39;s all then we will maintain two more columns one is called effective date end date so effective date we can mention like date and time column effective date is nothing but your system date only state you can maintain assistant okay this is date here and end it end date will be your default date so it&#39;s also date and time you can go for date and time is nothing but so we will maintain like this two underscore date of 12 31st 999 single quotes comma single quotes mm dd yyyy format 4y okay this is what we&#39;ll maintain okay validated apply okay here so we have to take next level source column along with customer key so we have to take customer key also next level why because customer key is the primary key column which by using this key only we are going to update that that&#39;s why we will take customer key other lookup column no need to take okay we i&#39;m going to take a router transformation to route it so look at here i&#39;m going to route it so take all the source column and customer key also and we will take all these four columns okay so these four columns only these two columns are driving columns like new record flag update record flag so here the target if you look at here this target double click on the router go to groups i&#39;ll create a two group one is insert group one is another update group so this is what insert a group this is what update group update group okay so in insert group we will insert it will make new record group new record flag is true so this is like update record flag is true okay apply okay here then the insert we will insert it customer id up to here we will make it an insert then effective date and date and customer key so what is the customer key we have customer keys nothing but we&#39;ll maintain so key values right so here we will maintain the sequence of numbers so for this we have to use sequence editor so we will use sequence enter one two three this is sequence editor take the sequential tree here so take next value to this customer key double click on this to properties so from start value one we have to maintain we should not enable the cycle option since it is a primary key and we should not enable the reset also okay so control yes then so this one we have completed this logic we have completed okay new record we have inserted effective date is today&#39;s date and it is 999 okay we will update it for update logic we have to insert and update so take this ctrl c ctrl v two times okay so we&#39;ll maintain two times here so for update we will insert right we&#39;ll go for insert so for update insert we will directly insert it okay we&#39;ll insert it here we have to take it from the update group okay remember we have to take it from update group so this is update insert effective date indeed so look at here effective date ended then the same sequence we have to maintain take this next value from the same sequence number from here also here also so why because here if you look at here already five records has been inserted for update insert also we have to maintain the same key only if you are maintaining a different key for update insert then again it will go for one only it&#39;s a primary it will fail so that&#39;s why we are going to use same sequence generator so we have to use same sequential for this okay update update so this is update update right for update update we have to update the date column so we will take update strategy here by using the key value by using this key value we will update by using the key value say for example this record here right by using this key value we will update end date equal to effective date okay take effective date from here okay double click on this update strategy go to update so we can main mention that dd update that is one a dd update is one then by using this customer key with this primary key we will update so this end date equal to effective date that&#39;s all okay that&#39;s all you will mean so this is for insert logic this is for insert insert this is sorry this is update insert this is update update okay so here in lookup we have to look up only active record right we have to look up always active record so go to lookup sql override so here just click on generate sql sql will be generated you can mention that where class where so look at here where end date equal to so this is what the end date right where end date equal to two underscore date of 12 31st 999 single quotes then we will make mm slash td slash yybaby 4y single quotes close it so this is what you will maintain for we will always take only the active record so this query will be executed on target schema in the course schema right so if you want you can execute it from here also so from here also you can execute you can check it whether this query is executing fine or not yes it is executing so we can make it like so control yes apply here so one more time i will tell you first we are looking up the target table okay we are looking up the target table against customer id this customer id we are matching it and here we are creating new record flag update record pack if you snill off customer that look up customer id then we are making this flag as true then this flag if all the columns are equal if any one of the if key columns are equal if customer id equal to the source customer id equal to target customer id okay then any one of the column is not equal we are setting up this flag if the new record flag is set up then we will go and insert the flag inside the record the update update record is set up then we will go and update the record in the target okay so for update logic we will insert it and update it for update we will go for by using the key value we have to update effective date equal to this end date okay any doubt on this so now we will create a workflow we will create a workflow and then we will see the result look at here i will create a workflow wf underscore scd date okay date type to date method so i will take this mapping so remember we should not to truncate and load the target so fail parent of this task fails go to mapping so should be pointing to source target should be pointing to target connection go to target here also maintain like a target don&#39;t go for truncate and load look up also we have to provide the connection so here just we will maintain apply okay save now i am going to execute the particular workflow before executing check check here there is no data in the target so i&#39;m going to start the workflow very first time all the record will get inserted into the target table so record got inserted if you look at here the statistics five record in the source so got inserted five records this is insert logic so this is update insert update update okay so if you look at here effective date will be today&#39;s date effective date will be today&#39;s date end date will be 999 this is what it will maintain okay if i&#39;m going to execute the same workflow again i&#39;m going to execute it should not insert any record it should not update any record in the target see here only it is reading the data five records from source it is not going for any insert in insert update insert update update okay it&#39;s not going wrong okay so i&#39;m going to update any record in the source so manually i&#39;m going to update say yes underscore customer underscore update so look at here this record i&#39;m going to update it the source side any one record i&#39;m going to update so here not update this is sdd this table i&#39;m going to update this mobile number i&#39;m going to update it okay this mobile number i am going to update it so i have updated the mobile number okay so commit it so we have to maintain the history for this right so it should be 5 0 1 1 look at here 5 0 for new record so this one for update insert update update so you can see here so this mobile number so previous mobile number this is new mobile number okay this is the new mobile number okay seven zero one double zero eight zero so this is what it has updated right so you can see here so effective date so since we are we have updated on the same date it is making two days date so this is what so which one is active record so these are the active record so by looking up these five records only should be taken for the consideration clear on this any doubt so this is what scd types okay type one i have already we have already completed type two flag method version method you just try it type three i have already posted in youtube so type two you can complete it okay so practice all the types if you have any doubt then please let me know [Music] you

hi everyone welcome to informatica power center complete tutorial session in this session we are going to learn complete informatica power center with practical sessions so please watch the session and practice it without skipping it if you haven&amp;#39;t subscribed our channel please subscribe the channel and click on the bell icon so that you will be getting notified whenever i&amp;#39;m uploading the videos if you have any doubts any clarifications while watching the session please add your doubts and clarification in the comments i will respond to it without wasting your time let us move on the agenda this is the agenda for today&amp;#39;s session we are going to learn about what is business intelligence what is data values and what is data lake so what what is the difference between these two and what is etl extract transform unload and what is informatica so once we have seen the basic then we will go with architecture of informatica and informatica different client components and we are going to learn about informatica different load types flat file table to table all the different types of load and the different transformations this is very very important so this different transformations once we have completed we are going to see all the practical sessions then the most importantly slowly changing dimensions and the performance tuning also [Music] so what is business intelligence business intelligence is nothing but each and every company at the end of the day they want to track their business how the business is going on okay at the end of the day they want to track their business you can take any any uh organizations like banking insurance automobile industries health care right so any industries even you can take any software companies also any manufacturing companies any small small companies also at the end of the day they want to track their business how the bad business is going up see this business intelligence tool is meant for business people like a manager senior manager director vp&amp;#39;s and see even sometimes ceo so what is their job so each and every day they will analyze the business okay how the business is going on so which product is moving fast which product is not moving fast if i&amp;#39;m going to increase some product price so what could be the impact on the customer&amp;#39;s feedback or so what how the customers is liking the product uh so which product they are liking which product they are not liking okay which product is having fast movement which product is not moving fast say sales and profit all this revenue and whatever we are doing it on the customer how many customers are there all over the globe so just we can analyze it everything like every customer every organization at the end of the day they want to analyze the business if you take online food delivery app okay how this app will be how they will analyze it right see at the end of the day okay how many customers today have they have ordered how many delivery partners are delivered by on time okay how many products 90 percentage 99 percentage of the delivery has been uh delivered within the time we have given okay out of which how many of them priced some concerns something like that right see a lot of analysis they will do they will improve their business okay this is what the dispute we are getting it how can we improve it so what is the sales and everything they will go for it on the end of the day they will make their reports right settlement reports lot of they need to set amount settle the amount to the vendors and the merchants different merchants will be the right hotels and everything they need to settle that after settlement they will analyze it okay how many of them are going for cash how many of them are going for online okay like cash on delivery and then credit card debit card paytm so different kind of analysis they will do then they will go for okay these customers are these many customers are the regional wise customers if i have all of the india business then which region we are having huge customers okay transaction sales profit and everything they will analyze it right then only they can they can improve their business in next day next quarter next year right they can project it so this is what the business intelligence see if you are the java developer you are developing application for the end users i&amp;#39;m going to use that then you are developing it but business intelligent developer you are the etl developer informatica developer are like a tableau or power bi developer we are working for the business people business people you know what is done by artificial intelligence right so artificial agents we are we are trying to make the mission to think and then take addition business intelligence is nothing but we are giving the data to the customer that is what business people then they will take addition accordingly if you look at here in 1980s we used to store the data in the database in the database we used to store the data in the database for business transactions at the same time we will pull the report from the database itself on the same database itself what will happen if you consider any banking any banking whatever we do atm transaction whatever we are going to the bank and then we are doing the transaction or online transaction or gateway transactions whatever the transaction we do all the transaction will happen will store in the database right so it will store in the database at the same time banking manager that particular regional manager branch manager they want to know in our branch what is the today sales what is this today&amp;#39;s deposit so based on their business they will analyze it right what is the today&amp;#39;s deposit and with travel how much is the loans personal loan journal loans different categories they will analyze it what is the amount we have taken today and then what is a current amount we have see all this analysis we will do so that&amp;#39;s what report okay that is what report this is a transaction see if i&amp;#39;m going to do all this in one single database right if the data volume is very high this particular system will get impacted impacted means you are pulling the data for reports at the same time you are storing the data on the same database then what will happen this particular database will get impacted impacted in the sense you will feel very slowness some network slowness right same way you also will feel some slowness so that&amp;#39;s what they have come up with some solution called okay don&amp;#39;t take any reports from this database we will create one another database okay we will create separate database for reporting purpose for reporting purpose alone for analysis we will create it nowadays we have the data lake and all right see the same concept okay data lake and it&amp;#39;s the same concept for the business analytics we will have the data on the separate database okay whatever the data we are collecting here at the end of the day we will move the data in 1997 at the end of the day only we will move so one day once we will move the data from here to here so this database we will call it as oltp system oltp so what is my olpp oltp is nothing but your online transaction processing online transaction processing your business will be there from different places you&amp;#39;ll be having the different places different applications your data might be coming from sap data might be coming from a sales force and oracle sql server db to different different applications right different different places also see all this data whatever the databases we have say for example here on database here on source applications here another source here under source these aren&amp;#39;t for business day-to-day business your payroll information will be there in the sap right general results general ledger crm all this related data will be available in the oltp this is for transaction say for an example if you take irctc whatever the ticket we are booking right so all the ticket booking and everything will be happening in this ol tp only okay then well what is ol t oil ap oil ap is nothing but this is what our ol ap oil ap is nothing but your online analytical processing whenever we want to do analysis i&amp;#39;ll go for the online analytical processing you do all the analytical okay so analytical process in this olap system don&amp;#39;t do on oltp see this is what the oil ap do analysis on the olap system olap means it is a data barriers what is warehouse meaning vargos is nothing but it&amp;#39;s a place where we will store all the historical things right same way here we used to have the data values initially we used to have the data variables nowadays we are moving towards on the data lake see data lake is nothing but more than the volume of data warehouse this data vargos will handle only structured data structured data in the sense it will have the data only in the table format like how we have the data and the excel sheet right the same way you&amp;#39;ll have only stable format we will have the data but in data lake we may have the data see not only one single table we have a lot of tables okay thousands of tables will be there not thousands even 10 000 more than that also but data will be stored in tables like a structured one this columns and rows but here we have the structured semi-structured unstructured data we can have it on the data lake all our facebook comments on all our data right ola ubers wiki zomato see all this youtube and everything like to store the data here now we are going with this data lake only that&amp;#39;s what the spark will do all this with the processing okay see we we need to store the date this is what our business business we are not going to touch this for analysis if i want to analyze my business you should not touch this one you have to analyze all the analysis here only data here only okay so i have to move the data from here to here right i have to copy the data from here to here so every day i have to copy see this is what you will have and this offer oltp to olap data lake as i told data lake will have structured okay it will have all the structured data so that&amp;#39;s what the data lake will have you have the azure sql right you have aws also bigquery uh sorry gcp we had a big query it will have structured semi-structured okay so i will say short form semi-structure and then unstructured data also that&amp;#39;s what it will have all this not only is construction it will have all the data okay that&amp;#39;s what your data lake can handle it all the type of data okay so this is called your oa lapis oil ap is nothing but your data lake or data minerals we have to move the data from here to here right we are the heterogeneous sources this is called heterogeneous sources as i told you might be having the applications from uh we may have applications from sap right we have we have data from sap we have the data from salesforce okay salesforce and we might be getting the data from a file file system flat file and we might be getting the data from your oracle your oracle and your sql server see microsoft sql okay ms sql see all this microsoft sql server or you might be having the data from db2 see other data sources also any different data sources we can get the data this is called heterogeneous source system we will take the data from here okay we&amp;#39;ll extract the data from here and we have to load the data into this data of arrows now we are seeing only structured data we are not going for any unstructured data in our informatica so we are having only structured data that&amp;#39;s what we are going for the tool called informatica power center so this tool as i told yesterday this tool will extract the data from source it will extract the data from source transform the data load the data into the data barriers on a periodic time okay scheduled timing this process we will call it as etl process extraction transformation loading it&amp;#39;s like a postman okay extract transform load the data into data values okay load the data into data values this is the etl process okay elt process what is eld process normally in data lake we will go for elt process extract the data from here load the data into data lake after loading it we will go for the processing that&amp;#39;s what extraction ingestion and then processing it that&amp;#39;s the elt process so we do have different processing type one is the processing we have batch processing okay batch processing what is batch processing right batch processing means see we will be having a scheduled timing like one hour once half an hour once one day once weekly monthly jobs will be running periodic time whatever the transaction happened here periodically it will move the data from here to here for the analysis purpose for an example now the time is 8 30 one hour once your job is running etl process it should have executed by eight o&amp;#39;clock right eight o&amp;#39;clock but now date that time is 8 30 if you go and check the data in our analytics purpose up to eight o&amp;#39;clock only data has been refreshed so when the now whatever the transaction is happening right right now the timing it will get reflected at the nine o&amp;#39;clock only okay nine o&amp;#39;clock only so why we have scheduled this etl process by one hour once that&amp;#39;s what etl see you you are doing some cancellation and all you are doing some some some transaction right in the online process on that sometimes you they will say like it will get reflected after four hours after one hour or uh something like it will get reflected after 12 hours or 24 hours or three days something right see it&amp;#39;s all about etl process okay it will move the data from multiple applications it will make an entry it will get some entry and then it will go to the other one that&amp;#39;s what it will go for this is called batch processing we do have live processing we have something like you have iot devices right iot internet of things iot devices or you may have having the data from sensors or facebook comments and like this data sources you have the kafka it will read the data it will have the all the logs it will pitch and then we have a lot of tools are there by using that we will have the spark engine then we will load it into the data lake we have other tools also this is called live processing live processing means we will be seeing the near real time data near real time means say 8 35 you are seeing 835 data okay that&amp;#39;s called live crushing it will take live data from here so it&amp;#39;s all your iot devices and everything okay it will take the data from here this is called iotr on the devices live processing it will take the data and then here the spark engine or kafka will be pulling the data and then here it will load the data here you will be processing it ingestion this is called ingest will say like it&amp;#39;s a extract ingest and then you will be processing it here we will go for the process okay this is called your data lake processing engine that&amp;#39;s what your aws your azure okay your gcp all this will do this is called live process we have batch process also in our data lake under that&amp;#39;s what it will have all this structure semi structure and structure also this data of arrows normally we used to have oracle data values or the other teradata heard about this teradata terminal right so you are the this is for the data variables okay we do have something like this is what your teradata teradata is the first database to handle the terabytes of data so you might be having oracle article also variables and you might be having sql server microsoft sql server also so these are all some of the data barriers previously we used to have netizer you are the natives are right all this so in data lake nowadays if you come to this second part you have the the microsoft azure right that the azure we have so as you and you have the amazon aws right we can store for s3 bucket and all but the data lake and we have something like a gcp google cloud platform and we have we can have this snowflake see these are all the uh some data lake we have in our real time to store our data and in addition to the storage it will do all the processing and machine learning algorithm all this it will do that&amp;#39;s a different part but we will use this one for our business uh process okay this is what we&amp;#39;ll go for the batch processing and live processing all this okay i need to have the tool right here i need to have a tool which tool i can use there are different tools are there that&amp;#39;s what our informatica power center will do one of the tool is called informatica power center okay one of the etl tool is called informatica or center see informatica is the company name informatica power center is the tool tool name see the informatica power center is used for data integration it&amp;#39;s integrate the data from here to here it will integrate data integration rule informatica will not store any data in order to load the data from here to here we are going with that&amp;#39;s what we are see if you look at iictc and all in that class they will not allow you to take the pnr status right why because this is oltp if you if you go and hit oltp the that collage that collapse is like more number of users will be accessing it okay if you are going to check in the something like reports on pnr status and all it&amp;#39;s again overhead to the oltp system it may impact on the particle booking enough so that&amp;#39;s why they are stopping it right same way in business if the business hours is happening business is happening here if you go and hit this database for this analysis purpose it will get impacted that is why we have so you can ask me okay here informatica also is getting this database right and then what will happen that is what we are not going to hit all the data we are going to get only up on our data on our data our one day data maximum and also we will have some replication server we will not directly hit this old pa we will have some replication server here some server it will replicate last 10 days of data here okay whatever the data here like a mirror server we will use that mirror server it will mirror the data whatever the data is available last 10 days of data only it will be mirrored we can have last one year of data but only 10 days of data will be there in this layer we will hit this layer only not this layer this layer is for database for business purpose okay clear on this that&amp;#39;s what we will have the databases and everything okay so you have loaded here after that from here we will extract the data for our reporting purpose reporting in the sense we do have lot of reporting tools are there so what are the different reporting tools are there the market we are the tableau from salesforce then we have microsoft power va so we have power ba power business and legends then we have uh click view from click tech we have the click view or click sends so like this we have a lot of tools are there from looker from gcp we are the looker gcp in the sense you are google we have a looker and we are the quick insight right so from aws we are the quick insight see all these tools are very much used for this reporting purpose okay reporting purpose how we are the informatica power center right same way we do have other tools i will show you okay what are what is these tools i can show the data to the customer in terms of chart and visualization chart by means of chart it&amp;#39;s very easy to understand them okay then where this machine learning algorithm will come into picture artificial religion will come into picture see here we are loading the data right we need to feed again data to our customer for the better user experiences so if you go to amazon they&amp;#39;ll be giving you the suggestions right okay you buy this product based on your search history they&amp;#39;re giving you the suggestions all this right so how they are giving us against your customer id they used to collect all the data they will process it based on that they will processing it again they are feeding back to the source system okay for the recommendations youtube recommendations amazon see a lot of a lot of application now they started with like a bestseller review comments whoever is getting good review then it will be on the top right see lot of filter condition lot of analysis they are giving that right see all this will be we are analyzing it all the data huge volume of data again we are feeding back to the oltp system for our business users okay that front end they will collect the data from here they will do use it this is what machine learning algorithm artificial religions algorithm will be applied on the database and then data lake and then they are feeding back to the source system again full stack developer under they will use it they will use this data lake as the source and then get the data they will develop it see these are all the data by using data lake life cycle here if you are going for data lake uh data center and all you can see this is what the data center so look likes it&amp;#39;s again huge volume it&amp;#39;s not only seeing uh small volume it&amp;#39;s like a huge volume only google cloud we are these are all the data centers data lake look at here how much is the data lake it&amp;#39;s alibaba you know right chinese china alibaba it&amp;#39;s like a data center here this is what they will have the data lake hash lake cooled data center right so google also they have the data lake and everything lake data center it will it is used to store the massive data that&amp;#39;s what we have the data center look at here this architecture even we can get the data from ingestion i told right ingestion we&amp;#39;re tracking all this we will collect even from data values also we will collect okay storage and then data catalog data breaks so that&amp;#39;s what it will move the data to be used in real time these are all the data lake in real time okay data centers we have huge volleyball hundred data values again it&amp;#39;s huge volume only but when compared to data lake it&amp;#39;s very limited one okay if you know this etl pipeline very well informatica and all very well learning this concept is very easy machine learning all this etl machine learning is very easy that&amp;#39;s what extraction transformational loading machine learning also we will do deep learning machine learning all this related to our retail process only once we are the etl do that etl data sources etl data vargos and business intelligence look at here how the architecture will be right as i told we may get the data from your online resources right your online shopping your mobile your cloud so all this you are collecting the data flat file databases iot devices and everything this data source you do the etl and load it into the data values are elt i told right elt also anal analytics query and reporting your other business intelligence business intelligence means your tableau power bi all your mobile and everything that&amp;#39;s what you&amp;#39;ll be having all this see logs files and media unstructured unstructured business custom application is a structured one so you are taking that ingestion storing and preparation and training and then you are doing that all the modeling and everything then you are going to the power bi or web application or front end tool or everything that&amp;#39;s what the pipeline of azure and everything okay ingest ingest means extraction okay ingest transform load into a storage then preparation and training preparation means machine learning and all this we will train that and then we will give the data to the applications front-end applications but the better user experience on that okay so this is what the etl process will do i hope you are clear on this etl process so that is what i have explained here so business intelligence is nothing but each and every company at the end of the day they want to track their business so that&amp;#39;s what the business intelligence so each and every this is the business intelligence tool for mainly used for the business people the business people like manager senior manager directors vps and even for ceo also so they they used to start their business day-to-day activities by analyzing the data so for this particular business analytics as a etl developer they will load the data into data values or data mark and then as a report writer they will present the data in the form of data visualization tool in the form of dashboard or scorecard or different reports the business people will analyze the data and then they they will take addition accordingly okay this is what we can go for this is the process and this is what the sales we we got it and this is the profit this is the margin so a lot of analysis they will do on the data they will take addition accordingly so this is what the business intelligence as a etl developer we should know all this concept very clearly so what is etl as i have mentioned etl is nothing but extract transform and load so this is what the etl process one more time i will explain so the left hand side you are seeing that heterogeneous source systems so this is what the heterogeneous source system we have so like we can get the data from oracle sql server db2 uh even we can get the data from even we can get the data from the social media and then we can get the data from any live sensors and all the data we use to collect extract the data transform the data and blow the data into a data variousing layer or data lay so this process we will call it as batch processing so batch processing is nothing but it will be a scheduled run so every day it will be scheduled run it will be so even for half an hour once one hour once the data will be pulled from the source system and then it will be loaded into a different layers like staging layer and multiple layers intermediate layers multiple layers or players and data mark so data mark is nothing but a subset of our data values so this subset is nothing but say for an example i am working for merchant reporting then i&amp;#39;ll be having merchant data mart and then supplier data mark finance data mark some different different data marks will be there for the particular subject area so from this so for this layer this is the source and this is the target staging layer and then staging is nothing but it&amp;#39;s a landing zone for our data so even we have the data from heterogeneous so system we will land the data into a staging layer from the staging layer we will move different work layers we will apply a lot of business logic here for this layer so this is the source and this is the target again for this layer this is the source and this this is a target the data will be moved like this one so source and target so all these are etl process only this is what data will be moved so this is like oltp system this is like olap system so we used to have lot of different etl tools are there in the market so one of the widely used etl tool is informatica power center then we are then the cloud system informatica intelligent cloud services microsoft ssis sql server integration service oracle data integrator different detail tools are there in the market we should know all this concept very clearly whenever we are going for an interview or we are going for a project so definitely this etl process will be taken place and we used to have the live processing or streaming so nowadays we are working on the data lake so data lake is nothing but the more than the volume of data values it can handle structured semi-structured unstructured data also unstructured data is nothing but our audio file video file how we are getting the data in the social media right so that the social media content mostly are unstructured data so in order to process the unstructured data that continuously we are getting the data from our source system this is called live processing a streaming like how the water is flowing in the pipeline right the same way the data will be flowing 24 7 from the source systems like our iot devices like mobile devices or sensors we might be getting the data continuously and then we are loading the data into a data lake from the data lake we will process the data again you can apply some machine learning algorithm or you can apply some artificial intelligence algorithm here and again feeding back to the data to a source systems for the something like uh recommendations our best seller something we are doing it some news feed so a lot of analysis we are doing on the data and then we are giving the data to a source system so this is what the etl process will be will be there in our day-to-day activities so oltp is nothing but online transaction processing to run the business we used to have the oltp in order to analyze the business i need to have the olap system so oil ap is used too far mainly for the business person to analyze the business they will go for the in order to analyze the business i need to have the data values or data lake that&amp;#39;s what we can do lot of analysis on the data after analyzing it i&amp;#39;ll be taking the decision accordingly so who is involved in etl process so whenever we are going for a real-time project so first we will be getting the data from a data analyst or the business analyst or the product owner or the business owner or from application owner so they will be creating the requirements so we will be getting the requirements from our business analyst our business owners and how the source to target and what are the different business use cases they will be giving us so based on that we are to the etl developers they will be doing the etl development right so they will be moving the data from source to target we have to validate the data so how the etl tester move the data whether it is accurate or not so what are the defects are there so we have to raise the defect and then we have to assign it to the etl developer and the etl developer need to fix it so this is what the life cycle will be going off so once we give the sign off then only it will be moved to the other environment data will be moved to the other environment the data architect our data modeler nothing but who builds the data store for us so it&amp;#39;s a data big data lake our data values are data marked they&amp;#39;re only going to handle it so if i&amp;#39;m going to do any business so what are the different tables it should get affected and what all the columns should be maintained the architect will be maintaining it as atl developer they&amp;#39;ll be moving the data from source to target as a etl tester we are to test whether they have moved all the data correctly they have implemented all the logic in our etl process our business analyst team they&amp;#39;ll be analyzing this particular reports only so they&amp;#39;ll be coming up some some use cases okay uh say they want to know if they are we are doing the some transactions transaction is getting declined okay declined so by the particular customer or by the system automatically some reasons okay so we have to analyze why that most of the transaction is getting declined the business analyst team they&amp;#39;re saying that okay 30 percentage of the transaction is happening just getting declined in the last stage or some stages it&amp;#39;s getting declined why it is getting declined so it&amp;#39;s a due to the card issue or the network issue or by by the for the person itself they are declining it okay a lot of reasons will be there right why they are declining it otp issue so lot of analysis they will do see all the analysis they will do and then based on that to arrest this they will be giving us some suggestions okay if there is a otp issue increase the server capability to deliver the otp very quickly okay if you are delay if you are delaying the otp like more than three seconds to four seconds so there is a lot of possibility to decline that particular product the last stage so whenever they are they&amp;#39;re ready for the transaction you deliver it to otp quickly and read otp and then fill it complete the transaction within two seconds or three seconds that&amp;#39;s what okay that&amp;#39;s what you are completing the transaction nowadays they are having the e-wallet right so keep it to the ballot inside the particular uh the particular site itself load some monies before that so you can just one click take the date take the money that&amp;#39;s all okay so it&amp;#39;s a com completing the transaction very quickly then the customer so most successful transaction will be getting it from the customer right so that&amp;#39;s what the business development team they are analyzing it and then they will be giving suggestions to the business team okay if you are going to do this we can improve the business like this then it will be implemented on the otp side okay the enhancement will be implemented on the otp site server enhancement all this so the volt this aggregating the data is nothing but we are integrating data from oltp to olap so database data marked data values what is data marked the data vargas we know what is data various the data variables so the data marked is nothing but a subset of your data various subset of your data values it&amp;#39;s again logical division logical division of this data variables say for an example i want to send the merchant report i&amp;#39;ll be having merchant data mark okay customer data mark some supplier data mod all this data mark will be there different different data mark based on the access and privileges we&amp;#39;ll be having the different data mark so from the data mark we will take the reports if i&amp;#39;m working for merchant report only merchant data mart i will have access other data mod i will not have access okay so it will have some pa informations for the secured information that&amp;#39;s why for the security purpose they are splitting the data values into data mark based on your business area you take the data from this particular small area okay small subject so that&amp;#39;s what data mark will be data mark is nothing but subset of our data values okay okay so once you have load we are aggregated the data into the data values once we have aggregated then we will present the data in the form of data visualization like a reporting table power bi see all the reporting tool we will create the reports so if you are giving the data in this format it is very easy to them right business people it is very easy to them to analyze it right then they will take addition accordingly okay so this is seven six days of five days of business why this business is coming down our last five years last two years to go with some business class are something like that different due to covet we are having a lot of uh transactions so that&amp;#39;s what they will analyze it they will take addition accordingly okay how to improve the business so how to improve the business in this particular area so we have developed these many products out of this this is the product is moving fast this is the product is not moving we do have the stocks in this particular store we have to move the stocks to another store for the supply and demand so like this lot of analysis they will do on the particular business and then they will take addition it is not only one particular business team we do have different teams will be there business development team they will be doing it all the analysis based on their area and then they will take addition accordingly they will be feeding back to the source system we will move the data okay so in order to change anything on the source they need to get a lot of approvals from the business architect team okay so this is what data bar hosting architecture whatever i have explained so far this is what the databar goes architecture we have the source here erp enterprise resource planning are all other social media networking and everything e-commerce site and everything structured data unstructured data i told right it&amp;#39;s a real time data we are pulling our batch data we are pulling from the source the data integration layer all the data profiling all the data cleansing we will do it by using some idq tool or some layers so from here from here we will consume the data to the to the data we are using layer like a staging layer and one staging layer completed we will go for different layers enterprise data vargos this is called edw from here the data will be moved to data marked from here it will be moving to the all the bi applications like you have the tableau power bi all this mobile applications and then data will be moved to some front end tool also on demand services also so this is what the data are using architecture see data various the ctl pipeline will be there okay even if you are going for new cloud technologies definitely this data warehousing architecture will be there so we can move on to the cloud technologies no issues still lot of companies are using informatica power center for their etl process so if you take this is one of the analytics this is one of the analytics reporting side say i&amp;#39;m giving the data so this is one of the channel analytics say for example for that for whoever is having the channel we are giving the some analysis here okay these many subscribers for last 28 days 41 percentage increment then this many views this money watch time okay then this is the day by day how many uploads you have done and what is the views here so it&amp;#39;s very easy to understand right for the channel holders right so how much is the overview the viewers and what is the interest viewer something like that see all this you are giving filter here last 28 days last one day last one month right our last this particular month current month you can go for last quarter last three months last year or from the beginning so whatever it may be they are giving the line chart here bar chart see the numbers they are say giving so this is called data visualization the reporting okay they are using different tools to provide the data it&amp;#39;s a near real-time data see here it&amp;#39;s the latest activity last 48 hours they are giving say every one hour the data is changing okay now updating live right so this is called live data update so this number will change often okay four eight three seven four eight three two something this number will change okay up front it will change this is called streaming the data and then you are showing this one will take the streaming the data but now here we are seeing this one right last 28 days this is not streaming the data this is like your data warehouse data only aggregations this is live data this is aggregation data so in a single page they will go for the scheduled data also on live data also that&amp;#39;s what they will be showing them this is what we&amp;#39;ll go for the data okay so this is the architecture of data vargos you can see here this is the functional architecture of our data values here we have the crm okay customer relationship management on inventory payroll sales general ledger production all the data we will be collecting from source we&amp;#39;ll be moving the data into a data vargos from the vargos we&amp;#39;ll be taking different metrics okay so how the customer profitability what is this we are we are having different suppliers how they are performing it okay productivity and strategic performance closed loop marketing product profitability so in real time environment we do have different different metrics all the different different metrics will be collecting from the data various data only okay a lot of aggregations will be running the data so you&amp;#39;ll be having last three months whenever we are going for any quota results right the company is going for quota results they&amp;#39;ll be taking all the three months data right all the three months data so if i&amp;#39;m going to take aggregation of all the three months data if i&amp;#39;m going to do that particular calculation the source system it will get impacted will not get one consolidated data so payroll will be there in sap okay payroll will be there in sap some different different systems will be there oracle sap different different system will be there it is not a consolidated data but here single place single unit it will give you all our historical data last 10 years i want to know last how last five years we are performing based on that we have to speak in the board meeting and all so that we have to take from the data variable okay this is what will be data various environment okay so what is data values as i have explained it&amp;#39;s a subject oriented it&amp;#39;s nothing but your data mark subject oriented like customer supplier product sales one subject oriented integrated we are integrating the data from source system to data various integrated time variant all our datas are historical perspective like we&amp;#39;ll be having information like from day one to till date we&amp;#39;ll be having all the time variant data and non-volatile non-volatile in the sense like we will never ever overwrite or delete the data in the data barriers remember we will never ever overwrite or delete the data in the data values we will just load the data and we will keep the data for historical purpose okay backup we will keep it so if any data change is happening the source system we will allow to change it if you are if you are going to a band site and you are changing your address the bank oltp site will allow them to update your address but in oil ap you will have your previous address and this address also will be inserted newly the new address will be marked as active address the previous address will be marked as inactive address so if you take amazon so they will be having all your addresses right whenever you are adding new address they&amp;#39;ll be keeping and we are going for any uh checkout then they will be giving all suggestion of your address okay which address you want to take these are all business tricks so you know every time you are going to type your address it will take lot of time then customer will feel that very board right so that&amp;#39;s why they are already taken the addresses and everything they are giving right everything they are making very easy for the customers to improve their business that&amp;#39;s what non-volatile collection of data to support management decision making process to support management decision making process we are using this data vargos for the business intelligence i hope you are now you are clear what is business intelligence business intelligence is nothing but at the end of the day i have to make my business in a profitable way okay that&amp;#39;s what the business intelligence so i hope you are clear this is the again the modern trend values definition integrated architecture and periodic copying data from various heterogeneous sources both inside and outside the enterprise into a centralized repository optimized for reporting and analytical processing this is what we are doing for the databasing okay data lake also this is what okay why the companies deploying the data values are data lake why because operational system operational system is nothing but your oltp side ostp are being impacted because of reporting request so if your customer is asking last three months of data for the reporting purpose then if you go and hit this operational system so if you&amp;#39;re the backend huge volume of data it will consume right see so amazon per per second order is 120 hearts per second worldwide 120 orders per second like you could think about the volume the volume how much volume would be generated and then one day volume one week volume one month volume one quarter volume how many records will be created one order so how they are handling all the data right see logs and everything how they are handling it so in that particular system if you go and hit this oltp for the reporting purpose it will get impacted right so that&amp;#39;s why and also it will not give the oltp cannot have all the historical data we cannot keep all the historical data we will have all the data mining operation data variables operation in the data lake our databar goes on so we will not get any consistent data so that&amp;#39;s why and also it&amp;#39;s very expensive to maintain this oltp system okay ap side our data lake side we have commodity hardware we can have the nowadays we have the commodity hardware site or data lake side so that&amp;#39;s why we are going for the data values or data lake so these two members are called as fathers of data warehousing in mod and ralph kimball so inbound only the first person has suggested so instead of having oltp we can have one more system called oil ap one more system called oil ap where we will load all our historical data we will analyze it that&amp;#39;s what he has suggested first during 1990s 80s itself he has suggested he has published international journal papers okay from that only they have created data virgos see his suggestion is top down approach see what is top down approach right so first we will we will create see this is what our oltp oltp from the oltp we will take the data and into a olap system that&amp;#39;s what olap is nothing but our data variables we will load the data into data variables even data values different layers will be there that i will explain later data variables from the vargos we will move the data to data mark okay after creating the data vargos completely then based on the subject you are creating the data mark and then you are loading the data in this data values to data data mark this process we will call this top down approach this is a data modeling okay data modeler will do it our data architecting our data modeling team they will do it they will take the data from source okay then they will load it this is one on time automation automation process see informatica is fully completely automated okay we will schedule it after developing it we will schedule it so once it has been scheduled the data will be moved from source system to target system okay this is what it will be moved here the data mark will be there this is what the data will be more clear once after creating the data variables if you create data mark that will be called as top down approach after 10 years ralph tumble has come up okay so it is taking lot of time to create the data where which is taking a lot of time by the time so what we can do here we are the source system based on the subject you create different different data mark okay initially we will create a different different data mark load the data into a data market directly say for an example i want to have the reports for merchant i will create a version data mark only whatever the data i need to have this person i can take it from here from from this source and then i will load it into this data mark okay so it will have only merchant related data so here customer related data supplier deleted data that&amp;#39;s what we can load it it&amp;#39;s one particular subject area after that if you need you can move the data to data barrows so this process we will call it as bottom-up approach bottom-up approach okay in this process there is some here also some advantage disadvantage so based on the company&amp;#39;s need either they will go for top-down approach or bottom-up approach okay it&amp;#39;s a data modeling team they will decide it or they will go for hybrid approach so what is hybrid approach oltp to data mark data mark to data vargos data var goes to again datum data mark okay so this process you will call this hybrid approach so this is what we will do it in our real time projects okay who will do it data modeling team will do it they will do it all the tables data modeling and everything as a etl developer we have to load the data into this particular model so if i&amp;#39;m doing one business okay if i&amp;#39;m canceling a product how many tables how many tables it should affect okay so what are the different data i have to capture so that will be decided by the data modeling team as a etl team we have to take the data from source we have to load it into the data variables based on the business needs this is what we will do okay the inbound model as i have mentioned this is our source system so our heterogeneous source system we will do etl process load the data into a data values see here directly we are loading but in our real time it will be there multiple stages okay stage by stage data will be moved to the data values we&amp;#39;ll be doing lot of etl logic and then the data will be moved to data values so once the data has been date moved into data vargos then again it will be moved on the data marked so data mark is nothing but subset of our data values okay how we have in our system so 1tb of hard disk we are splitting the 1tb into c drive e drive right the logical division the same way here also the other logical division of the data barriers is called data mark so in real time project if you are working for one particular data mart so merchant data mart customer data mart supplier data mark okay different different data mart will be there if you are the tablet developer or power bi developer or any other developer report developer so definitely you&amp;#39;ll be having this particular data mart alone access okay you&amp;#39;ll be having access to this particular data mark so other data mart you need any access to this table you have to get the approval from the customer okay from the customer and then you have to inform to the dba dba will provide you the access okay this is what in real time you have access and privileges okay so top-down approach bottom-up approach first we will load the data into data mark here also we can take from here here also we can take data can be so you you can take the data from this data mark our consolidated data will be available in data overloads here also we can take it for reporting so anywhere you can take so this is for the bottom-up approach okay this is for the bottom-up approach so we are splitting the data into a data mart based on business okay based on business not for the uh not with any schema so i told right it&amp;#39;s a merchant data mart so merchant related data will be loaded into the merchant data mark okay merchant related data into a merchant data mark then customer related say you are working for customer reports then customer related data will be moved to customer reports see data marked is nothing but again it&amp;#39;s a subset of our data values only it&amp;#39;s a logical division it&amp;#39;s a logical division okay these are all the logical division we are logically it has been splitted but it&amp;#39;s a single physical hardware only okay now we have this is what next one is like we have next one we have data marked and data values and top down bottom of approach so what is the difference between top down and bottom up approach we have in one model it&amp;#39;s a top-down approach gimbal model it&amp;#39;s a bottom-up approach it&amp;#39;s a more time consuming okay if you to build the data values it will consume more time and the maintenance is very easy maintenance and subsequent project development cost is lower but initial cost is very higher okay so so first we have to build data values then we have to take the data marked it&amp;#39;s a longer startup time okay it&amp;#39;s for enterprise wide so for the entire enterprise side we&amp;#39;ll go for the data values so this is what individual business area so one business area we will go for i told rights one subject like merchant customer supplier right so all these are gimbal model so it takes lesser time you can create a data mark quickly and then you can take the data but after that it will take some no initial cost but after that the maintenance will be high okay this is what the difference between the inbound model and gimbal model now we have both both model okay so hybrid model so from source do the etl load it into a data mart so from the data map will be loaded into the data vargos okay then it will be moved to the data mart again again data mark so from here also you can take the data write the user from here also you can take the data from the user okay this is what it will be hybrid approach so who will do all this in real time so we are as a util developer we are not going to do data modeling team they will do it okay data modeling team some separate teams will be there so we do have two different tables in informatica that&amp;#39;s what data various so this table is very very important so whenever you are going for enterprise data by those so definitely you are looking for only two types of table either it should be a dimension table or it should be a fact table so what is diamond and fact table if you go to the data various right data values i want to build a report so how we will load the data say for an example if i have employees data i will load all my employees data in the employees table okay all my customers data loaded into the customer table customer data so all the customer id and customers data employee employee data product information and location information see all this information will be loaded into a different different tables so these are all date table okay location store table different store so why we are loading all the data into a different different table right at the end of the day i want to know what is the store wise sales what is the product to sales what is the customer ratio sales regional is country ways right different different dimensions i want to analyze the data then i&amp;#39;m loading all the dimension into different different tables then i will be having a fact table you&amp;#39;ll be having a centralized fact table here it will have all the transactional data okay whenever we are doing the business see first we are capturing okay this this customer purchased this particular product okay on this particular date okay the from this particular from this particular employee from this particular store particular region particular country see all this particular store everything i will capture so this mini product he has purchased one product cost is this much and total price is this much so each and every transaction will have the data so this data we will call it as fact table this is the fact table okay it&amp;#39;s the centralized fact table fact table will have the transactional data so always factor will have transactional data all the keys from this table will be connected the key values from this table so this is foreign key primary relationship so you can watch my sql session so today i will tell you how to install the software and how to practice our sql that i will tell you so please start working on it so this is the fact table this is a dimension type so all these are dimension table dimension table will have always a primary key okay remember dimension table will have a primary key and fact table will have a foreign key okay fact ff you remember fact table will have a foreign key so what is the foreign key these are all the foreign key then measurable values these are all the measurable values right so measures you&amp;#39;ll be having it non-measurable attributes and primary key so what is primary key so i will tell you now so this is the data model we will create so who will load this table as an etl developer we have to load it okay as an etl developer we have to load it so how to do it so i will tell you how these tables are related if you look at here whenever we are doing transactions in our real time okay whenever you do the transaction now real time if you look at here i want to start a company so that i will have all the reports so how we will have the reports now real time right so i will load all the product related table in the product related details in the product table so each and every product which is available in my company will have a product id separate product id okay each and every product will have separate product id product descriptions all the related to product description this is called primary key this is called different dimensions okay attributes attributes of the particular dimension table if you look at here dimension table will have a primary key and attributes the primary key here it will not accept any duplicate value i cannot give for this any other product i cannot do the same number okay without primary key without this key i will not have the product name on double zip so primary key should be there so separate key should be there if you are going for any company you will be having a separate employee key right employee id the same way each and every product will have the product id each and every customer will have the customer id okay so every customer has been loaded into the table we&amp;#39;ll have the primary key the same way employees table accounts table all this table will have it okay i will have all the tables the store table locations table this is what the diamonds and table see whatever the diamonds are needed for our data model i will load all the dimension table so from there this dimension table will be loaded from source only from oltp to olap we will load but while loading the old ap we will maintain the history that i will tell you how to do it but we will get the data from ol tp100 okay oltp also they will maintain that is what er model entity relationship model we also we will use in our oltp system okay so now we have the fact table so if you are going for a particular store right you are doing any business like you are purchasing a particular product then you are you are giving that product to the particular person in employee so they will be maintaining that they will be reading the barcode of the particular product and then all these details will be captured so what are the details when you are purchasing what is the date okay which product you are purchasing what is your customer id they will take your customer id card okay or the number mobile number which employee is helping you to purchase the product okay and store id which store you are purchasing a product which regions so all this will be captured in this fact table sales table see these are all foreign key so why i am capturing so whenever i am going for the transaction i will capture that and quantity so how many quantity so two quantity has purchased unit price and the total price okay so one unit price is three thousand five hundred two unit price seven thousand all this will be captured for an example your customer is saying that okay at the end of the day how much you have today&amp;#39;s sales so what will you do you will go for sum of total price sum of total price date key equal to today&amp;#39;s system date will take and i want to know in today&amp;#39;s date how much is the sales for each store so group so here we have to make date key equal to today&amp;#39;s date and group by store id we have to make the sum of total price this is what we will maintain never reports okay this is what we will take but i do not know which store right so 7667 which is which is a store then i have to join this table this store id fat sales store id to diabetes and table store id i will join it i&amp;#39;ll get okay so mumbai kalyan store this is what kalyan region store we have the sales so highest sales so that i will take it from here but i have to make the summation so which product is which store is which region is so everything so this is what in our now real real-time environment we will be keeping that fact table and diamonds and table how these tables are related like a star schema snowflake schema the schema the star schema is nothing but the other star schema is nothing but like we have like a star this is the fact table so you&amp;#39;ll be having a centralized fact table this is a sales table you assume that this is a sales table this is the dimension table if you look at here this is a dimension table we are the city table so city id city description all other remaining columns also will be there employee table country table products table see all these four tables are dimension table this is the fact table this is the fact table so in this particular country this particular city this particular employee purchased this particular product so two quantity he has purchased one quantity sorry one unit price is 200 dollars okay so he has purchased two quantity and 400 dollars that&amp;#39;s what it will be maintained so from source only we are taking the data from source only we are taking the data but we will maintain in the fact table in our olap site okay see if in fact table the outer tables are dimension table the diamonds and tables are connected with centralized fact table to get a meaningful report to get a meaningful report this is for this task comma now we are the snowflake schema see in the star schema which table we have to load first so primary key table should be loaded first so it&amp;#39;s a master table reference table okay always we will maintain all the primary key but remember this dimension table will not change frequently so do we need to load all the country data every now and then no right so one time the country data will be loaded so always the country data same if you are going to operate new business in new country that time i will maintain one more country otherwise we will be having the same country this dimension table will not change frequently that&amp;#39;s what it&amp;#39;s a slowly changing dimension okay slowly changing dimension product table also so product table so we will have new products but very slowly it will be changing but fat table is having huge volume of data okay this is star schema in star schema we have to load dimension table first then centralized factor clear so this is one of the interview question they will ask so how the dimension modeling they are maintaining in your project so which table you will load first okay that&amp;#39;s what we will ask they will go ask questions okay so now we will have snowflake schema so snowflake is different one snowflake technology is different that is data lake i have yesterday i have explained snowflakes schema is different okay snowflake schema is in data various environment in our data various environments snowflake schema is nothing but okay the snowflake schema if you have this is one dimension table right this is one dimension table this one dimension table we are splitting this dimension into multiple dimension table look at here this dimension table we are splitting into multiple dimension table so instead of having all the data in the single dimension single locations table okay we are having the data into the another table okay store id store name store territory information before that we have handled here okay after that we have moved this data to another table okay we will maintain the store territory details in another table called territory whenever needed whenever store territory needed i will have this id okay with this id and then i will maintain this i will take this report from where this particular store is located whenever needed i will go for store name yes i will go for store name with the store id whenever i need the store territory then i will join this table i will get it okay so this is a primary key table this is a primary key table that is a dimension table this is also a dimension table see one dimension table refers another dimension table based on the normalization this is called normalization what is normalization so normally say why we are going for normalization so normalization we are going for normalization to avoid the data redundancy see in real time oil ap tables are denormalized okay denominate in entity relationship model in oil oil tp side will oltp side is a normalized table olap site it&amp;#39;s a denominated one okay our data variables is denominated so denominations is nothing but this is what a determinist but here snowflake again we will normalize it so what is normalization to avoid the data duplicate data redundancy repeatness we are going for the normalization look at here say i am keeping my employees details here i have the employee id i have all this id column right so i have the employee first name last name everything i have here right so this one i have see here i have the employee id okay so here i have first name last name all this i have so i have these details are from office address building address poc and everything is nothing but our office location so you have three different office location you assume that you are the three different office location you have one lakh employees okay if you are keeping your employee details and office details in a single table what will happen so every time you are repeating the same addresses officer three of our office addresses to all the employees correct if you have online employees the three office addresses are repeating for one lakh times here instead what we are thinking maintain this harvest header separately okay maintain this harvester separately like this only three office address right three different obvious addresses three different times you will make so in how is understable address id will be the primary key okay in employee table employee is the primary key so this should be a par in key okay this employee located in office number two three one like this you can have it right like this you can have it so how many times i maintain now now only three times so this is what i can reduce the data redundancy clear this is what i can reduce the data redundancy so this is called one dimension table i am splitting the dimension table into multiple tables this is called normalization first normal second normal third normal right so this is their normal form this is what we are maintaining so why to avoid the data redundancy right so this is what we&amp;#39;ll go for the data returns and everything see one dimension table splitting the dimension table into another dimension table so here if you look at here here you will be having the primary key here store id will be a primary key right store id will be a primary key and same dimension table will have a foreign key if the dimension table is there in foreign key now real time that&amp;#39;s there in the snowflake schema okay here it is a primary key here it is a primary key so what is primary key foreign key so that you have to learn from oracle only from sqr only okay so what is the two different data variables and data lake data vargos is nothing but the structure data only will be handled and the data will be processed okay processed data will be loaded into the data values in data lake the other structured semi-structure unstructured data will be handled raw data will be loaded into the data lake from there it will be processed scheme that&amp;#39;s what schema and write schema read raw data will be loaded after that we will read the data from the storage here we will we will have commodity hardware low cost storage here it is expensive cost is expensive that&amp;#39;s what we are moving towards on the data lake side in data values mostly the business people will use it business analyst okay product monitor all the business people will use in data lake environment all the a and ml data scientist they will use it so if i want to use a and ml concept here so we will use it in data value side okay this is what the data variables data lakeside will use agile is nothing but so less agile fixed configuration highly agile configure and reconfigure as needed right see agility is nothing but you are splitting that entire work into a small small work okay small small work and then you are you are doing parallely you are doing parallel okay so all the work you are doing parallelly so agility in the data warehousing environment you can do very less okay but here in data lake environment we can go for high agility so agile process we have right so we will be we will be loading the data and there is no need for testing very less testing is enough and no there is no failover there is no support team much needed support team there is no much uh support team in our agility enough so we will develop the code we will put it in the production server and then it will be running so if any production issues are happening the developer itself will take care it&amp;#39;s again depends on the team only but i&amp;#39;m telling so agility is nothing but in most of the areas we are implementing the agility in our data lake environment is gc but in data various environment is less such agility only running a deployment in sequence okay splitting the modules into small small modules and then we are having a separate separate team okay that is possible in data lake environment okay this is what agility if you are going for a real-time project so definitely these data vargas roles will be there okay business users so business usage is nothing but your product owner okay your application owner your business owner or your business analyst data analyst somebody else who knows the business they&amp;#39;re going to give business use cases the requirement to us okay so their business users they are telling that okay we need to create a report in this format this way we need to have a report the data is there in source okay do atl pipeline load the data into a data vargos and take the reports from there that&amp;#39;s what the business analyst will analyze it they will they will give the information okay these kind of reports i need for this kind of report so these tables should be loaded from the source that&amp;#39;s what they will tell you so what the report writers will tell okay report writers will create a data model that the data primary foreign relationship and everything they will say like okay if i want to create the report like this i need to have the tables and data values okay i need to have the data in the data vargos these are all the data should be there so report writers then our etl developers okay i will take the data from source i will take the data from source i will load it into a data vargos here based on the data model the data model will be created by data modeling team so if it is a new business definitely data modeling team they will suggest us okay go for this particular data model you load it into this table they will give us the data model we are going to load it report writers they will take the data from here to a report and then they will be creating a report dbs dba will maintain the database okay database database environment okay here all this database this oltp olap everybody is taking in addition to this we&amp;#39;ll be having a report admin tableau admin rpa redmin informatica admin okay everything admin will be there and we&amp;#39;ll be having a testing team also report testing etl testing team will be there okay data modeler so they&amp;#39;ll be using you heard about this airbnb ervin data modeling tool so this tool they will use irvine tool for the data modeling we do have some different data model i will explain that that&amp;#39;s what they will use it okay this is what the different rules and if you go to this uh in real time these are all the popular etl tools are available in the real-time project so you&amp;#39;ll be having informatica power center so informatica is the company name our center is atl tool okay informatica is a company name we have informatica id cube informatica data quality mdm okay master data management information big data management informatica acs informatica intelligent cloud services so different tools are there power exchange address structure lot of tools are there so by using this informatica power center we can do the etl process same way competitors of informatica is nothing but all our remaining tools are competitors like ibm data stage and microsoft ssis sql server integration service so sap bods business object data services ibm cognos data manager cognos is the reporting tool cognos data manager is the etl tool oracle data integration is the oda it&amp;#39;s the etl tool sas data integration studio oracle vargos builder avenue talent apache nifi pentago explaining so explain this cloud-based etl2 see these are all the etl tool if you know any one of the etl tool so learning other etl tool is very easy the same concept you will use even if you are going for cloud big data or rainy cloud data load right etl pipeline same pipeline only we will use instead of this database we&amp;#39;ll be having a data data lake environment that&amp;#39;s a connection and everything will be different but loading process is same okay if you are approaching oracle oracle for the entire data data process in your company they will say go for oracle as the oltp okay article is the oltp oracle vargos and then use word oda okay oracle data integrator is the etl tool and here use oba see obi is nothing but oracle business intelligence so this tool you can use it for the reporting that they will tell if you are going for microsoft sql server they will use sql servers database and data values use ssis is the etl tool and then use power ba as the reporting tool same way ibm so you can go for intes or data virus okay ibm db2 for source system so then you&amp;#39;ll be having a data stage for etl so they will say all the related to ibm product so that&amp;#39;s what see the companies will choose which tool they need to go for so out of all the tools informatica is leading retail tool it will handle huge volume of data support and failover mechanism and everything will be very high here okay all tolerance everything will behave so same way reporting tool you have tableau you have power bi you have click view or click sense cognos sapo okay business object zoom data and google looker we have ssrs sql server reporting service oracle business intelligence spot for your micro strategy a quick insight see all these are reporting tools so we need to know at least what are the different reporting tools are there why because if you are going for etl developer environment you will be working with these department these two teams only okay you&amp;#39;ll be giving the data for tableau timon so if they are finding any discrepancy on the data so they will be contacting the etlt model okay so this is what enterprise data teams will be there so i will show you the data model if you look at here so what is uh conceptual logical physical data model who will create this so normally this will be created maintained by data modeling team normally the year wind urban tool they are maintaining right so first of all how they will maintain this is us just uh give you an overall idea just i will give you an overall idea so first they will go for conceptual data model see what is the conceptual data model so they will take like a pen and paper and then they will have okay in order to create a data model for the particular company so i&amp;#39;ll be going like this product table store table sales table time table see all this how it is related to a fact table here see includes the important entities entities is nothing but table under the relationship how we are relating with each other no attribute is specified attribute means the column names no primary key foreign keys specified here okay just the conceptual design model they will maintain all them how many tables are needed for this data model okay finance data model and then sales data model so different different merchant data model something for that we how many tables are needed that we will maintain it first conceptual data model okay it&amp;#39;s very highly abstract and you can easily understand so we will not have any uh software tool to create a conceptual model just dependent paper is enough okay and then we&amp;#39;ll go for the logical model in the logical model if you look at here all the tables we will maintain the column names okay in time what all the columns should be maintained in product table what are the columns to be maintained in sales table what all the columns should be maintained and what is the primary key foreign key so here store id is the foreign key from where it is coming from here it is coming store id is a primary key product id is the foreign key it&amp;#39;s coming from this table date key then after that you are maintaining two columns okay so normalization occurs at this level normalization means we are we can split this table into two tables also okay so that&amp;#39;s what your logical model so we are maintaining a logical level then after that we will go in for the physical data model what is physical data model see after creating like this we will maintain each and every column we will maintain the length see what is the data type what is the length what is the all the models we will maintain and then by using the erbin tool we will create the model right after that no need to create by ourself create table table name all this so if you go to the web model if you are saying that create all the tables it will create a tone stretch in the back end no need to create manually see all the relation relationship which table should be loaded first sorry which table to be created first okay that will automatically created based on the primary foreign key relationship we are not going to create if you are creating first of all we are to create primary key tables then only foreign key tables right if you are trying to create foreign key table first we will get the error why because this is where we are having this foreign key here so that cannot be created but ervin tool will create automatically based on that clear so data type and the data column column name and everything is based on the database we&amp;#39;ll be creating here this is called physical data model so who will do all this by using the urban tool or power designer the data model team will create it data modeling team just to give you an idea i have explained this see what up we know already this is also one of the interview questions they will ask see volapq is nothing but multi-dimensional data okay if you look at here multi-dimensional data so hold up what is nothing but based on the multi-dimensional data model so enables a user to easily and selectively extract and view the data from different point of views so decision making the business process okay see it&amp;#39;s nothing but cube is for our reference only okay so this cube we can see this in different view right so top view side view front view all the view right the rear view and everything bottoms you everything so one cube we are seeing in different way okay product supplier category and everything so look at here see we are going to have this table model in this particular cube say category we are having the three categories so this is one category confections like this will be having this categories category dimension okay beaver edges so all this we are maintaining in the category dimension okay if i want to analyze in this category of data so from this dimension you will analyze it we&amp;#39;ll go for one more dimensions okay here this is like confections so this one another dimension this is another dimension this data will be showing only for convection data so now you can go for see this is one one dimensions each and each and everything is to each and every dimension in the cube okay so now you can see this this one we have seen but now if you look at here this is like time dimension january february march april so this is category dimension so each category we have one two three same way here we are the time dimension january february march april like this right are keyways so q is everything will be there the same way we&amp;#39;ll be having a measures in stock on order sales amount sales cost these are all the measurable values that we are that we are seeing in different dimensions see if i&amp;#39;m showing one value here this is for one category for one particular one particular month for one particular sales amount one particular measure right multiple dimensions are related so you can see data in multiple dimension weight this is called a dimensional modeling olap cube okay if you look at here i am showing this way so this is the data say for an example i want to know the january data so what will you do so i have to sum up all this right for january february i have to sum up all this march summer okay so here i want to go for one particular category then i have to go like this one particular category okay this is what i will be analyzing the data in different different dimensions okay the same way here i have the book category the product category quotas q one q two q three q four on territory okay so we are differentiated so if you are showing some value how it will be like you&amp;#39;ll be having the different operations like four operations these are all the questions they will be asking in interview what is roll up drill down slice and dice pivot operation everything okay so roll up is nothing but summarize the data see how the q1 q2 q3 and q4 if i&amp;#39;m asking the earwise data you&amp;#39;ll be summing up all the queue right that&amp;#39;s what roll up drill down means you have the q1 data so q1 i want to go for month wise so you are drilling down queue into month month into week week into date that&amp;#39;s called drill drum okay slice and dice instead of having all the dimensional data we are taking slice on slice and dice from that so pivot operation is converting rows into columns if you look at here roll up operation is nothing but roll up is nothing but you are summing up okay we have the data like this see the other data product table you have the pc book shoe and clothes all the data you are seeing only for q1 okay q1 q1 this part pc and this is for one particular locations this is for new jersey los angeles and this is for australia regions okay these two are us these two are australia so now we are split we are summing up see this new jersey and las angels we are summing up as 2000 okay this is called roll up and this path and then sydney we are rolling up okay for this one see 395 six six not five that&amp;#39;s a thousand was saying here right so that&amp;#39;s called australian region data so this is called roll up so you can roll up here also so q1 q2 q3 q4 you can roll up to a year level data yeah yearly sales so roll up on location roll up on time roll up on category so that we got we can do it are you clear what is roll up so roll up is nothing but summing up drill down drill down is nothing but again here here the q1 right 605 that q1 we are splitting into january february march so 605 so everything will be maintained like a table level so in the table level data like uh on date level they will be maintained so we are splitting the table into monthly quarterly yearly weekly all this will be taking so if you are summing up this 150 100 and 150 it will become 400 so what is that 400 so we used to have the 400 year right this 400 for clothes clothes for q1 400 this is what we&amp;#39;ll be having okay clear on this what is drill down okay slice slice is nothing but one dimension we are taking so out of all this right we are taking only one new jersey las angel&amp;#39;s birth and sydney we are taking for pc and category only so this is all we are not taking only one we are taking okay six zero five this one only one dimension we are taking okay slice and dice dice means this is what dies same like dice we have right we are taking one particular dice here okay sydney and perth for q1 and q2 for this two categories we are taking large cube into small cube the pivot operations for summing up some values okay we are going for the pivot operations like we are splitting the rows into columns columns and rows for different analysis so this is what the different uh roll up and uh cube and all so we are the old app is nothing but types of volume see we have the roll up on app olap and others okay roll up is nothing but relational volume relational database management system olap that&amp;#39;s what we have multi-dimensional roll-up so normally it will be multi-dimensional like only hybrid it&amp;#39;s the other hold up so these are all something like data modeling team they will take care we are not going to do but you have to understand what is this they will ask so do you know what is roller something you know okay so just you can tell them what it is this is what the older cube and everything [Music] so what is architecture what are the different tools will be there you will be having a server okay in informatica you&amp;#39;ll be having a server here so this is what my server informatica server it will be there in one single place so if you are going to implement informatica power center you will be having an informatica server so power center server you&amp;#39;ll be having it so where to be there so based on the company&amp;#39;s dation if you are going for a us project then server will be there in us only one single server it&amp;#39;s a developments of different sat r uq will be different uh the the production will be different server but via the server okay so on top of unix or linux only it will be running in real time the security results this server will be running on unix r index server what this server will do it needs to take the data from source read the data from the source it&amp;#39;s a heterolineal sources load the data into data variables or target or anything so it&amp;#39;s sourced to target it should load the data so here we are fetching the data from here okay from here we are switching the data and then loading the data into data values so informatica we need to inform right okay you go to this particular table which is there in the oracle database you load the data into a teradata database data table on this particular entire data we will call it as a database say for an example if you assume that this is the error data how many schemas are there to schema in this era data instead of saying it&amp;#39;s a schema we will call it as database one database two like that we will say say just i&amp;#39;m making like a it&amp;#39;s not database administrator uh you can make the database one and database two something like that okay we will call it as it&amp;#39;s a database database tbo or database the same way here we will call it as in oracle it&amp;#39;s called a schema this is another schema this is another schema see what informatica needs to do informatica needs to read the data from source always we will read the data from source we will not we will not disturb any data in the source source okay we will not change any data in the source just we will read it select operation will know okay what we need to do we need to write the data into target so here we will disturb the data we will insert the data if it is new sync data we will update it the target okay this is what we will do in our informatica okay we have to tell that server right see if you want to this is post you assume that this is the post the post you will be having a two address correct you&amp;#39;re having the two orders here you&amp;#39;ll be having the from address see from address to address we have to give it to the postman then only he will take the data this post from from address to two address okay this is called metadata from address to address is the metadata inside the cover you&amp;#39;ll be having actual data correct see what is the message you have to pass that will be there in your cover so that is what informatica will read the data from this table this is actual data load the data into this table but which table we have to read so thousands of tables will be there in the source slide which table should be loaded for this particular business use case you go to the schema schema to an oracle database by using this user id and password see how you log into the database you will see the data you&amp;#39;ll be having a username password based on the username password it will be having the privileges either you have the read privileges right privileged service you will go to this informatica will have his own username password that is called service id okay by using the service ide getting this database username and password it will go and connect to this database and then read the data so that information so you go to this particular database you take this table you go and load this data to a dbo on the teradata that information we will store it in the repository the repository is nothing but the from address to address the address should be stored what is the actual data actual data is there in the this table so what is metadata this is what the metadata this metadata will be stored in repository clear repository that is what we need to create a repository also it&amp;#39;s a db only database only okay you may get confused here also we&amp;#39;re saying db here also we are saying db again then what is the difference i told right this is a data actual data you have to load customer data to a customer data table in the teradata this is metadata information okay you informatica you go to this particular database it&amp;#39;s oracle you go and pick this data from this table load it into a teradata to this table that information is called as metadata the actual customer data will be there right in this table that is called a data actual data so while running that informatica to load the data so this is called repository it will have all the metadata metadata is nothing but data about data so from address to address all this right that is called a metadata okay this is also another db see informatica server will run on top of one repository only database only that&amp;#39;s why we are to first install repository on top of the repository we should install the informatica server then we should on top of this we should be installing the client so why client so we have to develop right based on our development only informatica will do that see as a developer i i&amp;#39;m going to work on this client tool only on the client tool how many client tools will be there based on the number of developers see we are all working on the informatica project you assume that for for everybody we&amp;#39;ll have one server in our system we&amp;#39;ll have a client so for all the members we&amp;#39;ll have we&amp;#39;ll be having the this particular client tool capacity manager designer workflow manager and monitor so this client tool i will explain later don&amp;#39;t worry this client tool i will explain later you assume that by using this client tool only we are going to develop a business logic and we are going to save it in the repository when we are going to run it this rapacity will give information to a server okay through repository services this is called repository services repository services is nothing but from this repository services only it will pass the information to server okay server you go to this particular table you read this data load it into teradata in this table that&amp;#39;s what information it will give you so server will do will have one services called integration service by using the integration service it&amp;#39;s like a servant okay by using this integration service this is called is it&amp;#39;s for data movement okay data movement we will use the integration service integration service will move the data to from to target that is your social target okay so it will have three different sorry three different servant that is your thread what is the three different thread right we have threads is called reader thread so reader thread reader thread will read the data from source transformation thread will transform the data okay that is what information right writer thread will write the data into target so this is the architecture of informatica power center when we are going to run this informatica server through client repository information repository will give the information to server okay you go to this one you pick this and then pull the data it&amp;#39;s like a postman where the developer will come into picture here the developer will come into picture etl developers here only the etl developers sitting here then they will do all the etl mapping then they will do it etl admin they&amp;#39;ll be coming picture here informatica server okay this is like who will create this table oracle this article who will create informatica server will create whenever we are doing the mapping right it will maintain informatica will maintain so whenever going to save that informatica will make an entry in the table saying that okay you go to this particular table you pick this data you do this transformation logic and load it into the target table that will will be it will be maintained in the oracle trip okay uh you assume that this is the etl process okay it&amp;#39;s a i&amp;#39;m saying you have you have different water resources okay you have different water resources you have more one okay you have some uh build okay you have some mill you have some lake you have some river see it&amp;#39;s a different kind of water resource right different kind of art resource i have to take water from these water resources and and i have to send it to a consumer the the people sitting there there in different locations can i take directly all the time from this water resources and then send it to them no right what we will do we will have some our tank here right we&amp;#39;ll have some over tank so we have to do the pipeline first you do the pipeline all the water resources will not have the waters on same time so this this one we have to do one one day in ones this one two two times in the day we can do this is like every uh four times five times we can do but this is like all the time we can do but somewhere we cannot have it something like that some different different scenario will be the right so here we are the motor here we are the motor and electricity we have given so we are on start button so we are doing the this one so this is called data arrows this is called heterogeneous sources we are doing some ro process okay ro process is nothing but your transformation logic okay so we are transforming all the different kind of sources into a single format our water is single format right single format loading into a over tank from there we are sending to the different consumers reporting team or tableau or powerbi or somebody else okay they cannot go and hit this directly to the water resource they will take it from the over tank only this process is called as pipeline right same way here we will call this etl pipeline either it is water pipeline the same we will call it as etl pipeline same meaning only now etl process cut it this is what will happen informatica will read the data from source and target right so that logic you are going to do in informatica server so informatica what is that informatica will do we are going to do the transformation logic we are not writing any queries uh add our queries here just we will write the transformation logic here it will read the data and load it into the target so this is claimed informatica client okay this is informatica power center client i have to the server i will connect it now through my username and password i will connect it has been connected to a server see you are also working in the project i am also working in a project same project multiple developers will be there right all the developers will connect to the server okay then they will do the etl logic say for an example i am working on this particular project i will join into this folder you are working in different folder to the different folder you open it and then do it see this logic we have to do it right where we will do the informatica client once you have done it save this if you are saving it it will be saved in repository okay repository it will be saved in as a metadata will it run will it move the data now no i&amp;#39;m going to run the workflow i will explain this okay you will get to knowledge see this is all the client tool then where the server server is there on single one place in our system in our machine it has a server okay services that is what before starting informatica you have to start the informatica service it is running right so this is the server server running back end server we have connected through client we have applied now it has been moved the data from source to target is it moved see four partitions see all the data is moving from source to target once you are executing it then only to remove the data so this is called a client i told right so designer workflow manager and monitor and repository manager okay server is the the server will do the server operations it will read the data from source database and move the data client cannot read the data from source that the client cannot read okay no it cannot read only server informatica server can read the data from here it will have the integration service it will initiate the integration service okay you go there read the data from here load the data here that&amp;#39;s all as an etl developer i need to do the mapping logic informatica client store it in the repository while running it it will give the information to the repository capacity will give information to server server will read the data to turn source and target when we are going to do this one we will come to know see this one is called repository services i told right repository services rep and this is called integration service is the integration service integration service will come into picture only on the data movement repository services will come into the while we are connecting it will be stored in the repository whether i have i have created this mapping now no right it has been created as some days before where it has been stored this logic in repository metadata so we have by using the informatica client we have created that metadata we are stored in the repository now i just connected through client i have seen that and i am executing it so this time whatever the logic i have already created that will go to the informatica and then it will pull the data from source to target in this table on this flat file you have 11 columns you have four columns these are all the column names you have to read from here you have to do joining up these two tables based on the some id columns okay department department id you get the information load it into the target table this table just we are telling that right that is what as a developer we will do it we have developed it we will save it while we are running it informatica server will have the integration service okay that integration service you have the three threads reader thread you go and read the data transformation thread you transform the data as per the logic as per the logic we are joining it right this is the logic actually okay load the data into target table so the table information should be there for informatica this table structure and everything should be there for informatica then only informatica can load the data [Music] hello friends welcome to nikki academy in our today&amp;#39;s video we are going to see how to install informatica power center 10.4 in our windows mission in order to install informatica 10.4 in our machine we have to follow these nine steps if you are following these nine step you can successfully install informatica power center in your machine for the informatica installation what is the system requirement so we need to have 50 gb of space in the hard disk and we need to have windows 8 r10 r11 64-bit os 64-bit os and we need to have 6 gb of ram minimum 6 gb of ram or 8 gb of ram so if you have 4 gb of ram we can have it like a 4 gb of ram but this will be working with very slow if you have 4 gb of ram you can upgrade with 6 gb or 12 gb or 8gb so you can check the slot based on that you can go to the service center and then you can increase the gb and if you have any antivirus as of now for the installation you can disable the antivirus why because antivirus software will remove the exe file of informatica so that&amp;#39;s why any of our installation will get disturbed so no need to have any antivirus software if the antivirus software has been expired you can uninstall and then you can try with informatica installation if you have windows 7 64-bit os yeah very well we can try with the 10.2 version so this is the requirement so let us start our installation as a first step we have to download informatica 10.4 software from oracle e delivery site or i have already downloaded and given the link in the descriptions you can download from the description itself once we have downloaded we will be having these folders oracle 11g 64-bit edition sql developer and we do have informatica server unclaimed we are using oracle 11g for our repository as well as source and target database i will explain the setup how we are maintaining the source target and repository so how do we create tables and everything so that you can also follow the same step okay consider this is the article 11g software this is a database we have so in so once you have installed the software it has inbuilt the user is called system okay inbuilt user is called a system that is a schema we have and inbuilt we have one more schema called hr so this schema we will create here and we will create one more schema for our informatica repository so that schema we will call it as info info user we will create and we will create one more schema for the target okay so we will keep so this database for source and target so this is the system schema info info schema for informatica repository hr schema for source and core schema for target we don&amp;#39;t have any other database here so that is the reason we are going to use source and target on the same database if you have any other database you can make article 11g as the source and sql server or db2 or teradata are the target database okay so in system schema we have metadata tables all the the oracle metadata tables will be there so you can see here oracle metadata tables will be there so this is the metadata tables so here we we do have metadata is nothing but all the dictionary tables will be there metadata tables dictionary tables available in the system schema on informat info schema so we are going to create the schema for schema is nothing but subset of our database for the easy access and privileges we are creating the multiple schemas so informatica schema we are going to use it for informatica repository so we are going to create informatica opacity to hold informatica related metadata tables so then we have the source source and target this hr schema we are going to make it as source and this course schema we are going to make it as a target through informatica we are going to pull the data from hr schema tables to core schema tables so this is the informatica tool we are bringing here in hr schema we do have so here we have the tables like employees locations departments lot of tables are there okay departments here you can create any tables like p underscore employees like target employees you can create a customer table you can create any tables here so product table any tables you can create so we are going to make this schema tables as source this schema table says target so through informatica we are going to load it so why do we need this in first schema to have informatica metadata repository so this is the rip we are going to create it here so informatica repository we are going to create so in this particular schema so that is the reason we have created in first schema by default system schema on hr schema will be available this informatica info schema and core schema we will create it so in order to query this database we are going to have the tool called sql developer so sql developer is an application development tool so this tool we are going to so we have downloaded already so you can download if not in download you can download so first we have to install oracle software then we have to configure this sql developer from sql developer we have to give the connections you can query this database so that this is sql developer just an interfacing tool gui tool graphical user interface so what data is available on the database we can query from the sql developer we have to use oracle 11g for informatic installation we do have latest versions like 19cr21c or 18c 12 cnl but we will though this this particular informatica version will work on article 11g only we can go ahead and install the oracle 11g as a repository as well as database then in order to work with this database we are having the sql developer then this is the software like server and client say right click on this particular software once we have we have given straight click and extract to this particular folder if you do not have winrar in your machine go to google winrar download and go to this particular link and click on download winrar so this is the authorized software only no need to worry about it and click on download winrar it will get downloaded once downloaded the exe file you can click on it and then install it it&amp;#39;s very simple step since i have already installed i am not doing it again you can install it so first we are installing this in winrar or winzip if you have then you can use it right click on this go to extract you can extract all the four folders once extracted this is what the folder you will be having it so we have done with first two step and third step we will go and install oracle 11g for repository as well as source and target database so once extracted you&amp;#39;ll be having this particular folder double click on this folder click on this oracle this folder go to disk one double click on the setup file click on yes then it will be preparing to install this is a page you will be getting it this is the oracle database 11g express edition click on next accept the license and terms and conditions click on next year so here this is what the destination for where it should be installed click on next year no need to do anything so this is the admin password for the user sys our system so we have to carefully choose this so always make sure that you are remembering the password so enter password i am giving like admit password is admin confirm password assignment click on next click on install it will get installed it will take five to ten minutes of time depends on your system speed we will wait for it once installed then we&amp;#39;ll go for the next step once installation completed we&amp;#39;ll be getting this particular wizard click on finish we are done with these three steps we have downloaded the software extracted and install oracle 11g and we will configure the sql developer to connect with this database and we will create source and target connections go to this particular page sql developer so we can copy this sql developer into our c drive so it will not be disturbed whenever you are changing your files we can copy it here paste it here the c drive okay we have copied sql developer into c drive double click on this go to sql developer folder this is what sql developer right click on this we will make one shortcut to the desktop so send to desktop so you&amp;#39;ll be having a desktop icon if you look at here we&amp;#39;ll be having this shortcut double click on it so next time you can double click on this sql developer directly no need to go to the folder the sql developer will be opened if you are doing it first time you&amp;#39;ll be getting that yes or no click on no here this is what the first page you can see here and click on this plus symbol we will be creating a connection from sql developer to the underlying database oracle database the connection name just i&amp;#39;m giving like admin username is system so this is the system username administrator username password is admin so this is the password we have given while installing the software you can click on test sql developer is testing it is pinging the underlying database and we are getting that success this status should be success so this is the admin user by using this admin user we can control any other user so we have to remember this admin password so you can click on this plus symbol again we will make the connection for hr schema initially hr schema will be unlocked state just you can check it here so this is a username and password host name will be localhost port number is one five two one s id is xe nothing but express edition if you are testing it showing like account is locked so initially the hr schema account will be locked state we have to unlock it so how to unlock it from admin schema only we can unlock it you can go here we can mention that alter user user name identified by password password is hr account unlock just click on account unlock it will unlock then you can click on the plus symbol and use hr go to username hr password also hr click on test now the success click on connect so we have connected hr schema left hand side if you look at here some seven sample tables will be there you can make use of these seven tables for our practice okay now we will create one more schema called core we can go to create go to admin schema make sure that you are doing it creation everything user the admin schema create user core identified by core this is nothing but password this is the username this is the password just we are creating the one user schema and we have to grant the permission grant dba permission to core first time permission i am granting the all the permission to the core user from the admin schema i hope execute then click on the plus symbol you can make sure that giving the name a score and username will be core password will be core you can create any other user also no issues we can create any other users look at here this is what the admin user and we can create one more user for the informatica repository create user the repository i&amp;#39;m creating for info okay info user just i&amp;#39;m creating it grant dba to infor just we can execute this particular user and grant the dba permissions so this user this schema i&amp;#39;m going to use only for informatica repository and hr schema is for source and core schema is for our target okay we can have any other database also for target our source also that is up to our requirement we can use sources oracle target is sql server towards something different so we can have it different different source and target connections only in informatica we are going to define the connections so we can create one more connection like infar the username is called info and the password is called info just click on test should be success then click on connect this info and you core we have created so nothing will be there the tables nothing will be there and info also we do not have any tables so what i&amp;#39;m going to do i&amp;#39;m going to create in code i&amp;#39;m going to create a table okay normally we have to create like this in order to load the data i can go and create admin from admin i will create a table so create in this way create table t underscore core dot schema name dot p underscore implies t underscore implies nothing but target table i&amp;#39;m just creating a table in core schema to load the data from hr schema let&amp;#39;s select star from hr dot employees from employees table just i&amp;#39;m creating the table structure where one equal to two and just maintaining click on control enter one table has been created you can go and check this table in the core schema now one table will be there here so this table we are going to use it for our target so hr schema we do have employees table if you look at here select star from employees table i have a source table by using informatica i am going to load it into a this table we underscore employees okay we have created all the setup configured sql developer and everything we have done with post chip now we will go and install informatica power center server in order to install server go to this particular 2 of 2 folder part 2 you can see the server you can click on server here click on install so you&amp;#39;ll be having the install application file so this is hot fix no need to install it just click on this install double click on it click on yes here you&amp;#39;ll be getting this particular page install anywhere is preparing so this is the informatica server installation step you&amp;#39;ll be getting this particular page so we do have 12 step so this is the first step install informatica 10.4.1 agree the terms and conditions click on next year so this is the prerequisite for this particular informatic installation you need to have disk space is 50 gb and ram memory is 6 gb minimum if you have 4 gb ram then your system will be very slow we can install it but it&amp;#39;s very very very slow click on next here we need to provide the license key so where the license key available click on this three dots go to this particular one off to part one here we will be having this oracle america this key dot key file i have given this key file in the description you can download it and then you can keep the file and then you can choose it click on next so this is the fourth step the available space everything it will be displayed and then click on install here so this is the fifth step it will get installed it will take minimum half an hour to 45 minutes depends on your system speed so we will wait for all the step all the 100 percentage to be completed then we will go for the sixth step let it be installed so once 100 is completed we&amp;#39;ll be getting this particular fifth step no need to do anything here just click on next here we can allow access click on allow access and it will generate the key store so here so this is what the repository just we are creating it enter database information for the domain configuration repository so we are using database type as oracle and either we can use system schema or input schema just i&amp;#39;m using info so user id is info password is info you can go for system admin also then we can have the host name as localhost localhost port number is 1521 without space service name is xe express edition xe and then click on test connection the connection should be successful that means informatica is bringing the oracle database for the repository creations and again we are getting the connection successful click on ok click on next so here we have to give encryption key see encryption key we can give any key here i&amp;#39;m just giving administrator at one a caps administrator up to one just we have to give in this format click on next here click on ok and this is what the domain name host name and everything so informatica port number is 6005 and domain is domain domain password is this administrator only so we cannot copy it from here copy and paste it here we are to type from here so type it a caps so username is informatica username is administrator password also administrator a caps administrator we can make administrator administrator administrator we do not want to create any model repository so click on uncheck this click on next here so it will go and create an encryption key it will take some time [Music] we will wait for some more time and then we&amp;#39;ll see the next step in this eighth step no need to create any other new user just uncheck this option go for next it will again this step also will take some time in this step we have to provide we have to create repository service and integration service in order to create repository service we can give the use database user id we can go for info and password is info and database service name we can give x e and power center repository service we can give rep r so we can give any name just i&amp;#39;m giving like repository i&amp;#39;m creating like rep and integration service is just click on next so if you are getting this particular path no issues click on ok click on continue if you are getting this particular error now you should just click on continue once installation completed you&amp;#39;ll be getting this installation status access and you&amp;#39;ll be having this informatic administrator home page so just to click on this home page so this is admin home page we will check the informatica services and everything so click on this is https just to click on advanced click on proceed to desktop so this is the page you&amp;#39;ll be getting it see always informatica administrator pages username is administrator administrator password also administrator you can just copy and paste ctrl c ctrl v copy and paste click on login we can see here services and nodes go to the services and node we will check whether these two services are available click on this repository services rep it should be available the operating mode should be normal mode and is both should be available see this is also is should be available so this is a repository service this is integration service just to make sure that both of the services up and running okay both are up and running now no need to do anything if you are facing any issues please raise your questions through chat i will respond to it okay we have created the repository service and integration service we have configured this all the six step has been completed now we will install the informatica client seventh step go to the folder informatica client one of two go to this one you can extract this folder and go to this client folder go to this client again so here we&amp;#39;ll be finding the install so click on install click on yes here then this is the page it will get installed the client software will get installed easily we want to go for informatica 10.4.1 client just click on next year click on next so here we can choose power center client we want to have informatica data quality then use this informatica developer so just we are using power center client just click on this option then click on next next uninstall so it will get installed very quickly the same way server side you can click on now ok done then client will get installed so informatica client has been installed successfully so this is what the page you will be getting it click on done here so completely has been installed so we have completed all the seven step we have installed informatica server and declined and configured the capacity and integration service so we have done with all the installation we will configure the source and target connection then we will do one test mapping so that you can test it whether the source and target connection is correct and then we can move the data so as we discussed earlier sql developer is the tool the frontend tool and we do have uh oracle 11g as the database we will have this article levitg for source and target so here we have the hr schema so we will use this hr schema for source we do have seven sample tables so through informatica so we will make this hr source and core course target so consider in hr we do have the tables called employees so employees table we will make this employee stable source and in our core schema we have already created a table called t underscore employees if you haven&amp;#39;t created then create this table i have given the create table in in the description so this is the etl tool it will load the data from source to target so we can load either from table to table here the article itself or we can go for any other target also take the data from hr schema and load it into a other database also like sql server db2 through etl so only thing is the connection will be vary from informatica level okay so here we will make the hr schema source in the admin schema i have created the table like this create table table name and the course schema if you look at here i do not have any data so i have given the all the queries in the description you can also create it we will make this hr schema table source for schema tables target we will extract the data from employees and load the data into t underscore employees we will open the repository manager so go to search type repository manager the informatica repository manager will be opened this is a four claim tools we have we do have repository manager manager designer workflow manager and monitor so this is the four client tools are interconnected with each other so go to repository we will add the repository here this is the procedure we have to follow add the trip click on ok the capacity will be added here click on configure domain the domain so you have to click on this box and then domain domain name is domain only gateway host is localhost so here you can check this domain this is what the domain we have entered gateway host is localhost only localhost without space and port number is 6005 so for informatica the port number is 6005 just click on 6005 click on ok here then you&amp;#39;ll be getting the rip rip from here so it should be available here then we have connected correctly and right click on this click on connect this is the page you will be getting it so definitely you should hit the page with security domain is native if you are not getting this page then we are the services is not up and running go to services.msc if this now if this page is not available and go to services.msc check here informatica service is up and running see informatica 10.4 is up and running if it is not running then the left hand side will be having the start button started and it should be running so wait for it five to ten minutes then after that you can connect to the repository so here if it is not available just start it so wait for five to ten minutes then we can go to the here the username is administrator informatica username is administrator password you can copy and paste here click on connect it has been connected you can see here deployment group is available so it has been connected so we will create one folder for us go to folder create we can create the folders training click on ok only administrator will have access you can see here folder has been created successfully and we can go to the designer open the designer whatever the folder we have opened will be opened here automatically all the four clients are interconnected so you can watch my video the other videos on the client tool you will get all the stuff very clearly click on training open and here so this is the five client tools i have explained all this details which is available in my in the channel so you can go and check it here so what is source analyzer everything firstly we will define the source go to the source here import from database so first of all odbc connection will not be there we have to create it so click on this three dots will be opened here click on add go to this oracle wire protocol click on finish data source name we have to give oracle underscore src the oracle underscore src is nothing but the database connection name i&amp;#39;m just giving us oracle source host name will be localhost this is oracle details number will be one five two one and service name will be xe so click on ok here will be created oracle src same way we will create for one more connection for oracle tgt so click on here so you can go to and same way you can create the oracle tgt connection host name will be localhost port number will be 1521 service name will be xe click on ok here so both the connection has been created click on ok choose the source connection from here username will be hr tab click on tab automatically populate the owner name order name is nothing but schema name okay so honor name is nothing but schema name this is the schema hr schema click on hr so in the hr schema then click on connect it will populate all the tables and views in tables we have the table called employees so we do have seven tables just take the employees table click on ok here go to target the same way target here import from database choose the target connection and click on the username is core for target we are making like more click on connect here so target we have created only one table that table will be available here so choose that table target t underscore employees click on ok this is what the page will be getting it the target table has been imported now next step we will create a mapping go to mapping create a mapping m underscore test mapping i&amp;#39;m just creating as test mapping so source if you want to get what is source target and what is the source qualifier and everything please watch my channel so i have explained all this very clearly just drag and drop source here drag and drop target on the right hand side so always target should be on the right hand side keep the cursor here select all keep the hudson on the header drag and drop here all the columns so all the columns should be connected correctly then control yes this mapping should be valid so mapping should be valid so we have created a mapping then go to workflow in the workflow manager you can see here workflow manager this is our opacity this is integration service go to workflow designer the workflow designer we will create our flow but before creating a workflow we will create a connection go to this connection just informatica we are telling that by using this connection you connect go on hr schema a source go and connect this core schema as a target you have to provide all the username and password just go here click on connect relational connection no connection will be available see what is your source and target based on that we have to create a connection click on new so source is oracle right just we have we have to choose the oracle connection click on ok now oracle underscore src just giving the name user name will be hr password also hr and the connection string is xe express edition click on ok so in our case target also oracle database at core schema if you have target this sql server or db2 or teradata here we have to give whatever the new we are going to have here we have to choose it so target also oracle in our case just click on oracle click on ok here with the connection name just i&amp;#39;m going to make oracle underscore pgt click on core over here and connection string is xe and click on ok then two connections we have created that&amp;#39;s all if you want to have more connection just to create it here now go to the workflow designer create a workflow create here wf underscore workflow name is just i&amp;#39;m giving like test workflow just i&amp;#39;m creating a test workflow it should not have any iphone just click on it should have only underscore test workflow click on ok here and click on the session icon keep the icon here then choose the test mapping click on ok then click on the link task link it double click on the mapping we have to provide the connections in order to run the informatica workflow we have to run the connections go to mapping the fourth tab click on source source should be pointing to source connection source is always a reader since database it&amp;#39;s a relational reader target is always writer subject target also oracle relational writer so here we are to bind to the oracle tgt connection click on this arrow mark choose this tgt and go little bit down you can truncate and load so what is in the truncate and load we are truncating the data before loading the data into the target table apply okay control yes so this workflow also should be a valid one and you can see here before starting it i&amp;#39;ll show you the target table doesn&amp;#39;t have any data right doesn&amp;#39;t have any data so right click on the white space start workflow monitor will be opened automatically look at here monitor will be opened running you can click on the task view it got succeeded but here it got succeeded the task you open choose the left hand side the workflow name this is a session name session so this is the workflow this is a session task you can go for right click on the session session this is a session right click on the session get run properties you can go to the source and target statistics you can see here 107 record read from the source and return into the target so this is what there is no rejected record right you can go and check here so ctrl enter you&amp;#39;ll be seeing that all the data has been loaded if you check the count all the onenote summon record loaded into the course schema tables from hr schema through informatica we are loaded into this table so always make sure that this table we will create it so normally we have to create like a create table table name and all but we have created by using the admin schema so i have given all the script sql queries what we have executed in the admin schema in the description you can also follow the same and once you have completed everything this is a test mapping it&amp;#39;s running correctly then once you have completed everything right click on the repository disconnected okay this is what you are properly disconnecting it so this is what we are properly disconnecting it so you can disconnect all the four client tools correctly disconnected so you can close it under pass to here close it that&amp;#39;s all so repository manager also you can just a disconnected this is one time activity repository manager next time you will not open it directly you can go for designer for example tomorrow i want to start i want to work on this particular tool so directly you can open this sql developer from here and then you can work on but informatica we have to go to the services.msc as i told earlier just click on this informatica 10.4.1 we can start the informatica service from here since already it is running no need to start but make sure that it is running if you are starting now wait for wait for some more minute wait for five to ten minutes after starting it after ten minutes go to the designer and then double click on it double click on it you should be getting this particular page native the domain name should be a native that&amp;#39;s what has been connected clearly okay so this is how the setup file please so we have installed successfully you can watch my channel for the rest of the session you can practice it and then you can learn more on the informatica thank you for watching if you haven&amp;#39;t subscribed my channel please subscribe the channel and click on bell icon so that whenever i am uploading a new video it will be get it will be notified thank you all [Music] hello friends welcome to nick it academy i am chandra shaker i hope you have seen my previous installations video and you have installed the informatica power center successfully and let us begin our complete training so kindly watch the playlist as well as the description for more informations so let us start our session so informatica power center has four client tools the first one is repository manager designer workflow manager and workflow monitor if you look at here so i&amp;#39;m just selected the repository manager here so here we have the repository so repository manager designer workflow manager and monitor so these are all the four client tools available in informatica power center so these client tools are interconnected with each other so all these tools are connected with repository so first we can connect with repository so let us see all these playing tool one by one first tool repository manager so this repository manager will be denoted as r here so this is the repository manager and it is used to do all the admin related tasks so what are the admin related tasks so we can create a folder for our project we can provide some access and privileges and once the code has been developed we can export the xml or you can import the example so for an example you are getting the data example from production environment we have to do some modification of the existing work existing logic then you can import the production xml into a repository and then you can do so all this you can do by using this repository manager historical run status we can see by using this repository manager so this is a repository manager just i&amp;#39;m connecting so connect this repository manager i&amp;#39;m connecting with admin user so in our real-time project there will be a lot of lot of folders should be there and just i&amp;#39;m creating a folder for this so go to this folder you can create a folder so here i&amp;#39;m just creating a folder called nic it academy so all these sessions we are going to see in this folder only so just i have created so here i have created a folder so you can you can see here the folder has been created and it will not have anything as of now so this is the one thing we will do here after some time we will export the xml file import the xml file in this repository manager so that we will see a later point of time so these are all the four steps you can do the client the second client tool is designer the designer by using designer we can do three things one is source definition target definition and mapping the mapping logic source definition we can define all our sources the sources might be a flat file or it could be a relational database table or it will be a xml file or cobalt file at the same way the target we can define and mapping logic so normally if you are the developer we will be using this designer tool very much around 70 to 75 percentage you&amp;#39;ll be working on this designer tool only so all this business logic we will create by using this mapping designer okay we will see all this tool one by one so from here just i&amp;#39;m connecting to the designer you can see here the designer tool so it will be automatically so whenever you&amp;#39;re opening the folder here this the corresponding folder will be opened here automatically so at a time you can open only one folder here so you cannot open two folders at a time so if you if you want to open the second folder you can right click and open it so folder should be opened if you are closing it if you are just closing it disconnect it it will be disconnected here also it will be disconnected if you are disconnecting this workspace will not be available if you want to open so there is a difference between connect and open if you connect the folder will be connected but the workspace will not be opened so you can open the folder by using this open option so you can open it so this is the designer tool you can see here there are five component in the design tool by using this designer we can make three things one is source definition target definition and mapping so go to this designer if you look at here there are five components in this design so first component is source analyzer so by using the source analyzer we can define the source if you are clicking on the source here source analyzer the top you can see the source if you click on the second one target designer the top you can see the target then transformation developer so that i will explain one by one then map let designer mapping designer so what are the five components source analyzer target designer transformation developer map designer mapping designer so as a first time it will be little bit confused for you but after you are using this tool for extra two to three days so definitely these component names will be very easy for you by using the source analyzer we can define the source definition so source search might be relational database table or a flat file or cobalt file or xml file for example here i&amp;#39;m going to connect the sources source here so import from database or a file or import from cobalt file or xml file or you can create an instant source definition here but it should be a same structure of your either your file or it&amp;#39;s a database table or anything so here i&amp;#39;m going to connect with import from database i&amp;#39;m just connecting it this source table from hr schema i&amp;#39;m going to take the table called employees table any table you can take so consider this is the existing architecture so this is the database i have so in this database i have three schemas so one schema i have created repository so i have the same i have explained in our installation so here i&amp;#39;m having so this is a structure we are going to follow for entire project entire session so so you can see here so we have the hr schema so from here i&amp;#39;m going to take so from here schema i&amp;#39;m going to take the table called employees table so from this table from employees table i&amp;#39;m going to take the data and i&amp;#39;m going to load the data into t underscore employees table so it&amp;#39;s a table to table note just for for the explanation i&amp;#39;m going to explain so how to load the data from table to table by using this informatica client component you can see here so source analyzer i have created the source and i&amp;#39;m going to the the second component is target designer by using target you can define the target here you can go to the import from database and see here the target might be anything so i&amp;#39;m just connecting target table for the explanation i i just connected one table t underscore implies just i have taken this this t underscore employees table from the core schema so this is the first component source analyzer second component is target designer the third component is transformation developer so what is this transformation developer you can see here these are all the transformations by using this transformation we can apply a lot of business logic so this is the update strategy expression filter and aggregator the same whatever we are doing in sql the same logic you can do by using these transformations in informatica the transformation developer is nothing but a single transformation if you want to do the reusable logic then you can create the reusable transformation here so this is the very early stage to understand this reusable logic but try to understand so going forward i will be explaining this reusable transformation very clearly if the single transformation can be reused in multiple mapping so that will be called as reusable transformation so here in project we can use this reusable transformation encryption decryption time conversion date conversion some conversion logic all this right so most of the projects most of the developers are using the same logic in this case we can one developer can create the transformations by using this transformation developer and then others can just drag and drop this like function in oracle the second one the fourth component is maple designer map plate descender is nothing but so maple designer also is used to create a reusable component but here in transformation developer a single transformation you can use but in so for an example that the reusable logic you can create by combining two to three transformations then you have to create in create it in maple designer the maple designer is nothing but a set of so here i have explained everything so i have posted this document in the descriptions so you can take this document transformation developer and map the designer transformation developer is nothing but it&amp;#39;s also reusable component a single transformation can be reused and it can be reused in multiple mappings and maple designer is nothing but so this is the maple designer a set of transformations can be reused in multiple mappings so this is also reusable component the fifth one is called mapping designer so this is the mapping designer by using this mapping designer we can create the logic in mapping designer you can see here left hand side the source and target definitions and everything then you can go here you can go to the mapping and create so m underscore employs details just i&amp;#39;m creating this employee&amp;#39;s details just to drag and drop this one employees to mapping so you&amp;#39;ll be having this source qualifier source qualifier is nothing but the source qualifier converts the source data type into informatica native data type so that is the functionality of source qualifier this concept we will see in the later sessions okay so i have just i drag and drop the target so this is called a mapping you can just drag and drop this one this is a table to table load you can see here this is called pipeline so if you have if you have connected wrongly you can select here you can delete and again you can connect it just you can drag and drop this one so just it will be connected you can save it should be valid so i will be explaining this transformation developer and map designer in later sessions in detail we will have separate session for transformation developer and maple maple designer in that session i will clearly explain okay so these are all the five components which are available in designer level so just i have explained here so now i can go to the workflow manager click on workflow manager the workflow manager will be opened so workflow manager is used to create a session and the connections and workflows in workflow manager three components are there task developer worklet designer and workflow designer this task developer is a reusable component so the in designer in designer we have seen transformation developer and mapper designer here single transformation can be reused the same way in task developer we can create a single reusable session so that i will explain in later session but just remember this by using this task developer we can create a single reusable session we can provide the connection also in task developer but if i want to create a logic a reusable logic by combining two or three sessions then i can go with worklet for an example in the project you can have some control table nodes so after loading the target tables you want to load some control tables it&amp;#39;s a common for all the projects so you can the one person has created already the same worklet you can use it so any of the developers can use it and this is called worklet designer so this task developer and workload designer is used to create a reusable component the third one is called workflow designer so here we can create a workflow go to workflow here just i&amp;#39;m creating a workflow look at here so this is what workflow we of underscore implies so just i&amp;#39;m creating a workflow the start button will be created and you can see here the session so here also you can create by using this task and create so you can create a task but here we have to enter the name but here we have a shortcut for this under session you can click on here you&amp;#39;ll be having the associated you have to select the associative mapping so since we have created only one mapping so only one mapping will be available you can select here you can click on this link task and you can link it you can see here lot of connections we need to provide the connections here you can see the connections can click on here we can provide by using this connections we can provide relational connections the file transfer protocol connections and application connections and loader something like interior data we have some loader fast load or multi-load or something loaded that connection we can create and the front-end application connections we want to create that application connections we can create so mostly we used to create relational connection if it is a development environment you can create a connection but in real time if it is a production environment you have to ask the informatic admin team to create a connection they should have already created you have to use that connections but normally we will have a parameterized connection so that i will tell you after some time for a security reasons we&amp;#39;ll be creating a parameterized connection so that is one that connection only we are we are using so not like this hard coded connection we will use normally parameterized connection so that will be passed through parameter file that concept we will see in later point of time just we have created a workflow you can save it so this is what the component in the three components in the workflow manager okay next one the monitor the fourth client tool is called monitor by using this workflow monitor you can do you can run the workflow you can see whether it is running or succeeded or failed or aborted or stopped so anything you can see the status it&amp;#39;s just a monitoring tool you can see the row count source and target row count how many recorded it read from the source and return to the target so that you can see here and throughput is nothing but number of record per second so per second how many records it has been read and how many record it has been returned into the target that you can see here and you can see the session log so session log is very much important in our real-time environment so whenever anything happened you can check the session log okay so these four steps you can see here in monitor so whenever you are running this workflow the monitor will be automatically open you can see here so remember whenever you are creating a workflow remember whenever you are creating a workflow that says the mapping should be valid so then only you can able to create a session and then you can able to save this workflow as valid one so here it is showing like start workflow has been started you can see here it is running so you can see whether it is running or it is aborted or it has failed anything so either you can go for gun chart or task view so it is showing like failed so we have to check why it has been failed as i mentioned here we have to go here get session log so you can see get session log you can press e here so that is what it will tell you it tells you that table or view does not exist where i am getting the error so that is on the right thread so reader thread it has written it has it has read the data you can see here read the data here from the source but while writing it is getting failed so you can check why it has been failed can go here so it is showing that session is failed but the workflow the parent the parent of workflow is showing like failed sorry succeeded it should not be the real time cases you have to go here you have to enable one option called the failed parent this if this task fails fail parent fail parent if this task fails if this task fails fail a parent this one workflow you have to fail it okay and also you have to check this option go to config object you can make this stop on errors equal to one so that two options so every every sessions we will give these two options whenever if any one error occurs it will stop the workflow so i can provide the control s it&amp;#39;s a valid now just we have pointed out can the the because of the connection it is failing and go to the mapping you can check the source and target connection source is always a relational reader you can see a relational reader so you can provide the connections here it is pointing to hr schema you can see your hr schema and password is available in target just you can connect to the target and you can provide the connections here run key target table option then apply okay here so should i give trunk a target a table option for in in for every sessions no need if you want to truncate before loading the table then you can do so again i&amp;#39;m just starting you can see so just i intentionally i did it to explain you the session log and you can see here it is running it will succeed now so right click get run properties you can see here the source and target statistics so here it will be showing like how many record has read from the source and then it has written the target and there is no rejected record throughput is nothing but record per second one not rec 1.7 record and here also not sound record so there is a low error you can go and check in the database table so here is a table the target schema so this is the core schema then you can check here so it has been loaded so this is a target table you can see here the data has been loaded into the table and it has one or seven records count the rows it has 1.7 records so this is the so four client tools in the informatica level you can see here so these steps you have to follow whenever you are creating a workflow these steps you have to follow the first one you have to do the source definition so that is what we have did source definition we have did first one second one target definition so these three things first one source and target and mapping these three things you have to do in designer so that is what we have done source and target and mapping then fourth one is the session we&amp;#39;ll be creating a session if you want to create a session reusable session then create a reusable session and task developer then here it will be available so we have just created non-reusable session so what is reusable what is non-reusable that i will explain in later sessions we have provided the connections by pointing to source and target database and then we just created a workflow and we have executed it and monitor it so remember these are the naming conventions we have to follow the mapping names should always have m underscore the lowercase m underscore table name or a meaningful name a session name should be yes underscore the mapping name it should be yes underscore m underscore the meaningful name and workflow should be w k f r w f underscore so this is what the workflow name mapplet is mplt rml worklet is wklt or wl dimension table should be underscore dim or dim underscore or d underscore fact table should be fact underscore f underscore view should be table name underscore b or v underscore table name materialist use table name underscore mv so these are all the naming conventions these naming convention will be slightly different from project to project so but this is the standard one you can follow in your project also kindly practice this session we will meet in the next session welcome to the informatica power center tutorial session in this session we will see the flat file to table note watch the complete session without skipping and subscribe the channel for more updates as well as look at the playlist for all the videos happy learning in our previous session we have seen how to load the data from a relational database table to a table so that we have already completed so if you haven&amp;#39;t seen our previous session so kindly check in our playlist it will be there you can practice and then you can practice this this current session so there are different types of data load so first load we have seen already the table to table so we have one table in hr schema that is called employees then we have one table we have created in course schema that t underscore implies just we have loaded the data from a table to table here so now we are going to read the data from a flat file and going to load the data into a target schema so you assume that our customer our third party vendor our customer is placing the file the secured directory so from the directory we have to read the data and then we have to load the data into a table so file to table load then next session we will see a table to flat file load so current session we will see this flat file to table load we will create on file so you can execute this whatever the select statement you want to have just you can execute and export the data the format will be you can mention that csv format the first option will be csv and you have to save this file in a folder so where i have to save this file so you have to save this file in this particular source file directory you can go to c drive informatica 10.2.0 server infrashare and this is the common path this is the root path you can see here we have the backup files bad files cache files lookup files and session log directories and source file directories and all these directories will be there in real time so this c drive will not be there but from here server in fashion it will be available you can go to the src source file directory here we are going to keep the source files so i&amp;#39;m keeping all this file inside one folder and here i&amp;#39;m going to create a file so just go here and go to the file you can see i&amp;#39;m going to name it the file s this is the file name i&amp;#39;m going to have so click on browse it you can provide this name click on next and go to the finish you can see here the file should be created here so this file we are going to take a source now so since here we are creating the file i&amp;#39;m just opening the file you can see here the data will be there so everything the data will be a comma separated file you can see here so from this file i&amp;#39;m going to read the data so first data i&amp;#39;m going to make as nic it academy so just i&amp;#39;m making that just i want to show you this data is this particular session is reading the informatica is reading the data from this particular file so this is the source file and the file name is source underscore flat file underscore implies so from this file i&amp;#39;m going to read through informatica and load it into the t underscore employees table so this table this table i&amp;#39;m going to load so go to informatica now so we have already seen how to so what are the different components are there in informatica just you can go here source analyzer if already source definition is there you can right click you can clear so don&amp;#39;t delete any source or target definition if you want to delete it you have to delete it will be deleted if you are deleting it it will be deleted from all the mappings wherever it has been used so before deleting it be careful you can go to the sources so this time we are going to take the data from a file so you can give the second option import from file so so you need to have the so select the file type as all files then the file will be available you can select the file and click on ok all the information i have given in the descriptions you can follow the description also you can go to the you can check here the file type the file type is delimited file are fixed with the file so this is the delimiter file so sometimes customer will place a fixed width file also each and every column will be a fixed width so that we will see later point of time but now we will we are taking the delimited file you can mention this delimited file and if the file contains a header you can enable this option so we are informing informatica that okay the first line should be a header so you can check this option then informatica understands first lines first line is a header then go to next one so here you have to provide the com here you have to provide the delimiters the delimiter here is comma so if any other delimiter is there you can mention this delimiter in the other option so since here we are using comma delimited file you can mention the comma if you look at the difference if i&amp;#39;m enabling the comma so automatically the fields are separated we can go to next since we have double quotes in the file it is automatically taking the text qualifier as double quotes you do have seen here the double quotes is there right so this is a text qualifier it is automatically taking informatica is automatically predicting that double quotes so it is removing the double quotes here you can go to next you can provide all the fields data type so this is very important step here the first one employee id you can provide the employee id is a numeric and it&amp;#39;s a three digit if you are thinking in in future the length as precision may vary then you can provide here the maximum length and precision you can give here and first name it&amp;#39;s a text so if you are thinking in future it may differ more than 14 then you can provide now itself then here the last name is a text email also is a text and this is a phone number so this is what we will do lot of mistakes here so kindly make a note of it since i have two dots i have to mention this is a text field if you forget to mention this text field then session will fail then next one is date column so date column should be very careful you have to give so this is a date and time the format you have to note down so what is the format dd iphone mon iphone y so this format we have to give in the properties i will show you where to give the next column is job id it&amp;#39;s a text column you can give and salary set numeric column the next one is commission percentage so that is also numeric only so you have to provide numeric and the next one is manager id also numeric department id also numeric so normally this source definition this ddl definition will be provided by the client so we have to get clear the requirement so what are the maximum length of the column that we have to provide here so finish finish it then you can see here the source file definition has been created you can see here the file the source definition name and the type of this source flat file the target since we have already taken this t underscore employees i&amp;#39;m not going to import once again and go to the mapping the third step you can follow here so go to mapping the third step is creating a mapping go to mapping designer mapping create a new mapping m underscore yes underscore source underscore flat file underscore implies just i&amp;#39;m creating a mapping and go to the mapping here mapping has been created just drag and drop your source from the flat file so you can drag and drop the source from here and target from here right hand side you can drag and drop source always should be in the left hand side target should be here in the right hand side you can just click on select all you can connect it control yes the mapping should be valid and if you want to arrange it you can arrange it here arrange all it will be arranged clearly or arrange all iconic it will be there on iconic so if you have lot of transformations you can put easily to understand you can put iconic r can go for arrange on so it will be there this way so before creating a session it should be valid so it&amp;#39;s as up now it&amp;#39;s valid you can go here we forget to provide the source properties date and time go to the properties here you can see here date and time just you have to provide dd iphone mon iphone y so this format whatever the format the source data format that you have to give here apply okay save it again it will be updated you can create the workflow and everything so go to workflow manager create workflow wf underscore source underscore flat file underscore employees the naming convention just i&amp;#39;m giving to understand what is this you can create a session by clicking on the shortcut icon session icon so here look at here so here you can click on the session icon you can keep it here so informatica will ask which is the mapping you want to associate then i&amp;#39;m just selecting this mapping source flat file click on ok so click on link task and connect it double click on it go to mapping source now it&amp;#39;s a reader it&amp;#39;s a file reader so file reader will not have any connections here so it gets the file so you&amp;#39;ll be having a path so this is the source file directory so here it is mentioning like dollar pm source file directory so this file is a parameterized now so this is nothing but this file path is nothing but so this is the path so where it will be defined in the informatica admin console it will be defined so you can check here admin console if you open so there it will be mentioned like this is equal to this one so that is why informatica is understanding that okay this path is nothing but this path and then source file name you can put the file name if you want to create any folder and then you want to keep your your own file over there your project file over there then you can create the folder here so that folder you have to mention here for example here one is there if you want to keep the file in one then you have to put the folder name is one one so here so that is what the changes you have to do but i&amp;#39;m not doing it here directly i have kept the file in source file directory source file name you have to mention then go to target the target you can see here the connect the target table is pointing to source so this is what we will do lot of mistake here your session is getting failed because of connection error so you can table our view does not exist that particular error you can see whether you have pointing the target table correctly or not you can go to this particular table you can bind to the target and you can go little bit down you can do truncate target table option it will truncate the existing data and it will load freshly i&amp;#39;m just starting the workflow so it will be running here so i have just started our flow it is running you can see here get run properties you can see the source and target statistics it should read the data from a flight file to table so this is the flat file and table 107 record has been loaded so we can go to the target table you can see here so this data has been loaded in the table so this is what you can practice so kindly practice the session so in the next session we will see different concept thank you for watching kindly practice the session and we will meet in the next session with another topic thank you for watching welcome in this session we are going to see table to flat file load so in our previous session we have see we have seen table to table load and the flat file to table load so now we will see the table to flat file load consider we have the table here in hr schema in the employees table we are going to fetch the data from here and we are going to generate the flat file so we will do lot of processing and then we are doing it in the we are loading it in the flat file so how to take the data from a table and how to load the data into flat file that we will see now go to designer the designer we can see first source analyzer the source will be a table and we have already taken this table so no need to take again so we can consider this table to this table to flat fire we are already table here only to take again in target and go to the target target designer just we will create it how to create so i can go to the target we can use the file structure if i have any file structure we can import but since we are creating a target file we will not have this file structure here you can use create and you can provide that flat file and you can mention the target flat file name and create but you have to provide each and every columns but instead of that since the file type is in this format t underscore employees table format just then selecting this table ctrl c ctrl v just to make a cup i&amp;#39;m making a copy it&amp;#39;s asking do you want to copy this target yes i have to give yes since the target table name is already available we are getting a conflict you can see here you have to give rename and you can apply here edit and you can provide this is the flat file target underscore flag file underscore employees just start giving the name of this target click on next finish it you can see here the one of the target has been created if you look at this this is also a table only oracle database only you can see here it&amp;#39;s a table can i convert table definition into a file yes double click on that you can go to a database type instead of oracle you can mention flat file so click here flat file so you can use this flat file definition go to advanced what is the delimiter you want to have in the target you can provide the target definition delimiters and if you want any optional quotes that you can give click on ok apply okay here so we just created copied from this table structure and we have created a flat file so this is the oracle table so this is a flat file just to save it source and target has been created now go to mapping here i am creating a new mapping called m underscore target underscore flat file underscore employees just time providing the naming convention you can drag and drop the source the table employs table and target is a flat file you can keep the target in the right hand side and click on employ id select all the columns keep the cursor here drag and drop from here to here so it will be dragged and draw so you can if you have wrongly connected you can select the columns you can select the pipeline and you can reconnect again so ctrl s it should be valid the mapping should be valid before creating the session first three steps we have completed we can go to the workflow manager the workflow manager go to workflow designer create a new workflow create wf underscore target underscore flat file underscore employees so this is the employees so file name just i have given the workflow name you can click on the session you can select this particular mapping m underscore target underscore flat file underscore employees and click on ok then click on the link task you can connect it so double click on the session go to mapping you have to find the source to the source connections target is a flat file writer so target will not have any connection since it&amp;#39;s a file you have to provide the path so where it should be created you want header in the file yes if you want to have the header then you can provide output field names you can go little bit down you can see here so output file directory so this is the target file directory so this file directory is nothing but if you look at here so previously we have seen that source file directory now the same in first at the root directory we will have one more folder is called tgt files so this is the same structure we will use in our real time also i&amp;#39;m just deleting all the files from this folder i&amp;#39;m keeping the empty folder so it is not necessary to have empty folder so you even you can create one more directory here and then you can place your file so just i&amp;#39;m loading the file into this target directory so so this is the file directory here so dollar pm target file directory is nothing but so this is the one which we have already defined in the admin console so target file directory is equal to this one so this is what we have already that admin team will point at this particular parameter value to this particular path then informatica knows that it will decode that particular path okay so what is the file name you want instead of dot out i&amp;#39;m making the dot csv file just to make that csv apply here you can see the set file right so this is the file property you can see the same file property will be there delimited file go to advanced you can see the column delimiters so whatever the delimiters you want you can place this delimiter here click on ok just i&amp;#39;m not doing anything here cancelling it apply okay so click on yes so if i&amp;#39;m going to run this workflow it will create a file and then it will load the data into this particular file directory so just it&amp;#39;s running now you can see here the data the workflow is running you can click on run properties you can see the source and target statistics here so 107 record has been read from the source and returned into the target you can go and check the target file directory target has been created as a csv file you can right click and open the file and then you can see so this is the file it has been created so always the file will be created with hash symbol in the beginning so we can remove so how to remove this has symbol from the file that i will we will see in later point of time so just now if just we have seen that the file has been created you can see here the date and time format since in the source we have only the date we have given the date and time format here it is loading only zero zero here all the zero zeros but if you are thinking that i need only date the format we have to defend where we have to define the format the target so remember the target definition you cannot change it here so you can change here but the port you cannot change it here the properties you can change but both you cannot change go so always if you want to change anything the source source and target definition go to the source analyzer or target designer so here you can go to the properties or port you can change it so just i&amp;#39;m going to the properties i&amp;#39;m just changing it so what is the format i want i do not want this hours minute second just removing it i&amp;#39;ll have only mmddy by the way format so i&amp;#39;m going to save this just i&amp;#39;m going to run it no need to refresh it why because i haven&amp;#39;t changed anything on the mapping level just start workflow the workflow will be started it will replace the file so always it will replace the file it will overwrite the file you can see here it has been whether it has been completed or not it&amp;#39;s running it has been succeeded you can go here the file has been updated the timestamp has been modified and see here the date alone has been loaded now so this is what you can do so you can ask question if i want to append the data in the existing file then what i need to do so we have one option called the session level go to target definition so here you have one option called append if exist if the file is exist you can append the data in the existing file just click on this even you can create a directory here so that the directory is not present that you can enable it so this is the one you can enable it so just i&amp;#39;m not enabling just i&amp;#39;m giving that append if exist so click on it so how many records are there in the file so one not someone will be there after making that upend file how many we will check so here i have one not seven records including header okay just i&amp;#39;m closing it i&amp;#39;m going to after enabling the properties up and if exist the the workflow is still running it has been executed you can see here the data has been appended you can go little bit down so previously we were seeing only up to here we have seen so now again the same data has been appended into the file so 107 or not 7 you are getting that 214 including header 20215 so this is what you have to create the file in target table so if the customer is asking that the downstream application is asking different delimiters you can provide the delimiters in this session level so up to now we have seen table to table flat file to table and table to flat file load so kindly practice this session at least two or three mappings you have to create for each and every scenarios you have to practice so then only this tool will be much practiced this tool will be very easy to you and kindly practice it thank you for watching we will meet in the next session the transformations are two different and transformations are available one is active transformation passive transformation so first of all what is transformation right the transformation used to transform the data so from one format to another format it is used to transform say for an example if you take this is the transformation if you take this is a transformation see whatever we are doing by using sql or anything this transformation is used to transform the data from one format to another format see transformation you can give n number of input input port this is called input port will be having different number of ports here then you&amp;#39;ll be getting the output port here so what the transformation will do transformation will transform the data from one form to another format if you look at here this is the transformation so this is the transformation if you are giving 100 record to this transformation you are doing some filtering out or you are doing some rank or you are doing some aggregation or anything you are getting different number of records from here out of this transformation is changing the number of record if the transformation does change the number of record password pass through it then that is called an active transformation and also if the transformation does changes the row number of the record so first record it&amp;#39;s coming like a 10th record here but the number of record is same but the row number of the record is getting changed then you can go for the then you can call this particular transformation as an active transformation okay so this is an active active is nothing but so whenever you are having the transformation the transformation does change the number of record pass passed through it that is called an active transformation okay if the transformation does not change the number of record passed through it you are getting you are giving 100 records you are getting all the 100 records out of it see some transformation something like you are calculating salary or a salary increment or you are calculating some bonus or you are calculating for tax see this these calculations it will be applicable for all the record rate so whenever we are going for all the records then we have to go for the we can call this particular transformation as a passive transformation if you look at here this is what an active and passive active and passive a transformation does change the number of record fast through it then active a transformation does not change the number of record passed through it that is a passive okay and also we have one more uh difference like connected and unconnected unconnected transformation what is this connected and unconnected see connected means the transformation will be then pipeline so pipeline means it will be connection some transformations we have that will not be in pipeline something like a stored procedure or so different transformation we can go for lookup we can go for unconnected unconnected means it will not be there in pipeline okay simply it will be alone standalone it will be uh looking so this is called the connector connected means the transformation will be in pipeline okay that is called the a transformation will be in pipeline so this is called connector and unconnected means the transformation will not be in pipeline so separate it&amp;#39;s a standalone so that is what unconnected you can see here see we have we are going to see some 17 transformations 17 transformations so either it will be connected or it will be unconnected whether it be passive or active we will go with very first transformation is called source qualifier so what is the source qualifier transformation right the source qualifier transformation is nothing but it&amp;#39;s a very first transformation so this is the whenever you drag and drop a source into a mapping you&amp;#39;ll be getting one transformation called source qualifier so what is the main advantage of the source qualifier what is the primary task of source qualifier it converts source data type into informatica native data type that means your source might be your oracle sql server db2 flat file sales force or anything right the source might be from anything any different data sources see look here this is an oracle so oracle we have varchar2 right but in the sql server we don&amp;#39;t have any error in teradata we do not have anywhere to simply aware only so whatever the data type we have it converts that into a string here okay string number it will convert into a decimal date will it will convert into a date and time so i have given the explanation here you can go through it it converts source data type into informatica native data type source qualifier is an active and connected transformation it&amp;#39;s an active under connected transformation source qualifier has many properties if you look at here these are all the properties if you are not writing any properties you are not overriding any properties in source qualifier it takes all the record from the table with all the columns you how it will write the sql statement select all columns from the table name this is what it will fix select all the column names from the table name this is what it will fetch i will show you but we have many properties we have many properties you can see here the properties are like first properties are sql query then user define join see first one by one we will see the first property is like sql query if you double click on the source qualifier right you can find the properties here go to properties the properties you can see first one is sql query then user defined join source filter number of sorted port tracing level it is a common for all the transformations i will tell you what is this tracing level then distinct pre-sql post sql output is deterministic and you can see here output is repeatable so these are all the one you can see these properties are mainly used the first seven to eight properties are okay first i will go to the sql query what is this sql query see here if you if you don&amp;#39;t write anything on the source qualifier it will write select star from the table name star means all the columns column names it will mention so that&amp;#39;s why if your source team making more columns your informatica will not fail select all the columns from the table name that is whatever sql query we can overwrite say for an example in the source table normally it will have more data right not only one year data you have two years three years all the data will be there in the source but we have to take only whatever the data we want how can i take so that&amp;#39;s what we have to go with sql query so by writing sql query we can restrict the data from source itself by writing sql query we can restrict the data from source itself that&amp;#39;s what sql query okay next one is user defined join see here previously we haven&amp;#39;t joined anything just we have taken one table and loaded into a table but we can able to join but the thing is here you we have to join homogeneous child i will tell you what is this each and everything we will see clearly so that you can understand what is this user defined join means homogeneous joint homogeneous means tables from same database tables from same database that&amp;#39;s what homogeneous join then source filter source filter means for an example here you are applying some filter condition in the sql query so instead of that you can go for the source filter that filter condition you can write it here any filter condition look at here salary is not null something we are writing right so this is the so those who are getting salary it will be it will be picked up from the source country equal to india okay state as equal to active something like that if you want to put some filter conditions you can put the filter condition on the source filter select distinct it&amp;#39;s nothing but enable or disable that is a full row duplicate it will select say for an example in the source you have the full row duplicate see everything i will show you practically we will see then you can get to know it will remove the full row duplicate pre sql and post sql so what is the pre sql and post sql right pre sql means so this sql will be executed before this particular sql okay see this sql executed to fetch the data but before that i want to execute any sql then i can go for the pre sql post sql see after data fetch i can go for the first scale that means see you can do the index creation or you can do the correct stats so all this you can go for the pre-scale and post sql then number of started port number of sorted report means you can do the order by class now i will show you what is this okay if the source is flat file if the source is flat file all of the above properties are disabled okay remember this if the source is flat file all of these source qualifier properties are disabled except the tracing level why because if the source is flat file you cannot do anything see whatever you are writing here so everything is like yes sql query one sql go only that we will see now we will see one by one okay first of all we will execute the table we will execute one flow just i&amp;#39;m doing table to table look at here going for create m underscore source qualifier underscore transformation just by making the transformation i will take sources a table employees table t underscore employees table whether i&amp;#39;m doing any any changes on source qualifier no if you double click on this there is no changes just i have taken as it is control yes then it should be valid you can right click and go for arrange all right then go to the workflow manager you can go to the workflow manager clear create workflow wf underscore source qualifier underscore transformation we can create a source qualifier transformation then we can take the this particular mapping and connect it double click go to mapping source should be pointing to source target we are pointing to target just we are taking the data from source table to target table without any transformation after that we will do some some transmission logic okay i&amp;#39;ll go to the i&amp;#39;ll start the session it will be started and it will be loaded into the target table right it&amp;#39;s a target table it will write how many data it will write see whatever it is going to read from the source it will write you can see here or not seven record read from the source and one not seven record return into the target so okay now we will analyze the session log see session log analysis is very very important i will show you some other things in session log you can able to see three threads three thread what is the three thread so one is reader thread first one is reader thread second one is transformation thread see transformation thread reader thread it will read the data read the data from source transformation thread it will transform the data from one form to another format if you do not have any transformation logic then transformation thread will not do anything it will just pass the record then righter thread writer thread will write the data into a target table see wherever you are getting any error either read a thread or transformation thread are right at first so there you have to concentrate on it see if the source itself we are not able to get the data then there is an error in the source qualifier right so if you look at here it will show you the information and then timestamp node so which node it is running what is the thread it is running see whenever you are seeing the director or manager are mapping right these are all about the informatica level something you are seeing that reader threat reader so it&amp;#39;s a source side writer is nothing but target site then message you can able to see here the messages like so what is the date time format and everything it will take and if you go little bit down see whenever you are getting some time say please go through one by one all this dtm you know right dt means data transfer manager you can go little bit you can see all this right at that right and thread source side will be having the look at here target writer thread say insert into right insert into table name all the column names then values will be there values question mark question mark means run time it will it will load the values see if you want to copy this if you want to copy this you want to make it like a google or something you want to google it then you cannot copy from here here you cannot right click and copy from below you can copy it from below you can copy it okay if i want to copy below only how to copy okay let&amp;#39;s go little bit down you can see here the source qualifier query right sq instance means source qualifier instance yes q employees source qualifier query you can see you can copy this query so what is this query look at here what is this query read rr means relational reader sql query issue to the database first row returned from the database that means symformatica can able to read the data establish the connection right or not sound record right see this what you can able to see and uh what is that end the loading the target table it has been in depth and load summary it is showing how many they cut it from the source how many record return under the target everything you&amp;#39;ve been able to see then it will complete session completed without any issue you can go and check in the source qualifier the query so where you have to execute source qualification you have to execute it in source site you got executed correct so informatica will execute this query in the source qualifier the source database and whatever the result you are getting right so that result will be picked up from the source if you look at here select all the column names from table name from source table see how it will be how it is writing select table name dot column name table name dot column name see why it is it is mentioning this way right if you are writing this way the performance will be improved instead of mentioning like select column names along employee id first name alone if you are mentioning like table name dot column performance improved instead of writing star if you are mentioning the column names so that is also will improve the performance okay and also if the source team they&amp;#39;re adding more columns it will not impact on our informatica load see this is what okay it will fetch the how many recording fetch you can see how many recording will fetch all the one not sound records so this is what informatica will pitch the data from the source okay see instead of this i can go for somewhere class where i can write any class okay so where commission percentage is not null where commission percentage is not now that means who are all getting commission i&amp;#39;m picking up is not null not null means value should be there that is a not null see just i am writing the not null here okay okay so just time executing it out of 107 how many records you are getting i will go on execute on not seven you are getting 35 records okay see even out of uh those who are getting commission and i can go for some other some other condition also the salary should be more than seven eight thousand dollars and salary right you can put condition greater than 8 000 some condition see whatever out of 35 you are getting only 20 records see you can write this sql query based on your scenario based on our scenario you can write the sql query clear on this so now what i will do i will select this query normally this is what we will do we will write the query in sql that is what uh source sql source database and then go to this informatica you can double click and go to properties here i am writing this sql query okay okay apply okay save then you can refresh the session right click refresh mapping and execute it now informatica will fetch only 20 records from the source see look here instead of 107 it will pick only 20 records see whatever the query we are writing execute the query in the source quality source database and then if it is executing correctly then place it here that&amp;#39;s what we will do okay okay look at here it has been executed only 20 records it has taken okay that is what it will go on right into the target table so i&amp;#39;m right i&amp;#39;m saying that if this is the record should be loaded then you can take say for an example you have employees data employees table a particular company it has all the employees who are who left the company or they they went for some medical url you have taken all the records and instead of that you want to take only active employee status record see whatever the condition whatever the query we are writing all the source qualifier properties will be available in session level also if you go to here go here the source qualifier you can see all the properties sql query whatever we have written over there it will be available here i&amp;#39;m overriding this instead of 8 000 i&amp;#39;m writing 9000 i&amp;#39;m overwriting this or 10 000. okay so session level i have written one query but mapping level i have written on query so now you you guess it which which one it will take any idea informatica anyhow it should take only one right so informatica will give reference to session level properties remember this i&amp;#39;m going to run it now it will take different records so that&amp;#39;s why in real time and all if you go and check sometimes what will happen production production database sorry production server we used to take the we we will check it the queries and then we are getting this particular record only and then we are going to check the this query this query here we were seeing some different properties here but the query is running it&amp;#39;s different if you go and check why it is having different they should have written something different in session level so what will happen some new see look at now that only taking eight records right that is what it will take sometimes what will happen in real time under we may face some issues see a center in fix we have to apply status equal to active some some filter condition we have to apply if i want to do any changes in production it should be go with releases releases change liquid change ticket all this right so we cannot change anything in the production asset is so it should go with all the cycles uh development testing and pre-prod and then approval everything it should even one small properties also it should go with but it will take a lot of time your business is waiting for the particular fix to be applied conditions should be applied over there so what we will do we will raise one request we will get the approval from business saying that this is an issue to fix it immediately we will go for session level override we will give that information to our business owners system owners you will get the approval they will say like go ahead once we get the go ahead then we will go to the informatica support team we will assign it to them and ask them to change the properties in session level normally okay so normally we will ask them to change okay put some condition here uh some filter condition you apply source qualifier you change it something like this because of this it is causing an issue something like that we will apply will tell them they will immediately they will apply here they do they do have the access right write access then they will change it they will save it then they will run it okay then they will ask next time you go with the change request see real time always check whether we have something like a different logic in session level okay so now i will remove this i will remove i will apply i will save i will save here if i&amp;#39;m going to run it how many record it will read from the source so it is taking all the record it is not taking 20 records why because 20 records previously you are seeing 20 only after that we have changed the query in session level we have removed it why it is taking 107 now can be able to guess it if there is no query then what informatica will think it will take all the records correct select star from the table name that&amp;#39;s why it is taking 197 okay remember this these are all interview questions they will ask okay so now i what i will do i will revert it back to the previous one that 8 000. now it will take i have remitted okay now it will take this is called source qualification so what i will do i will remove the source qualifier for you even you can write whatever the queries you want to have you can ask me can i write something like a giant query yes we can write but whatever the column you are taking from the join query so that column should be there out of this sql out of the source qualifier that column should be there say for example department name you are taking the sql query that department name should be there from here okay so that is why we will take a source qualifier query and what i will do now i&amp;#39;ll go for simply instead of this much query it&amp;#39;s very simple condition right simple conditions then that condition i can write here source qualifier filter source filter i will apply something like job id say for an example job id equal to sa underscore sales representative okay job id equal to srf i will write apply okay save and i want to refresh it refresh mapping control yes so you will check it whether it has been that source qualifier query is there or not so there is no query right okay now i&amp;#39;m going to execute it you can see here so 30 records now if you go and check the session log so informatica will write that session the query okay you can see here from employees where employee dot job id is here we haven&amp;#39;t returned this query we have written only the filter condition informatica will take the filter condition it will write the sql query so this is called sql source qualifier filter source filter so you can write any filters the same way you can try to overwrite the filter condition on session level also okay now i am not writing any filter conditions just simply removing it i&amp;#39;ll go for the number of sorted port what is the number of sorted vote means say for an example i am writing one one means first column what is the first column employee id see while taking the data from the source based on the employee it will start it order by on employee id order by on employee that&amp;#39;s what it will take two means employee id comma first name not second column alone if you are writing number of sorted put is two it is one comma two three means one comma two comma three that is an order by class i have explained order by class in our sql please go through that sql order by class okay if you look at here it will take same data only but it will be on the order by if you go and check the session log you can find the order by class you can get session log you can see here there is an order by class right you get session log you can see here it will write order by class okay so from employees order by employees.employee okay employee tooltip why we are writing the same order by so in order to improve the performance we&amp;#39;ll go for this order by class from the source itself it is order by then its performance will be improved that&amp;#39;s why see i i am writing 2 here number of sorted vote is 2 it will not go for second column alone it will go for employee id comma first name see in this case in this case our real in this scenario since employee id is primary key even if you are making 1 comma 2 that is same only why why because first column itself it is a unique column unique values right it will not go for the second one if you look at here i will write i will show you in the the source level so that you can able to understand select star from employees order by where per order by okay order by one one means first column this is employee see here i&amp;#39;m making two two means second column alone i&amp;#39;m adding so it will make second column but informatica how it will write like see if you are making two one comma two it will make one comma two means first column if you are getting first column duplicate value then it will go for the second column since it will not have duplicate value it will not go to the second column okay informatica if you are making like number of sorted for this two it will write one comma two only look at here i&amp;#39;ll show you get session log you can see here look at here order by employee id comma first name that is the weight will write do not write order by first name alone clear on this what what is mean by number of sorted port next one is called user defined join can i get a data from other table as well yes you can get it but if you want to join it by using the source qualifier both table should be on the same database either from the oracle or from the sql server so only one you can go for n number of tables not an issue but all the tables in same database same database okay all the tables in same database look at here i will go to the employees table look at here i have the employees table here right see this one i will take employee stream i want to know which department she is working for which department she is working for nancy its department so department id 100 do you know which department it is if i want to know department id what you have to do you have to look up say how we will do the vlookup in vlookup in the excel right the same way we are to do the giants here giants senior joints departments 100 is finance department where it is located 1700 thousand seven hundred where i can check okay so department is finance department located at thousand seven hundred thousand seven hundred which location i have to check it in the locations table 1700 location is so washington us so that is what you have to do joining okay first of all you will join this employees table and departments table for each and every employee we will load the department name we&amp;#39;ll go here so how to do the join say for example this is what our employees table employs table department&amp;#39;s table employees table department table both are in oracle so that&amp;#39;s very important okay both are in oracle both are in sql server i either both are in an article or a sql server or teradata or anything but everything it should be a same database it&amp;#39;s a department&amp;#39;s table department so what is the table here is an employees tables employees teams okay the condition is both should be on same database okay so now you can take the column names like employee id so you have all the employee id i i hope you should have already gone through this employees table you have what is the columns different columns are there department up to department you have in real time also we have to analyze that table first whenever we are going for any development we have to analyze analyze it all the tables what are the different columns are there primary key foreign name department underscore name and location id location id all this okay here i have the target table here i have the target table see this table you need to create it you want to create it i have given the syntax for this table creation look at here i have given the syntax by using this you can able to create the table okay i have added two more columns department name and location id in the existing table like from this t underscore employees table i just copied i have created like t underscore employees underscore department there i have added two more columns so i will this is the table name target table so this is the target table this table contains these columns like employee id first name last name uh some different background so i am taking all this basic concept for the joining okay see this is what the columns i have so for each and every employee i want to find out which department name they are working which which location they are from location id alone i have to bring these two tables column into one single table see here you&amp;#39;ll be having one source qualifier right for this you will be having one source qualifier see remember whenever you are going to do source qualified china so how many tables you are joining no no not an issue you can have one single source qualifier so you can remove all other source qualifiers you can remove all other source qualifier you can have only one source qualifier common for whatever the number of sources you are using so even you are using three three sources four sources even sometimes you use ten different sources so stable but everything should be at same database remember this this is called source qualifier right in this source qualifier you can join it first of all these columns from first table then department id and department name and location need these columns only so what you have to write simply you have to write the join condition join condition is employees dot department id department underscore ide equal to okay departments dot department id so this is only simply you have to write departments dot department head you have to write only join condition not a query it will write informatica will write that query even if you want to write a query you can write in words sql override that source qualifier override you can write but no need to write simply if it is a giant condition or filter conditions you can write it here say for an example i want to load only finance department what you will do this is joint condition this is joint condition after that you have to write filter condition you have to load filter finance department in the filter condition you can write any filter condition you whatever the condition you want say for an example i want to write department name equal to finance department finance department so that you can write it see the above one is called join condition the below one is called filter condition you are filtering out see only the employees who are working in finance department will be loaded department name location id department id say department id you can take it from here also clear on this okay i will go to the mapping i will create it now you will get to know so m underscore source qualifier underscore joiner first table what is the first table employees table so what is the second table department table is that department stable now no i have to go here first department stability controller you can connect departments table this time you have to take the buttons table also at the department&amp;#39;s table this is the one you can it so departments table also see this is the source the source side so what is the connection here it is a database level connection okay database level connection not in informatica level so database level this employees table department&amp;#39;s table is there connected with department id employee and all okay then target target which table i have to take see don&amp;#39;t delete any of the source or target tables don&amp;#39;t delete it so instead you can clear it clear all clear all means it will be cleared from this workspace it will be available on the left hand side okay don&amp;#39;t delete it if you are deleting it it will delete it will get deleted wherever it has been used in mapping okay how many mappings you have used already it will get automatically from there it will become invalid mapping clear on this okay see all the mappings are now it&amp;#39;s well right valid if you have deleted then it this will become invalid that is one concept okay we have we have taken the source two sources but we need to have a target go to target bring the target table here core and you can take target t underscore employees underscore department i can take the second table second table department drag and drop so this is the department table see this blue color is different this blue color is different this blue color is atl pipeline that is informatica pipeline this blue color is database level connections okay so we can have only one source qualifier the second source qualifier you can&amp;#39;t remove it delete it so you can have this source qualifier common for both this source qualifier will have department ide from here also you will be having the department id department id department name location id only these are three columns i&amp;#39;m taking i do not want this manager id no need to take it department id okay you can ask me so here itself you have the department id why you are taking one more department id column right so these two columns only we are going to join it department id and department right so that&amp;#39;s right double click go to properties the properties user defined join only write the conditions what are the conditions so just click on it double click on the department id click on ok employees.department equal to department start department id click on ok apply okay and then you can go with the target see here target employee to department id you can take it from here so no need to take the department id one more time you can make department name and location id from here and take it control yes it&amp;#39;s a valid now okay it is a valid noun so this is what you can go for the source qualifier joiner joiner double click go to mapping so it should be source target should be pointing to target truncate and load i will truncate the target table and load it start workflow it will load based on the join it will make inner join it will make inner join okay i will show you what how to do a left outer right outer randall but it will make in the join inner join left outer right and outer door if you want to get clear idea so please watch the sessions on joints i think it&amp;#39;s the fourth session or fifth session i have clearly explained what is giants why do we need this giant soldiers okay got succeeded you are seeing only one or six record so that means one record not loaded why because some some giant condition is getting filtered out okay here you are having it here underscore dept you have the department name and location that you got loaded for an example i want to do left outer join left out design means all the record from the left table matching record from the right table so how will you write left outer join here simply this is an oracle table right either you can write sql query or if it is left outer join the right hand side you can make plus symbol on the bracket like this if you are making plus symbol on the right hand side table then it will become a left outer join okay so if you if you want to know how it is please watch the sql sessions sql join session it is called a left outer join okay so i&amp;#39;ll go here i will refresh the mapping control yes start workflow you can see here it will read from the source itself it will read one not sound records okay see here it has taken one not sound record you can see here there is a null value on the right hand side see null value on the right hand side this record will not be there in the previous one okay null there is no matches that is why inner join it didn&amp;#39;t take but now it is taking okay clear on this i can apply some filter condition also i can go here apply some filter condition source filter see what is the filter department name equal to some department name i can put i can put department name so i can put department name is purchasing [Music] okay purchasing department click on ok ok save purchasing department it will make the giants see without join you cannot you cannot say how many of the employees from purchasing right without join can you say can you load a purchasing department alone employees alone to the target no joins only you can able to load it that&amp;#39;s why here you can do the joints okay so look at here it&amp;#39;s running this time it will load only purchasing departments only six records six employees are from purchasing department without joining it you cannot able to understand you cannot able to find it so that&amp;#39;s why it is called source qualifier joiner okay clear on this so what is it this select distinct what is select distinct say for an example here i have one table called select star from employee slide select star from employees underscore one employees underscore one this is same table empty like same same like employees table but i have added one two records with duplicate value look at here this is one record one record this column is different last name column is the front this will not be taken as duplicate but here it will be taken as duplicate full row duplicate okay this is full row duplicate this is not a full duplicate so if you are selecting distinct informatica will remove the full draw duplicate only this one only okay if you look at here look at here this one source qualifier transformation i will go to the source qualifier transformation here what is a table we are fetching the data employees table correct employees table if you go a little bit down the last you can see one option called source table name source table name here the source table name i&amp;#39;m overwriting it employs underscore one do you know which table it will take now data whether it will take employees table or employees one employs one but that mapping we have created for employees right mapping it has been created for employees what is the predefined conditions the condition is all the columns should be matching with employees see employees one and employees one sorry employees and employees one should be matching see now i will take this employees underscore one if i&amp;#39;m taking that employees underscore one year it will read the data from employees underscore one table only source table here i have written some different table it will not take from employees table it will take from employees underscore one clear on this okay how many cards it will read one not nine record so one not nine we have right or not nine so one not nine record it will read from the source i have to apply some distinct if i&amp;#39;m going to apply distinct 0.9 if i&amp;#39;m going to apply distinct here select distinct how many record it will read or not eight why you are not eight this is not full row duplicate this only full draw duplicate see from the duplicate it will remove one record you will get one not eight record even target also this is target see instead of loading into a t underscore implies i want to load it in 30 underscore employees under history table you can mention the table name here sometimes they would have mentioned that table name differently here okay some history table they will load it see they will take a mapping session and everything copied source qualifier query they will write some different queries target table they will change it here that&amp;#39;s all then it will go and load it into the history table that is what they will do normally see pre-sql post sql before reading the data you can create an index after reading the data you can remove the index that we can go for three sql post skill so this one you leave it i will teach you separately okay but other properties you practice it now it is fetching how many records one not eight instead of or not nine it will remove full draw duplicate hello friends welcome to nik iit academy this is chandra shaker welcome to the informatica power center tutorial session let us move forward and learn filter transformation today watch the complete session without skipping and subscribe the channel for more updates as well as look at the playlist for all the videos happy learning filter transformation is an active transformation and a connected one so why we are calling it as active right so your filter transformation it will filter the number of record pass to pass through it so it will it will filter out right so we are giving 100 records we are making some filter condition then it will filter out so source qualifier itself via the filter so this filter transformation source qualifier filter is used to filter out the record from the source itself only source after source anywhere in the pipeline you want to filter out then you have to use filter transformation filter transformation is used to filter out filter out the record anywhere in the pipeline so that is what anywhere in the pipeline if you want to use it you can go for the filter transmission so it&amp;#39;s very simple transformation it is used to filter out the record that&amp;#39;s all for an example this is the filter condition the properties you&amp;#39;ll be having the filter condition by default it will have the condition like true true means it will pass all the record from the filter okay it will pass all the record from the filter if it is false means it will block all the record okay it will block all the records it will not pass it the condition if you are making any condition the condition passes only the condition satisfied record so that alone it will pass then it will unsatisfy the condition unsatisfied record it will block so that is the use of filter transformation for an example you have salary column you are the salary column you have salary like 10 000 50 000 null value you have 8 80 000 6 000 12 000 i have so i&amp;#39;m making one condition like not of isn&amp;#39;t love salary so what the filter will do filter will pass only these salaries and it will block the null values why because we are making not of is nala is means it will pass only this record null value but not optional it&amp;#39;s opposite so that is the one and also so why do we need this filter so anywhere in the pipeline if you want to filter out the record you can go for the filter transformation you assume that i have a flat file and it has some record okay it has some record with all the countries okay i have i want to filter out i want i want to load so india&amp;#39;s data alone okay india&amp;#39;s data to the target if it is a flat file can i use source qualifier no if it is a flat file you cannot use source qualifier filter why because for the flat file source called source qualifier properties are disabled one you cannot use it so that is the reason we are using source and source qualifier you cannot do anything on the source qualifier you are going with filter transformation in the filter you can make country equal to india under saturday equal to 50 000 something some condition you can make it if the country is coming like india this india and this india something like three different format it&amp;#39;s coming you can make the source qualifier filter something like lower off you can use like this lawyer of country equal to india so you are making that uh lower country and you are making the lower case understandably greater than 50 000 something like that you can make it it will pass only that record now we will see practically so what is the use of source qualifier what is the use of filter transformation so i will go to the folder here go to the folder i can use the source is a flat file i can use the sources of flat file just we have already taken this flat file this employs flat file and a target i can use this t underscore employees table flat file to table i am going to the mapping create a mapping m underscore filter underscore transformation just i&amp;#39;m creating the filter transformation so taking this flat file and taking this target but so in the flat file if you look at here the flat file it has some the commission percentage is another column it has some null values and normal values those who are getting commission those employees only i have to pass it to the next level this target table those who are getting commission that means this column should not contain any null value so that record alone i have to pass it can i use source qualifier properties here no i cannot use it why because source is a flat field if it is a flat file all the properties are disabled one all the source qualifier properties are disabled one but i have to filter out how can i filter i can go with filter transformation so this is called filter transformation you can click on the filter transformation you can keep the filter here drag and drop the all the columns to the filter transformation here you can double click on it go to the properties you can see the filter condition right true means it will pass all the record but we are we we need to filter out right so what is the condition you can go to the fourth so you can make is so you can make is null of commission percentage that means it&amp;#39;s commission percentage means those who are getting those who are not getting commission it will be passed i have to put negate negate function that is not not of personal law of commission percentage you can ask question how can i find what all the different functions are there you can go to the functions okay lot of functions are there all functions are character functions right you have all the character functions you are the instinct substring l trim altering all these are like same like oracle single row functions right by using this function you can do any uh logic here okay any filter conditions here so you have that data cleansing operation conversion function then you have the date function so all the date functions are there so different functions will be there numeric functions are there right all the numeric functions and you can go to all function you can see all the functions by using these functions i have taken this null of right so you can see this number of state of spaces of islam right this one i have taken so below itself you can see the explanation is null of expression as expression so that is what you can see if condition so everything will be there by using this you can derive your logic okay validate it it should be valid click on ok go to apply ok then we have to complete the pipeline i can select all this drag and drop itself like columns are matching with the target columns just to save this it should be valid it should be valid mapping should be valid then only you can able to create the session i can directly go to the workflow manager click on the workflow manager you can go to close this existing one you can see here go to workflows create one wf underscore filter underscore transformation this filter transformation right you can create a workflow this is a session icon click on this you can choose the mapping corresponding mapping click on ok the session will be created click on the link task link with start button double click on it you can enable this properties right fail parent if this task fails that means so this property we will enable for all the sessions why because if the session is failing corresponding the parent or blue also will get failed automatically and go to the mapping you can find the source source is the file reader now so we are reading the data from a file that is a file reader for the file you will not have any connections you&amp;#39;ll be having a path so in this path we have to keep the file right source underscore flat file underscore uh implies underscore csv so target you can check target is relational writer you can provide the connection here you can go to down you can if you want you can truncate target table and you can give and before that we will check the file okay so since we have taken the file definition we will check the file this is the source file directory right so dollar pm source file directory this directory is nothing but so this is the source file directory i&amp;#39;m going with source file this is the file i can open the file you can check here what is there okay here we have used to deliver is some different delimiters so dollar hash and percentage this is a delimiter we are using so same delimiter we will check it here whether the same delimiter is available source we have seen this concept in our file to table load itself so that is why i&amp;#39;m taking directly here delimited file advanced go to column delimiter you can put this dollar hash percentage you have to enable this treat multiple delimiters and why because these three delimiter we are making a single delimiter so that is why this option we have to enable first line we are skipping it this concept we have seen already in the flat file to table load just i&amp;#39;m giving that and they can go to the config object make this top on error equal to one so what is this top on error equal to one right if any one error occurs it will fail it will fail the session it will not proceed so click on ok it should be valid save it it should be valid now i&amp;#39;m going to start it so right click on the white space anywhere in the white space you can click on start workflow it will start it now the workflow will get started now automatically the monitor will open here so this is the filter transformation you can go here and go to the source and target statistics you can see only 35 record has been loaded so 107 record right from the source actually and then it is filtering for an example the same filter condition if you have applied on the source qualifier if it is a source is a table then you&amp;#39;ll be having statistics like 35 and 35 why because informatica will filter out record from the source itself since it is a table but here since it is a file it will not filter out it will fetch all the record into informatica then in the informatica cache memory it will filter out the 35 and will it will be fetching that if you go and check in the database you can see you can open this core for course schema so just i&amp;#39;m connecting to the core schema you can check whether it has been loaded with those who are getting commission that is what we have applied the filter condition right so that i&amp;#39;m taking so go to select star from t underscore implies this table i&amp;#39;m checking if you check here in the employees table you&amp;#39;ll be having those who are getting commission it will be loaded here okay those who are getting commission even you can put one more condition also like those who are getting commission and salary should be more than 10 10 000 so where can i put go to filter the filter you can go to the properties you can put under condition and one more condition also i can put go to port salary greater than 10 000. like this you can put under our conditions any number of conditions no issues apply okay since after creating the session i have changed the mapping i can refresh the mapping here refresh mapping ctrl s then run it now it will not load 35 records it will reduce the number of record why because i have changed the filter conditions it is not sql operation just filtering out we are fetching the data from a flat file right for an example your customer is sending the flat file from the flat file you want to filter out that we are doing it that&amp;#39;s all same like sql only but we are filtering out the card if you check this session now it is loading only eight records to the target you can go here you can check the target now so the employees who are getting salary greater than ten thousand dollars and also those who are getting commission so these two condition it is satisfying it is fetching all the records informatica is fetching all the record from the flat file then it is filtering out so this is the filter transformation you can see here this is the filter transformation anywhere in the pipeline you can use this filter transformation so this is the property of filter transformation kindly practice the session and we will meet in the next session with another topic thank you for watching hello friends welcome to nikeite academy in our today&amp;#39;s session we are going to learn expression transformation this expression transformation is very much used in our day-to-day project to implement business logic so expression transformation will use lot of functions to write expressions if you want to learn all the functions i have given the description link in oracle single row functions please watch this session to get complete idea about the functions which is used in expression if you haven&amp;#39;t subscribed our channel please subscribe the channel and click on bell icon so that you you&amp;#39;ll be getting all the notification thank you let us start our session [Music] previously we have seen source qualifier and filter both the transformations are active transformation but this expression transformation is a passive transformation passive means i we have already seen that so how many record you are giving to the transformation the same number of record you will get it in the outside so that&amp;#39;s what we will get passive right passive one and we can see this this is the connected one it is a passive and connected and whenever we want to apply some business logic in our informatica then definitely we will go for expression transformation so whenever we want to apply some business logic in our informatica then definitely will go for expression transformation in expression transformation there will be three port one is input port output port variable port and we&amp;#39;ll be having expression expression is not a port just you can write the business logic okay what is this input port and output port so input port we are getting the data from input and this output port will be enabled if both input and output enabled you cannot write any logic here why because it&amp;#39;s coming like input it&amp;#39;s going right so that&amp;#39;s what we cannot enable any business logic here if you want to write any logic either it should be output port or it should be a variable port variable port is nothing but if you want to do any intermediate calculation intermediate calculation then you can go for the variable port okay here look at the logic just we are getting one column called salary it&amp;#39;s a input and output if the salary is 10 000 it will come as 10 10 000 and it will go as 10 000. there is no changes here but we are creating one more new column called new salary this column is nothing but just we have created this columnist output column since we are creating the output column this this has this will be available only in expression transformation okay since it is output expression will be enabled just i&amp;#39;m making the expression like salary this column i&amp;#39;m just using it if it is ten thousand ten thousand plus ten thousand into zero point one that means ten percentage increment on the salary so saturday it will come say for an example salary is ten thousand so here we&amp;#39;ll be getting ten thousand here you&amp;#39;ll be getting eleven thousand that&amp;#39;s what the new salary what is variable port variable port is intermediate calculation say remember if you are writing any variable port you cannot take it outside see here so this is what one expression transformation you can see this is one expression transformation in the expression transformation we are passing one of the column called salary and salary we are taking its output so there is no changes here but we are creating one more port called newport we are creating new salary that&amp;#39;s what we are creating new salary this is what our what called salary this is what our salary that means if the salary is 10 000 so here we&amp;#39;ll be getting same ten thousand here you&amp;#39;ll be getting same ten thousand but here you&amp;#39;ll be getting new salary say for an example this is what incremented salary or salary bonus or you can we can say anyway it&amp;#39;s a normally we&amp;#39;ll go for o underscore o underscore means output column so normally we&amp;#39;ll create o underscore is nothing but output column v underscore is nothing but variable port so o underscore salary is nothing but we are getting the logic salary plus salary into 0.1 just we are making one simple logic here if the salary is 10 000 the new salary will be 11 000 will be taken from here that&amp;#39;s what if it is variable port if you are creating any port as variable port variable salary just on creating as variable salary this port you cannot take it outside you cannot take this port outside this is what the expression transformation we are having input port this is output port this is also output port just we have created a new port here this is new output port this is output only just we have created output port this is a variable port variable port you cannot take it outside that&amp;#39;s what i have mentioned and see here this is a variable port that&amp;#39;s that that is what i have mentioned variable we underscore variable port a plus b so i am adding two numbers and storing it in the variable called c and if i want to take this variable port outside just assign this variable to output port and if i want to use the same value v v underscore c then i can use it but i cannot use o underscore c anywhere okay that&amp;#39;s what output port you cannot use it for next level but variable port you can use it so remember in expression transformation the order of execution of the port is so this is what first input port then variable port then output port this is what it will apply so remember expression is a passive transformation we cannot apply any filter condition in expression so that&amp;#39;s f it will not filter out any record okay it will not filter out any record look at here this is one simple logic i&amp;#39;m going to apply if the employee is getting the salary greater than ten thousand if already the employee is getting greater than ten thousand dollar then just incrementing by ten percentage it&amp;#39;s already if the salary is less than 10 000 i am incrementing the salary by 20 percentage the new salary okay this is what the calculations how to do it so remember in expression in any of in informatica this is the logic actually if logic iif if is nothing but iaf here if condition you write the expression here condition if the condition is true what i have to do if the condition falls out i have to do that&amp;#39;s what if condition iaf there is no if okay there is no ef here this is called a logical if condition same like our other languages this is what in our informatica also if condition iaf there is no if okay remember this if condition you can go for nested if condition also like if condition if it is true one more if condition you can go with and one more if conditioning can go so this is what we can go for multiple nested if conditioner so definitely we will be writing this kind of logic in our expression transformation okay so this is for just a theory we will go for practically we will see so that you can get a clear idea we can go to informatica just i&amp;#39;m closing that existing mapping and i&amp;#39;m taking the source tables employees table so i&amp;#39;m creating a mapping called m underscore expression transformation expression transformation just you can try take the source drag and drop source into a mapping you can see here so this is what the expression transformation we have and right hand side right hand side you can take t underscore employees this is a target table just we are going to implement this particular logic salary logic and we can load the data into the target so implement the logic we are bringing the expression transformation here just to take this is the expression transformation f of x is nothing but expression transformation we can click one time here we can post it here and then you can place it here and then you can take all the columns select all drag and drop here to here and we can use the column here okay just we can make double click on this expression expression is a passive you can look at here input port output port variable port and expression if the port is coming like input on the output then just it will go from it&amp;#39;s like same like our normal transformation it&amp;#39;s coming like input getting us output here there is no changes there is no changes if it is coming like this but if you if you want to do any logic changes you cannot apply on the same column remember in expression if i want to do any changes you cannot apply on the same column since it is a input so if you want to apply some logic only input you can make only input you can remove the output port that means only input we are getting from this source side okay then you keep the cursor here cursor here you can click on this add a new port to this transformation this icon this icon will add a new port this icon will delete the port this icon will copy about this one it will move the port down or up okay you can see here okay up and down you can keep the cursor here just to create one new port called o underscore incremented salary o underscore incremented salary just i&amp;#39;m creating that oh underscore to denote these this this column we have created just i&amp;#39;m making like lowercase if the salary is decimal oh underscore strategy also should be decimal right decimal so you can go for decimal point if you want to decimal point i told already it should be output or expression then only the expression will be enabled if coming like input and output there is no expression okay it will not be enabled either it should be a output port or it should be a variable port so then only this expression will be enabled so just i&amp;#39;m making like output port there is no intermediate calculation that&amp;#39;s what i am taking that output port go to this icon this expression you can click on this arrow mark this is what the expression window will be opened as as i told previously we can see all the functions are available we will cover most of the function here okay just a look at this if you want to learn all the function very clearly so watch my session on sql single row functions okay it&amp;#39;s like a one hour session i have posted you can watch it clearly that what is this apart from that we do have some expressions are there some more functions are there additionally some functions are it will be changed but that you can learn it from here okay so now i will take this if condition since this one this condition we are going to apply if salary is greater than 10 000 this is what right so i have to write if condition iaf so here this one iaf if you select this you can see this you can find the explanation here just to keep iaf if salary okay if go to port always choose the column from the port don&amp;#39;t type it you you might be doing some mistake okay if salary greater than ten thousand if salary greater than ten thousand comma if it is true then what i have to do greater than 10 000 10 percentage increment so i can make 10 percentage increment salary plus salary into 0.1 0.1 is nothing but 10 divided by 100 right so you can make this what salary plus this one this is a true condition if it is false what is false then you can make comma here then this is 20 percentage you can make 0.2 then you have to close it you have to you have to close the bracket so always make sure that you are opening the bracket and then the closing and you have to check everything validate it always you have to validate any expressions the expression should be valid so just click on ok so this is what and apply here okay if you look at here if you are making any port as input port alone then you cannot take output from here you can see there is no this arrow mark from outside of this expression you could you could not able to see here right so if you drag and drop this column you cannot drag and drop it so in the below it&amp;#39;s showing like cannot start a connector in input only both that you cannot do it but look at here this is output only port right so you cannot pass any input to this port so it&amp;#39;s a look at here there is no arrow mark in the input side so you can take this port to a next level so you can ask me the question can we make this whole underscore salary to salary port yes we can make the column name might be different no issues the data type should be matching okay even you can ask me here it is a decimal why it is number so as i told so inside the informatica everything is like a informatica native data type okay here this is what inside the informatica it is inside we have the native data type after that source and target will be based on the target and source database you can make control yes and it should be valid go to workflow just to create a workflow create new workflow wf underscore expression transformation i&amp;#39;m just creating expression transformation click on ok and click on the session choose the expression mapping click on ok click on the link task double click on it go to mapping source should be pointing to source target should be pointing to target connection and just time truncating and loading the existing data it will truncate and run the road for our academy purpose we are just a truncating and load we should not do it truncate and load always in the real time project okay you can go to the config object you can enable the stop on error equal to one as i have already explained stop on error one means if any one error this will be stopped it will be failed the session will get failed go to general you can make failed parent if this task fails if this task fails fail the workflow so control yes the workflow also should be valid workflow also should be valid here and then right click on the whitespace anywhere start workflow it will load all the data so you can choose whichever the workflow here double click on this r you can go to the get run properties so source and target statistics we can watch it and then you can see what is the source what is the target so this is what you can see here what not sound record or not seven record got loaded it will not filter out any record you can go here in the target we can check what is the data it has been loaded if you look at here for steven what is the salary here in target it is 26 400. if you go and check here select star from employees table this is source table right source table steven is twenty four thousand so twenty four thousand is greater than ten thousand right greater than ten thousand so it will increment only ten percentage twenty four thousand plus two thousand four hundred right two thousand four hundred so that&amp;#39;s what twenty six four hundred right okay this is what it is making then if i have less than ten thousand this is what for alexander we have less than ten thousand here we have nine thousand so nine thousand plus thousand eight hundred right so it&amp;#39;s a twenty percentage increment so what is the salary here ten thousand eight hundred so this is what it should be loaded you can see here we will check it in the target so steven is having like salary like 26 400 and alexander we have alexander we have salary as 20 percentage incremented ten thousand eight hundred that&amp;#39;s what you can check it for all the records it will be loaded with all the records so this is what one simple logic we have applied like if condition okay we do have many other logic just we will see one by one okay the first name so look at here these are all the condition just i&amp;#39;m applying it here the first name in the source whatever the source we have in first name it should be an upper case in target so convert the first name in source into uppercase last name the source it&amp;#39;s like last name we have but in the target it should be like concatenation employee id underscore last name just i have to make 100 underscore king just i have made it like how to do the concatenation informatica okay we will do one by one so that you can you will get a clear idea how to apply some logic okay email column email column look at here if the salary of the employee is greater than ten thousand i have to make the email as gmail.com if the less than ten thousand i have to make email at yahoo.com okay so this is what you can make email here and phone number so phone number is nothing but we have this format we have to remove the dot from the phone number we have to add plus one iphone okay in the phone number just we have to add it and job id we have to take first for from fourth character we have to take java id and the commission percentage we have to take if it is zero you assume that in target it&amp;#39;s a not null column so we should not load null values on the target we have to add some not null values here okay just you have to make some zero here okay manager id just i&amp;#39;m passing the sequence number i want to make sequence number i have to generate sequence number from one to sequence number you have to generate it so we will do one by one first name we&amp;#39;ll go to the same mapping double click on it first name create a new first name so o underscore first name create a logic for o underscore first underscore name so you should have the underscore not hyphen underscore here it is a string and string you can have the same 20 characters and you can make this as only as input this is output port and then you can make this is output you can see here upper first name how do i know the functions you can go for function name here just you can click on up or off you can make upper first name validate it it will convert the first name into uppercase okay you can go for if logic any other logic like this also you can go for what is the last name last name you have to make employee id underscore last name we have added so how can we do it i&amp;#39;m just making like underscore last name it&amp;#39;s nothing but output port oh underscore last name and then string here i&amp;#39;m going with 25 characters and since i&amp;#39;m making that employee id underscore right i will go for 30 characters is it possible yes it is possible so only 25 characters coming is input 35 characters we are taking but next transformation done it should be accomplished 30 characters okay last name what is the last name it should be employee id underscore so employee id here concatenation how to do the concatenation pipe symbol so you can use concat or pipe symbol so pipe symbol is very easy so just to use pipe symbol and i have to use space or underscore right just you can make here one space or underscore just like this and you can make like this and this is what you can go for first name underscore last name you can make like full name that is also possible okay last name that is what we can make just to validate it should be valid okay so now i&amp;#39;m going with email column so same way if logic right so underscore email so what is the logic here if the salary of the employee is greater than 10 000 i have to make it like email gmail.com if it is less than that email at yahoo.com okay i can go here you can make this is 35 characters are 40 characters this is only input this is only output then expression will be enabled here you can write the logic okay how to write the logic if condition iaf as we have seen already iaf if salary greater than ten thousand then i have to make email at gmail.com along with this i have to add at gmail.com gmail.com if any other conditions like is there any regular expressions yes we have in our functions we have regular expressions we can use some special characters you can go for regular expression replace regular expression extract regular expression match we replace all this we have these three we can use it something like only alpha alpha number one be numeric number that you can use okay that will take it as exercise i will tell you email thiago.com this is ellspart at yahoo.com you can make the sales part and close the bracket so clearly you have to close the bracket you can see here so this is what if condition the condition is true what i have to do if the condition is false what i have to do okay validate it it should be valid and click on ok then phone number you can click on here you can add a new port below to that and oh underscore phone underscore number so it&amp;#39;s not necessary that this is what same weed we need to have just i have to add some character right i can make 25 characters so i can make here and apply the logic what is the logic plus one iphone it&amp;#39;s hardcoded value right so you can use single quotes plus one iphone single quotes this is character right so any character any hard coded value you can make it like single quotes here okay so now concatenation we have to remove the dot symbol from here we have to remove the dart symbol so for that we have to go for replace character replace character you can see here you may ask question how do i know all these functions you have to practice it one by one you have to practice it one by one that is the way you will be getting it replace character it will replace the character replay string the entire string will be replaced so based on the scenario we have to go with just i&amp;#39;m taking the replace character replace character you can give four argument look at here four argument first flag is integer so what is the flag right replace character you have to give four argument remember four argument if the flag is true if you look at here if the flag is true this function is case sensitive okay that means case sensitive replace or case insensitive replays if i&amp;#39;m asking you to replace a with x okay you may ask question so i&amp;#39;m replacing a with x if small a comes should i replace or not you&amp;#39;ll be asking questions right yes even both the case is coming you have to replace with x this is called a case insensitive replace okay case instead is to replace case sensitive replacements only capital a it will be replaced with x okay only capital here this is called a case sensitive refresh so this is you have to make case insensitive you have to give zero zero here case sensitive you have to make one one here okay clear on this that is what we will make first zero will make case insensitive then what is the column we have to change phone number what we have to replace the dart single quotes you make the dart single quotes comma two single quotes two single quote is nothing but null value either you can use this wave or you can use null that is also fine okay just i&amp;#39;m making like this way this is dart and this is null okay you can see here just to validate it it should be valid and okay that is what we have made it here then job id job id from fourth character we have to take right i&amp;#39;m just adding that to o underscore job underscore id then string here for the character from fourth character how will you take it&amp;#39;s a substring right so you can go for sub string s u b s t r substring of jaw by d i have explained some substring very clearly that is what you have to make from fourth character from fourth character means four four means from fourth character right hand side it will take that&amp;#39;s what so if you want to check all the functions please go through that on our video on single row functions okay in real time if you want to apply any logic if you want to clearly apply that you want to think so what could be the logic so definitely you have to practice all the functions whatever you have okay so please make sure that you are practicing it okay next one salary we have made it and the commission percentage so whenever we are getting null value we have to replace with zero right so i can go with o underscore commission so this is also decimal make sure that it should be a data type everything is correct then here we have to write if logic is null of commission percentage if is null this value is null then we have to add zero okay replace with zero otherwise we have to go with commission percentage itself okay if is no law of commission percentage this is condition if is null of commission percentage then zero otherwise commission percentage itself so this is what you have to write the logic if condition in expression validated it should be validated okay this is what logic i have applied a lot of logic i have applied you can go for manager id sequence of number should be generated how can i generate sequence of numbers you can go for o underscore sequence of numbers okay it&amp;#39;s nothing but sequence of numbers to generate the sequence number we need to use variable port to use the variable port that&amp;#39;s what i want to tell you that how to use the aerial port i&amp;#39;m making this is a decimal port i&amp;#39;m making like variable port so what is variable this value will be taken as first zero zero plus one it will be incremented one by one variable plus one next time this value will be taken so variable port one plus one two two plus one three like this it will convert then we so as i told variable port you cannot take it outside look at here the variable sequence is not visible right so you can take first name here yes you can take last name here yes you can take from here email port so you may ask question previously we have connected why this connection has lost right since we made it like that port has only input the connection has been blast that&amp;#39;s what you can make the phone number here drag and drop then java id you can drag and drop then commission percentage can drag and drop then manager id look at here there is no manager id here right you can remove this and i want to take the sequence of numbers i i cannot take variable port outside that&amp;#39;s what i told so you have to assign this variable to the output oh underscore sequence it is also decimal then we can make output port assign a variable to this okay click on ok then you can take it this this one so this manager id will have a sequence of numbers okay you may get some some little bit confusion on this very first time if you are watching no issues you have to practice by watching this practice okay so whenever you are going to practice it you will get a clear idea it will get memorized and you can go for some other logic also i will give you exercise okay now i will go to the right click refresh the mapping since we have already created the workflow then load it since we have given the truncate and load it will truncate the existing data and then it will load we can see all the new logic will be loaded it gets succeeded it will not filter out any record you can see here or not some record got loaded and we will go to the target table we will see one by one all the columns whether it has been loaded or not we&amp;#39;ll check it so it is in upper case yes and then it is making like last name underscore but you might be saying underscore is not there but if you click on it you will be seeing that underscore okay then so whether the employees salary is greater than 10 000 you are seeing that at gmail.com are yahoo.com plus one iphone has removed and date column also we can apply some leap year only piranha job id and commission percentage and sequence of number by using expression how will you generate sequence so this is what we have to use it yeah i have given some expression logic so one or two exercise i have completed already just you follow it okay i have given some uh answer also in phone number we have two types of phone number one is this format another one is this format so whatever the format we have to split up this way plus one iphone first five characters iphone next to five characters plus one iphone first five characters iphone next five characters like this you have to load it okay and then you have to look at this one you have to complete it if you are not able to complete it i will explain in our next session okay tomorrow session i will explain and look at this one generate sequence number i have made it already and load the input data in reverse order so that you can apply you can complete it apply ten percentage salary only for first ten employees okay that how to do it uh you have to reduce the logic okay i have given the answer for the first question also here don&amp;#39;t go for answer directly so you go for first you try it and then you can go for the answer so kindly practice it you will get a clear idea about expression all this so remember expression transformation will not filter out any record thank you please practice it you will get all the stuff very clearly [Music] hello friends welcome to nikkei academy in our today&amp;#39;s session we are going to see another important transformation called china transformation so this giant transformation is very much used in our real time project so we will see this concept in our today&amp;#39;s session so that is what is joiner transformation why do we need this china transformation and what are the different join types are available we will see one practical session to understand on this channel transformation if you haven&amp;#39;t subscribed our channel please subscribe the channel and click on the bell icon so that you will be getting all the notifications thank you let us start our session [Music] so what is the use of source qualifier joiner source qualifier joiner used to join the homogeneous sources like joining from same type of database same tables data coming from the same database right but here if you consider i have one table okay i have one table from one database some other tables from another database i have one table from this is from oracle okay another one is from platform or sql server another source from sql server so i have two different sources so this is one source this is one source one is from oracle the another one is from sql server or db to r cloud file okay so this is oracle you assume that it&amp;#39;s an employee&amp;#39;s table okay any table you can go for so it has employee id right see whatever we have seen the same concept for source qualifier same thing we are going to see but the source qualifier we have taken both of them from oracle but here we are going to take so one is from oracle the other one is from flat file okay so this one we will take it like a flat file that file is nothing but our department&amp;#39;s table so normally if it is a source qualifier what we can do so we can have only one source qualifier here so it&amp;#39;s qualified java we can have only one source qualifier how many number of sources are there so with respect to of number of source we can have only one source collector but in joiner we need to have so this source qualified into hub and for this also we need to source qualifier we need to keep we should not delete it and we can we can take one joiner here one joiner here see whatever the first table if you drag and drop into the joiner the first table will become like it is a detailed table the first table is detailed normally we will call it as left table right table right left table right table but in informatica the first table will call it as detail table the second table we will call it as master table master okay master table then we have to do the join condition join condition what is the join condition we can write the join condition here you will be having the department id department underscore id we have department name on location id right department underscore name and location so like this you&amp;#39;ll be having a column location id all the columns will be there so here you can take all the see all the columns you can take and you can take all the columns here okay so first table is detail table second table is master table join condition on the join type we have to take the join type also so what a join type i will tell you but join condition is nothing but see join type is nothing but whether it is a ignorant left out right outer will take i will take this joint type after this see join condition is what is the giant condition sorry implies that first you will take implied stability implies that department underscore id equal to departments dot department id so like this you&amp;#39;ll be taking the columns right so you&amp;#39;ll be taking the columns so this is called join condition this is called join condition normally is what we go for the joint condition okay then after that you can take the columns department name all these columns you can take if i so there is no filter condition in designer if i want to filter out the record either you have to before the joiner or after the journal it&amp;#39;s based on the source which source we are going for so what is the first table it is a detail table what is the second table is a master table clear so we have four type of joints the joint type will go for four type of joints see we have four type of join type so one is the first table if you take the first table it will it is called detail table second table it is called the master team the first table what we are taking that table is detail table second table is monstrative always remember this first table is detail second table is master the first giant type is called normal join what is the giant first turbo it&amp;#39;s a normal chart normal giant means inner join so inner join means matching record from the both the tables right it&amp;#39;s matching the cut from both the tables you&amp;#39;ll be having matching non-matching record so only matching record this is called normal join second join type is called second join type is called if it is left outer it is a master router master router what is mastered right it is a left outer join left outer means all the record from the left table all the record from the table matching the cut from the right table this is called master router okay third one is detail outer detail is nothing but right outer okay detail outer so this is what we will we will explain if they are asking about your joiner transformation you have to tell them either it is a normal join or master router or tt loader or full loader so what is this left out right after how it will perform so please go through my joiner video okay please go through that before this joiner please go through them so then you will get to know all the joining as well as your a source qualifier properties okay detail router is nothing but this is a detail order see how can i easily understand how can i easily know this which is the detail which is the master see here if it is normal join means it is a inner join you know already master router means it&amp;#39;s a left order all the records from the left matching record from the right see master router means master will be out okay that means all the record from the detail detail outer means detail will be out all the records from the master clear fourth one is fourth one is called full outer full after you know right floater so what is the floater joint full loader is nothing but matching non-matching record from both the tables so where it will be used in real time so normally if i want to compare two tables so if i ask some questions you can you can take the data from two or more tables right so that is what these table joints are used okay remember this see whatever that table we have smalls table two tables we are taking right whichever the table is having small table like small number of record list number of record so that table you take it like a master table so why write why we have to take like master table so because informatica it will take catch memory catch a memory on this master table only okay whichever the table small table informatica will take this table into a informatica catch memory then it will create a caching so if it is small table then cache will be very less space to log back otherwise informatica will take lot of space performance wise you will go for the degradation see if they are asking how to improve the performance of china you can tell them i will go for the small table that master table small table as master table okay if you have n number of tables n number of heterogeneous table n sources you have to use n minus 1 china remember this one n minus one china why so if i have two sources how many giant needed two sources one china needed three sources two china needed remember this okay n sources n minus one china small table you have to go for master table clear on this if i have three tables like this how many china needed two joiner first employee is its oracle table departments is a flat file just i&amp;#39;m taking these two like detail and master just connecting through the join condition okay through joint condition i&amp;#39;m connecting after joining i&amp;#39;m joining this to another china china two this is a detail so this is the master this is the giant condition we are making okay so this is like joiner one and joiner two so this is what you can go further if it is three sources say for an example this is oracle this is oracle you can use by using the source qualifier and then you can have only one joiner okay if it is possible to use source qualifier use it in the source qualifier itself say concept wise you have to be very clear how it will perform then joiner transformation is very easy only okay i will take m underscore joiner it&amp;#39;s very simple only i will take one is from a table another one is from a flat file i will take employees as a flat file now okay this is a flat file departments are stable oracle okay there&amp;#39;s a table then i can go for the joiner here so this is a joiner okay two into one right first table if you drag and drop this will become a detail detail second table if you drag and drop this is the master you can see here this is the master first one is detail second one is master okay then you can have target simply double click go to join condition just to make the join condition add a new condition see operator will have only equal symbol okay so department id equal to department id here not employee then properties you can choose the join type you have the four type of chart normal join master router detail or triple so i can choose the normal time so you&amp;#39;ll be having null order that null order i&amp;#39;ll take i will tell you what is this while taking the next transformation i will tell you what is this null order started input and null i will t i i will tell you this properties in the next transformation then you will get to know joiner will take data cache and index cache indexing is nothing but the join condition it will take the indexing okay so here i will go to the employees underscore department for that online we are going for the joiner right so i can take department name and i can take all this column from here control yes it&amp;#39;s a valid one okay then go to the go to workflow or close create wf underscore joiner underscore transformation it&amp;#39;s very simple only double click on it go to mapping source should be finding the source target should be pointing to target so you can choose the source and target a little bit down you can do the tranquility and load apply okay save you can start workflow if you look at here so 107 record from the source source qualifier 27 records from the departments it&amp;#39;s a flat file and then 27 you are getting one or six records so one record since it is a normal join it is getting filter out filtered out okay you can go and check in the target it should get loaded if you are doing like a master router okay for an example you are doing master router master router is nothing but left or right master router is nothing but left order i&amp;#39;m choosing master router then if you execute this you&amp;#39;ll get all the 107 record why because from left hand side table you will get all the records okay you can choose here select star from t underscore employees underscore department right you can see here just one not soundtrack one not six records will be there right still the informatica is running so there is no null value on the right hand side but if it has been completed it will have null value on the right hand side see here 107 record has been loaded you can go and check here not seventh records you&amp;#39;ll be having some null value on the right hand side all values from the right hand side so this is what you&amp;#39;ll be having left out or right outer floater all the joints [Music] welcome in our previous session we have seen the joiner transformation now today&amp;#39;s session we will learn the shorter transformation okay sorter transformation is an active transformation and also it is a connected one so if you look at this the sorter uh in sql we have the order by class right so in in sql we have order by so order by is used to do uh sorting algorithm either by ascending order or descending order the same way in informatica we are the transformation called sorter so it will do the sorting algorithm right it will make the data either ascending order or a descending order so where it will be used in real time so whenever we are facing some performance issue like we are using some aggregator we are using some rank transformation we are using some joiner or we are using some lookup so data is like not sorted order then it will take a lot of time it will have some performance issue so to avoid this performance issue we are going to use this water transformation so sorted transformation is used to sort the data okay so whatever the data is coming to start the data then you can ask me the question why it is inactive or if you are giving hundred records it will start right how would you be active that we will see the later point of our session if you look at here in sql we have the order by class if you look at this this particular these queries if you i&amp;#39;m going to execute this query in the hr schema if you look at here just i have made it like select start from employees table order by salary so if i&amp;#39;m selecting only select start from employees just i&amp;#39;m selecting only select start from employees it will select whatever the way we have inserted the data but i want to order based on the first name or something so for an example instead of salary i&amp;#39;m going with first name so first name i&amp;#39;m making it and it will make the order by on the first name so look at here this is from alphabet order a and then it is alphabetically it is increasing right so we are getting some j here why it is like j is last right small j is lost because of this lower case okay the ascii value of lower case is higher than the upper case that is why this john is on last okay if you are making first name on descending order then this this john will become the first one right then it will go to the order by under descending order so this is what the order by for an example instead of order by first name i&amp;#39;m going with order by salary so it will make the salary built on the order so if you if you are not mentioning any ascending order descending order it will by default take ascending order so this is what it will go with lower salary to higher salary it will increase in this way the highest salary is five nines right so this is what the uh the order by class will do my skewer you can make order by salary descending you can see here the order basically descending it will make the sending order of the data so for an example here if you look at the order by salary and descending we are making the employees based on the salary from highest to least if you look at here there are two employees are getting the same salary right two employees are getting same salary if you are ordering this way this employee may ask question that okay why i am the second person right among us i am the second person he will she will ask question and also if you look at here there are four members are getting the same survey so it is making order by on the salary only but within this particular salary how we are ordering so that is called the second column so second column or third column you can make it so order by salary descending then one more column we are giving like higher date so what will happen now so it will make order by on the salary only first if the same salary if i&amp;#39;m getting more values here more repeated values then it will go for the second column so second column is look at here there are four employees are getting the same salary rate then it will make the order buy on the higher date ascending order we haven&amp;#39;t mentioned anything ascending or descending by default it will take us something within this particular salary it&amp;#39;s making the ascending order look at this so this is what the salary column you can do it okay for an example so this is more than one column you can make any number of columns you can make okay and also if you are making this uh order by any column if this particular column itself the first column itself i have the unique value then oracle will not check the second columns so this column it will not check why because here itself it is unique right so you will be getting higher to higher salary to least salary then it will not check the second one if if i&amp;#39;m getting any duplicate value on the first order by class then it will go for the second one so this is the order by one okay you can ask one more question how the null values will be ordered so always the sql in any database null will be treated as highest value look at here select start from employees order by i am making commission percentage why because this order by commission percentage this column contains some null values so i&amp;#39;m making like commission percentage it is making from here then it is making from point one to point four after that point four only it is making null values why because null treated as highest value by default if you are making order by descending then this is what the descending first null values then only the highest value so all the null values then highest value then it will go for the least values so null always will be treated as highest value so why i am telling all this right the same concept is applicable in informatica also you can make one column order by on one column or two columns or a number of columns okay and also how how the null value will be treated so that is also we can use it in sorter transformation so in sorter so if you look at here sort of transformation we have uh port some different port will be there like you have the input port it will receive the input so we know already the input port will receive the input and we have output port then keyport then direction we&amp;#39;ll have if i&amp;#39;m giving salary as input port and i&amp;#39;m enabling the salary as key port here then make that direction either it&amp;#39;s ascending or descending from top to bottom whichever the port is enabled the first one it will be ordered first okay so first it will order by salary descending if the salaries are same then it will go for the second column is higher date if you are making this way first column is higher rate then salary from top to bottom which is the first one now higher date so higher date will be ordered first if some some employees have joined on the same date then it will be order based on the salary so this is what it will do i can make this way so this way also i can make here okay so control is that i can make here also okay now i will go to the informatica and i will do the mapping go to mapping designer since i have already taken this employees table and t underscore employees table i&amp;#39;m going to make this employees table as source p underscore employees tables target and i&amp;#39;m going to create a mapping create a mapping m underscore sorter transformation just try making the mapping name is m underscore starter transformation then drag and drop implies source and we can drag and drop the p underscore employees on the target side so this is the target and this is what the source you can keep the sorter here you can keep the sorter here this is air2z it will do the sorting algorithm you can take the sorter here you can make the sorter in between source qualifier and target you can drag and drop all the columns to sort or now you can see the keyboard right keyboard here keyport is nothing but order by whichever the column we are giving that is a keyboard double click on the transformation you can go to this key port and direction you can see whichever the port for an example i&amp;#39;m i&amp;#39;m making that keyboard like salary you want to make it like ascending order descending order you can enable this keyboard and direction you can give as of now i am giving like salary order by on the salary descending order then go to this properties the properties you can see sorter catch size this catch memory is nothing but so sorting algorithm it will compare all the records and then it will do the sorting so that is why we are the working directory informatica will keep all the data it will fetch all the data to informatica cache memory in the cache memory it will make sorting algorithm then it will make ascending or descending based on the keyboard we have selected keyboard and the direction we have selected and it will create a sorting so definitely it will fetch all the record to the informatica level and then it will start it will take some time okay sometime it will take and you can see the catch size you can give auto don&amp;#39;t need to give any other values here why because if the data volume is getting increased so this will take automatically the cat sizes automatically it will uh assign it case sensitive so what is this case sensitive case sensitive is nothing but if i&amp;#39;m giving like this okay if i&amp;#39;m giving like this some data so this is arun okay this is babu and babo this is how the case sensitive which one should be order first either this one or this b or this b okay so that is what the case sensitive if you are giving case sensitive yes then this will be taken as first this will be taken a second why because based on the ascii value the high the upper case will have the lower value right so that is why it will take the first one then it will take the second one then it will take so this is what you&amp;#39;ll have the case sensitive you can give always it will be enabled we can give that working directories dollar pm temp directory so it will keep the data in this particular directory it will make the order then distinct i will tell you after some time the null treated as low so we have seen right null value will be treated as highest value we have seen here if i want to make the null treated as low value then you have to give here like null treated as low value so this is transformation scope for all the inputs so these are all the properties we have one property called distinct this property if you are enabling this property this sorter transformation will allow only the distinct record pass through it it will remove the duplicate from the sorter we have this distinct option in two transformation one is sorter the other one is source qualifier source qualifiers distinct will be used to select the distinct record from the source itself but anywhere in the pipeline if you want to filter out sorry anywhere in the pipeline if you want to select the distinct record you can go for the sort of transformation this distinct because of this property only the sorter transformation is call it as active transformation since it is removing the duplicate right if i have any duplicate it is removing the duplicate so that is the reason the sorter transformation is an active transformation so based on your scenario you can select the properties here now apply just i have given only this salary descending order click on ok then select all the port and then you can give it to target control yes it should be valid then go to the workflow manager we can create our flow and we can run it we will check whether it is making the order by on the salary descending order so wf underscore sorter transformation click on ok then click on this session task you can keep here session then select the starter transformation click on ok now you can link the task here from here double click on it go to mapping source should be pointed to source source connection the target should be pointing to target one so we know already the connection details we have seen in the installation itself and we are using this for more point of time then truncate target table option if you want you can enable it so incremental data load you should not enable it only if you want to have some staging load or truncate and load in the sense it will truncate the target table and it will load the data so for for our practical purpose we are just truncating and loading it you can double click on it you can enable this property like fail parent of this task fails and they can enable one more property in the config object stop on error equal to one it&amp;#39;s nothing but if any one error comes it will stop the execution and also fail parent of this task file in the sense fail parent if this task fails if the session fail the workflow will get failed so normally these properties we will enable the real time okay now it&amp;#39;s valid and right click start workflow so the monitor will be opened now you can double click you can go to the get run properties you can see here how many records it is reading from the source and writing into the target you can see here one not seven recorded from the source and uh 107 record return under the target you can see so this is what you can you can see the statistics and there is no rejected record and throughput is number of record per second if you go and check in the target table you can go to the course schema and check this t underscore employees table you can see here the data has been loaded based on the seller right based on the salary descending order it is loading but look at here here i&amp;#39;m having like if i&amp;#39;m having two employees are getting same salary it is not able to do the ordering right it&amp;#39;s not able to do the ordering so within this 10 000 it is not in order or if you want to order based on the employees based on the higher date you have to go to the so these four employees will ask question right so herman may may ask question why i&amp;#39;m also getting 10 000 why i&amp;#39;m the fourth person right among us then i will tell like okay whoever joined to the company first or higher date will be the first i&amp;#39;m making that order within this particular employee then i can go to the designer double click go to the higher date and enable the higher date first so as i told already from the top to bottom whichever the port is enabled first the informatica will make the order by on the first column so first column is higher date it will make order by higher date then it will make the salary look at here i will show you the result then i will make the second one okay now it will make the order by on the higher date only since i have changed after creating the session refresh mapping control yes start workflow it will again it will start and then run so since from the top to bottom now it is higher date is first one it will informatica will order based on the higher date if some employees have joined on the same date it will order based on the salary look at here it will order based on the salary you can go to the target now you can check it it will not based on the order by the salary it is based on the order by on the higher date 2001 then 2002 then three like this but if you look at here 7th june 2002 there are again four employees right seventh june there are four employees have joined on 7th of june but based on the salary it is making the order why because here we have given order by higher date first then salary descending right salary descending that is the reason order by higher date then it is making the salary deciding but our case our requirement is different right order by salary then higher date how can i make keep the cursor here you can see the error mark here you can move the selected port down you can make it down so from the top to bottom you can see the salary is first one the next one is higher date apply here then apply so something if you look at here the link will be twisted one no issues then you can refresh the mapping then you can run the workflow it will order correctly now it has been completed you can control enter you can select it you can see order by on the salary now employees are getting same salary if employees are getting same salary it is making order by on the higher date ascending order june 2002 then jan 2004 jan 2005 march 2006 so this is the functionality of sorter transformation okay so starter transformation i have explained everything that uh the one exercises i can give you so as i told we are the property called distinct you can take some sometimes the interview they will ask okay if i have a flat file it has some duplicate record how can i remove the duplicate record from a flat file okay you cannot use source qualifier distinct right you cannot use source qualifier distinct why because the source is flat file so after the flag file you can put source qualifier after the source qualifier you can sort you can place the sorter transformation in the sorter you can place the distinct option if you are enabling the distinct option here if you are enabling the distinct option here then all the port will be keyboard will be enabled to remove the full duplicate it will make the keyboard here so this is the functionality of this distinct if you want to remove the duplicate anywhere in the pipeline we can use starter transformation you can take one example you can take one flat file you can keep some duplicate record then you can place some starter transformation here then you can use distinct you can load only distinct record so full row duplicate only it will remove if any one of the column has been changed then it will not remove it will not consider that as a duplicate record so this is the functionality of sorter so sometimes interview they will ask why is arteris inactive because of this property saturation active so kindly practice this you will get all the functionality of sorter [Music] hello friends welcome to nick it academy in this session we will learn about aggregator transformation so we have seen sorter transformation previously and now we will see the aggregator transformation so what is aggregator transformation this aggregator transformation is an active transformation and connected one so we have seen in sql we have seen some aggregation function right so minimum of maximum of sum of average of count of all this are aggregate functions irrespective of number of record this aggregate function will give you only one record in the output if you are doing some minimum of some salary minimum of first name something like that if you are giving like minimum of some transaction date or date column it will give you the minimum rate so maximum it will give you only one record minimum will take you on record sum of average of count off it will give you only one record we will see one example then you will get some idea about this aggregation then we will go to the informatica aggregator transformation if you look at this table the students table student id student name year of study and mathematics physics chemistry biology marks are there okay our aim is to take the total mark of each and every student the total mark of each and every student and average mark of each and every student so this is something like a row level right this is something like row level so i used to ask this question in many of the interviews they will ask they will answer it like i will go for some off so make a note of it my question is each and every every student&amp;#39;s total marks i need like total marks like this row level if i&amp;#39;m asking the if the interviewer asking questions or you are getting any scenario like this row level you should not go for aggregate function candidate will answer like i will go for sum of all the marks no you have to do like this the select statement we have did it right so we have to do like this you have to do something like this you select mathematics plus physics plus chemistry plus biology as total marks this is our last name then you have to add all the marks divided by 4 is the average mark interviewer asking questions like row level then you have to go with expression you have to add it plus you have to use it if you are getting any scenario like this aggregate function aggregate function will do the scenario like this column level so remember this column level if i&amp;#39;m asking what is the maximum mark in mathematics then you have to go for the aggregation minimum mark sum mark average count everything you have to go with aggregate function so what is the maximum mark in mathematics then you have to use select maximum that max of max from table name so if you&amp;#39;re making that max of math then you will be getting 98 right 98 you will be getting minimum you will be getting 72 right so this is what you&amp;#39;ll be getting what is the sum sum will be getting okay so this is what the sum 779 the nine members have nine students have appeared for the examination then average mark in mathematics is 86 so average or sum or anything this aggregate function will give you only one result remember this so this is average mark in mathematics average mark in physics only one column only you can make in aggregate function you cannot make average of mass come up with six that you cannot make it you can make it like average of max comma average of physics that you can make it but you cannot make two columns in the two are more than one columns in the average function okay if you look at here there are some color right color difference like green color and yellow color green colors the student list from first year then the second one i have taken like year of study second year so why i have grouped it it&amp;#39;s a group by call so we can group by year of study then we can make okay what is the maximum mark in the second year what is the maximum work so this is the same concept would be applicable in informatica also group by on the column and then you can take aggregate value so normally you will get how many group it is creating so those many records you&amp;#39;ll be getting it okay in informatica level if you look at here i will go to the aggregator so this is summation is the aggregator i will create one aggregator transformation i have explained whatever so this is what group by for example sum of salary group by department id i&amp;#39;m making so what is this meaning so sum of employees salary in each department so that is the meaning okay so that you can go with group by column okay now we will practically see then you can get clear idea so same employees table i mean i&amp;#39;m going to take it so i can create a mapping now go to create mapping m underscore aggregator transformation aggregate underscore transformation just i can create it drag and drop the employees table here drag and drop the t underscore employees on the target side you can make here the aggregator is nothing but so the summation this is summation right so this is an aggregator transformation you can take this aggregator transformation here you can drag and drop all the columns from the source qualified to aggregator in aggregator you can double click you can see you&amp;#39;ll be having input port variable port output port variable port and you can see the expression and group byte so one column extra is there like a group by port is available so this is a salary it&amp;#39;s coming like input and output i&amp;#39;m creating one more port is called sum salary sum of salary o underscore o is nothing but output o underscore sum underscore salary so that is what the sum of salary i&amp;#39;m taking it if salary port is decimal you can make one more the output port also decimal you can make this is an output port you can make this is an input port the sum of salary you can make some aggregate functions so what is the difference between expression and aggregate transformation in expression you will not have this folder in aggregation aggregate transformation you&amp;#39;ll be having one folder extra called aggregate if you click on this you can do these aggregations in aggregator transformation okay you can do average count first last maximum minimum median and then everything can be do now i&amp;#39;m going to do some of the salary so i&amp;#39;m going to do sum of salary just i&amp;#39;m making it validate it&amp;#39;ll be valid then click on ok then i&amp;#39;m past i&amp;#39;m passing salary here i&amp;#39;m taking that sum of salary out from here and click on apply okay i&amp;#39;m not selecting any group by port here select from employee id to job id from here to here and you know already this is an output port i&amp;#39;m just taking output the sum salary to the output okay you can ask question this is sum of salaries like a aggregate function right how it can be passed to all other columns so if you look at here in employees table if i&amp;#39;m making like sum of salary what is the sum of salary sum of salary for all the employees it will make the sum salary right so it will make sum of salary is something like this is the sum of salary it is making okay so can i take the sum of salary for something like first name here can i take first name here in the adjacent to it no i cannot take the first name or other columns i cannot take why because so you cannot take non-aggregate column with aggregate column this is on the we know already that we cannot take it if i want to take first name of the employee who&amp;#39;s getting higher salary or something then you have to go with some sub queries but sum of salaries it is not meant for one particular employee right so this is meaningless so this you cannot select it so that is why we are getting error then how would it be so if i am going to give all other columns how it will be in informatica it will be taken as so in informatica level if you look at here select star from employees table right so what is the aggregation value seven lakh seventy four thousand 000 415 this is the value so informatica will take all the salaries sum of salaries and since i have given this column these columns to next level so informatica will take last record okay informatica will take last record instead of this salary 8300 it will make some salary here okay so in most of the scenario you have to take only this salary output this summation value to out next level since we are taking all other columns the last record it will make the the output it will make the summation value look at now i&amp;#39;m going to create a workflow i hope you are clear now we will see how it will be created aggregator underscore transformation you can find source and target go to truncate and load just time truncating for this year for the testing purpose just to save it run it even if i have millions of billions of record in the source it will give you only one record in the target i haven&amp;#39;t selected any group by port so irrespective of number of record it will give you only one record output that one record is last record it will make the summation value look at here it will take the instead of william gets salary 8300 it will make the salary yes only one record it is giving you the result right so one not seven record you are getting only one record even million billion record it will give you only one record same last record instead of his salary it will take the last salary you can ask question okay this is wrong right why it is his salary is making some salary the aim is to take some of sunday only since we are taking all other columns okay to the next level to the target level it is making the the last record it will make the the summation value okay in interview they will ask this question so how many records you will get how this summation value will be loaded into the target level so it will load the last record it will load the summation value okay i&amp;#39;m going to give some group by group by on the department id so i want to segregate the 7 lakhs 44 000 in each group in each department id so as a just i have given group by on the department id refresh on the session refresh mapping then run it if i have some ten departments each department it will give you the sum of salary you can see here you can select here right the sum of salary the seven lakhs forty four thousand this is source level just i can make it group by department id you can make group by on the department id you can select the department id here you can make order by on the department id order by one so one is nothing but first column so each department it is giving you the sum of salary you can see here it is giving the if you add all the salary it will become the seven lakhs seventy four thousand four one five so this is what distribute the salary for each department so 12 records you will be getting it in target also you&amp;#39;ll be getting the same you can look at yes 12 records will be populated so each department you have to consider the department id this column and this column how it will be populated each and every department last record it will load the data okay if you add all this right it will become some that seven lakhs forty four thousand four one y so this is the the functionality of aggregator transformation you have to consider the aggregation group by if you want to filter out okay for an example i want to load only the departments who is getting more than fifty 000 okay who is getting more than 50 000 normally these four departments are getting more than 50 000 right so in oracle we will be using the having class so group by department id we are using having right so in order to filter out record will be grouped by our result and we will be using having plus having you can use this sum of salary greater than 50 000 so if i want to use like this sum of salary greater than 50 000 then in informatica you can use filter transformation i can use something like filter transformation for the aggregator we have to use the filter transformation you can use the filter transformation here and they can keep all the all the columns whatever the column you want you can keep all the columns double click on it go to filter in the filter you can use this column go to port sum of salary greater than 50 000. so this you can make it validate so only those the department those who are getting sum of salary greater than 50 000 the filter will allow you can save it you can refresh the session refresh mapping you can execute the start workflow now you&amp;#39;ll be getting only four records out of it you&amp;#39;ll be getting only four records you can see here the aggregator is running now even though it is reading millions of record in this scenario we&amp;#39;ll be getting four records so four records right only four departments are getting more than 50 000 of the sum of salary what are the departments you can see department id 50 80 90 100. you want to know what so what is the department name then you have to go for the join so i can go for source qualifier joiner or join joint transformation that we can see so this is the functionality of aggregator transformation okay so now in aggregator transformation we have some more concept in properties you can see option called sorted input so aggregator transformation normally will do two operations one is sorting first it will make the sorting algorithm it will sort the data then it will do the aggregation then only it will do the aggregation so normally this is what the two operation it will do if i want to improve the performance of the aggregator if i want to reduce the overhead of this aggregator transformation then you will have option called sorted input okay so here i am enabling the sorted input is nothing but if you are if if you checked showing like you&amp;#39;ve checked the input to the transformation is assumed to be already sorted by group by port so whatever the group by port we have selected like group by port we have selected department id right so informatica will assume that okay if you are enabling this property informatica will assume that okay i am giving some sorter data based on the department id based on the department id so just i&amp;#39;m enabling it but i&amp;#39;m not giving any sorter i have informed aggregator that okay i will give you sorted input but i am not giving started input what will happen it will fail the flow look at now it will fail the flow you can see here failed parent of this task space i am just executing it so normally sorter will normally the aggregator will do two operations sorting and aggregation but for this scenario i have informed aggregator that okay i will give you the sorted input don&amp;#39;t do this operations then aggregator will try to do the aggregation for the sorted data but i&amp;#39;m not giving any sort of data here right i&amp;#39;m just directly giving this because of this it will fail the flow look at now it has been failed you will be getting this expecting keys to be descending order that it will fail that okay what i have to do you have to enable before aggregator you have to make one sort of transformation i can select it remove this i can enable the sorted input you can go for the sorter we know already that sort of transformation we have seen in our previous session so drag and drop then you can take the salary up to here from here then you can take commission percentage to commission percentage so double click on it which port we have to select sorted we have to select the department ids descending order or sending order any anything okay i&amp;#39;m taking descending order the keys to be descending order apply okay now so if you are enabling that sorted input in aggregator we have to in we have to place one sorter before aggregator now it will succeed i&amp;#39;m going to refresh the mapping i&amp;#39;m going to start the workflow so this is the property of sorted input in aggregator transformation in many of the interview they will ask this question so what is the sorted input for sorted input option in aggregator okay if you are enabling this this property is used to do the it is used to improve the performance of sort aggregator if you are enabling this property we need to place one sorter transformation before aggregator so whatever the port we are making like group by look at here now it has been succeeded you are getting four records whatever the port we are using group by here so that port we have to choose here one more concept here if you are not enabling this group by if you are not enabling this group by simply or making some salary right sum of salary then if you are placing the sorter or if you are not placing the sorter it is not an issue so here just i have enabled the sorted input but i am not placing the sorter why because i am not using any group by port here if you are using the group by port it is necessary to have this order but if you are not using the group by port no need to have so salary also you can drag and drop then commission percentage you can drag and drop control yes i have enabled sorted input but i since i haven&amp;#39;t used group by port here no need to give sorter so this is another concept you can refresh the mapping you can control s you can start it you&amp;#39;ll be getting only one record out of it why because i i haven&amp;#39;t selected a group by right you&amp;#39;ll be getting only one record even though it will you&amp;#39;ll be getting it even though i&amp;#39;m using some filter i&amp;#39;m using some filter is somehow salary greater than 50 000 only right so i&amp;#39;ll be giving getting seven lakhs forty four thousand because of this it will will get one to one record you can check that record so that will be loaded here in the target so this is the functionality of aggregator transformation okay remember this what is the property of aggregator transformation i&amp;#39;ve interviewer if you ask that okay how will you improve the performance of aggregator you can place before aggregator you can place one starter transformation then double click you can go to the properties you can see here a caching directory you know right aggregator will do the sorting and then aggregation and that is what will use the cache directory data cache the aggregator will do data cache will have that data and index cache it will make the sorting algorithm index cache so that is a two cache it will it will do it so that is why the agree whenever you are using aggregated transformation in your real time project so definitely it will you will be facing some performance issue you can place sort of transformation before aggregator whatever the group by you are using so sort it accordingly so this is what the aggregator transformation i hope you are clear aggregator transformation kindly practice this aggregator transformation if you have any questions you can post then i will answer i will give some exercises to you on the aggregator transformation so this is an ex exercises for you load the department&amp;#39;s name where more than 30 employees are working i need like this you can take our our sources flat file so one of the employees as flat file and department as table then you can do the china since it is a heterogeneous right you can do the china transformation an aggregator you can do the aggregation you can load the data to the target like this department id department name and count of employees these three columns only i needed you can create one flat file or table with this three columns and load the data you can use joiner since both are like heterogeneous sources then you can use the joint transformation aggregator then filter to filter out more than 30 employees right so these three transformation you can use it and do the exercises here you can create one target table like department id department name and count of employees you have to load only this this result you should be getting it so complete this exercise definitely you&amp;#39;ll be getting all the stuff in a joiner as well as aggregator and filter transformation practice this if you have any questions you can post your questions then i will answer it hello friends welcome to nik iit academy and we are learning informatica power center in our today&amp;#39;s session we are going to learn router transformation i have explained the various properties of router transformation and the created example mapping explained real time usage so kindly watch till end of the session and you will get clear idea about router happy learning so router transformation is again it&amp;#39;s like a filter transformationally it&amp;#39;s an active transformation unconnected you look at filter transformation filter transformation will filter based on the one condition one condition or you can put the multiple condition with under r now the filter transformation like this consider this is the filter transformation if you are sending some record to this filter transformation then this set transformation will filter out the record for an example this is a customer table you are sending all the countries customer you want to filter out india customer alone you have to send it to the next level then what is the condition you can make country equal to india you can make so this is the way we will filter out if any other countries record is coming out coming up to this filter it will get filtered out it will not pass other countries out of this filter it will block the unsatisfied record on satisfied record in this particular filter so that is what the filter transformation will do you cannot take out of this filter transformation the unsatisfied records it will block you can get only the satisfied record you can put condition also like you can put under condition or condition condition you can put whatever the filter condition is satisfying that condition that conditions record only it will be passed other record it will get filtered out okay i want to take that filtered record also i want to put lot of groups i have to take for an example i have to take india separately you are separately uk separately in that case you can go for router transformation router transformation is nothing but you can pass single source pipeline okay single pipeline to multiple pipeline so that is a router transformation you will be having one default group so default group by default it will be added whenever you are creating any group in router transformation default group will be created so router is nothing but multiple filter conditions multiple filter in single transformation that is what the router transformation is very simple transformation whenever you are having single pipeline for an example you are having the single pipeline source you are getting the data like a card details debit card or credit card or anything you have to separately you have to segregate this into multiple groups so this is one group and this is another group you want to filter out you can filter out like this you can create multiple group so this is one group you can get credit card this is a debit card details and an example all the four columns will be there so here you can get all other remaining call cards default is nothing but whatever is not matching these two group it will be taken out of this default so whenever we are creating one group the other the default group will be automatically created so this group will be automatically created you are not going to create this default group you automatically create you can put this is like a credit card this is like a debit card okay this is another whatever it is not there like a card type other than this credit card and debit card all the record will be passed to default group that is also you can take it out but in in filter you cannot take the unsatisfied record so that is the advantage of router transformation you can trout the transformation is nothing but you can give single pipeline input multiple pipeline output so that is the router transformation okay we will see one example here i&amp;#39;m going to take some employees from different countries employees from different countries i can take this is the employee&amp;#39;s country&amp;#39;s information employees are there in different countries i can take in this way there are different employees like employee first name last name all the countries are there you have employees from india employees from usa employees from uk again uk then usa then india like this i have lot of employees are there like uh we have different countries employees are there in this table uk usa then uk like this i have yeah like this i have employees details they are from different different countries you have the country column another column is country column you assume that i have three target tables one table has i have to take all the employees who are in india so that i have to take it separately here in this particular target table india then i have to take another target table so in this table i have to take the employees blocks to uk so that i have to take it like a separate target here and then i have to segregate the other roof file us employees that you can take it separately so i have to segregate can i use here filter transformation no it is not advisable why because if you are using filter you have to use three filters i just use three filters one filter i have to use country equal to india another filter you have to use country equal to uk another filter you have to use country equal to usa so that is what you can take if any other countries are there it will be blocked here but you want to take any other countries to next level that you cannot do by using this filter transformation instead of making three transformations parallely you can put all the transformation in one single transformation so that is called router transformation so this is nothing but router transformation so router transformation will have single input single input we are passing input into the single input and this is one group one output group this is another group so this is another group okay taking that different group i&amp;#39;m passing i assume that this is the input group you have different current so only i&amp;#39;m mentioning four columns so this group we will call it as input group this will be where you can create multiple groups you can create for an example here you can write the condition country equal to india you can write the condition here here you can write country equal to uk that we can write here you can write country equal to you say whatever the filter condition you want to segregate the data you can put this way and you&amp;#39;ll be having one more port called default group right so this is the default group by default you will create we are not going to create this group automatically this will be created whenever you are creating one single group it will be created so you can take this group into so whatever the level you want to have you can take it next level so this is one group so i&amp;#39;m taking that into country equal to india so this is i&amp;#39;m taking and these groups i&amp;#39;m taking to country equal to uk this group i&amp;#39;m taking country equal to usa even if you want some different countries is coming like china or australia some other countries is coming up you can take this default group also to one more target so here you can have one more target table you can keep all this default group also on target so this is called router transformation router transformation it will segregate the data from one single group into multiple groups that is called router transformation we can put something like in in real time we can segregate the null values into one group and not null values into one group duplicate value into one group not duplicate value into one group odd numbers even numbers something like this you can put what i have explained right debit card credit card and the customers active status inactive status that if you want to segregate you want to apply some different different logic you can apply here so this is what we will use in the router transformation okay you can ask one question here if i have the same if i have the same country called india again also a country called india country also india what will happen you will be having the same record in this group also in this group also in this group also that you&amp;#39;ll be getting three times here so single if i have 100 records here you&amp;#39;ll be getting 300 records here why because here 100 here also hundred here also 100 so that will be passed so that might be interview questions they might be asking an interview so how will you solve this now we will see practically i have the table here in hr schema i have table select star from employees underscore country i have the table in place underscore countries you can see employee first name last name all these columns are there you can go to here you can see the department id i have one more column called country separate column i kept i have created one more column here i have mentioned like a country you have the india usa uk all these countries and the japan and everything i have mentioned but i&amp;#39;m going to take only three groups other than that i&amp;#39;m going to make it like a default group so i have created four targets in that course schema i have created four target you can see here i have made it like india t underscore employees for india then i have kept one more uh usa and then i have kept a uk right uk and i have put one more default okay as of now we don&amp;#39;t have any record in these three tables okay so first i&amp;#39;m going to the source analyzer i&amp;#39;m going to pull the source table employees underscore countries table go to source here import from database since source is a database table click on hr you can&amp;#39;t connect so it will be connected here you can take this host table go to source tables the table name is countries employees underscore countries i can take the source table so you can see the one more column i have just added country okay i can go to the target the target i have the four tables you can pull all the four tables at a time so i can go to the import from database you can connect to the target schema you can select all the tables the t underscore employees underscore default then india then uk then usa all the four tables just i have selected click on ok so it will be all the four tables will be imported now it&amp;#39;s importing you can see all the four tables has been imported okay now i can go to the mapping designer create a new mapping m underscore router transformation router underscore transformation you can create router transformation now you can see here we have the source table you can drag and drop the table called employees underscore countries and drag and drop and you&amp;#39;ll be having the source qualifier also and target table you can have this t underscore employees underscore all the four tables you can select all the four tables and drag and drop here i just drag turned up all the four tables that is what i have uh you can drag a drop at single time also then you can put all the the default table i&amp;#39;m bringing below that first table i&amp;#39;m going to take india so that is a table the second table uk then usa then i&amp;#39;m making that default so first i&amp;#39;m going to make the router so this is the router single to mini if you look at here this is a single group to mini so this router you can place here in router you can drag and drop the columns remember if you drag and drop it will have only input you cannot take output from here this is why because the input group you cannot take out you can see there is no arrow mark here single input group that&amp;#39;s all okay you have to create a router group output group go to the groups here it&amp;#39;s very simple you click on add new group just one group if you are clicking on one single group it is creating the default group by default okay first step you can make whatever the group you want i need three groups just i have created same like filter you can see the condition is true you can put the group name group name in i can put country equal to india so india just you can make whatever the group you want and then uk you want usa right just this is not condition this is just a group name just it&amp;#39;s a group name you can go to the first this is the condition here you can put the condition you can make country equal to india so it&amp;#39;s a very it&amp;#39;s a case sensitive you have to make correctly if you are thinking that it may you may get country in this way also india also you have to put upper of country equal to india so that if you are making like this then there is no issues in this okay so whenever the country is coming like lower or upper so that will be make it like it is converting country into upper then equating with uppercase even though the country is coming like lowercase it will be equated then validate should be succeeded then you can select this one this group click on ok go to second one you can make country equal to this group we can have the uk condition you can make you can ask question can i put one more condition yes you can put with and r so that condition you can make separate condition you cannot make separate group only you can create you can go with country equal to usa here other than these three countries will go to the default group so that it will automatically create if you look at here this is the input group and you can drag and drag below and then you can see you can drag and drop the columns so this will not have any input group you can see here so this is what you cannot see input to this particular group you can see the output only you can see the output port has been enabled you can drag and drop all this column to the first target table so that is your t underscore employees for india then you can minimize it so why because there is no space much right you can keep this target this target here uk you can little bit down uh go down and then you can see here you can drag and drop the uk also then you can have the usa you say here so that target table you can ask questions should i have always target table no need you can have different pipeline also here you can put send it to some different logic and then do some logic again you want to combine you have to do union operation that we will make after some time that we will learn after some time how to combine the segregated pipeline into single pipeline that will be our union operation so here we are the default group right even the default group also you can take it out i am not sure where it is right you can have this toggle view you can see this toggle over here right you can keep like this if i have more number of transformations are available you can toggle like this the window then i can see this this is one target table available you can drag and drop here and they can put the default group also here the default columns also here ctrl s it should be valid now even you can make arrange on right click arrange all it will be arranged on you can see that will be arranged on you can have all these four target tables and even it can put arrange all iconic so it will be showing an iconic way also you can see this way also so this is the single input group and multiple output group which is converting single group into multiple single source pipeline into multiple groups so this is the router transformation now look at here we will create one workflow then we will we will run and then we will see the result how it is segregating the data here in this folder i am going to create the workflow go to workflows create wf underscore router underscore transformation so that i&amp;#39;m just creating router transformation i have this session here you can select the associated mapping click on ok connect this by using link task double click on the session go to mapping check whether source is pointing to source and target is pointing to target but target is pointing to source itself you can change it to target connection i want to make all this for all this target table i want to make this connection so right click so this is the connection value right apply connection value to all instance if you are making that all instance will be passed to the same connection okay you want to do truncate and load yes the same way you can put truncate target table option what is that before loading the data it will this option will truncate the target table then load so same way you can put apply attribute to all instance instance in the sense this is a target instance right you can have the same and you can go to the config object you can make the stop on errors equal to one if any one error occurs it will stop the execution you can session execution and workflow it will fail you have one more option called fail parent if this task fails if this task fails it will fail the parent so that also you can enable it this is also normally we will enable see that these are all the some basic uh checks we will do control yes it should be valid now workflow should be valid right click here you can go to the start workflow if you are clicking on start workflow the monitor will be open you can see here one input group and multiple target if i have 100 records it will split up 100 records into four different groups now look at here i will show you the job is workflow is running now and go to get run properties you can have this it&amp;#39;s it got completed you can go to this one and then you can see here i have one source 107 record and that default is going with 19 other than this india usa uk we have 19 india it has been splitted into 36 uk 36 again usa we have 16 records and if you add all this 19 plus 36 plus 36 plus 16 it will become or not 7. this is what the applied rows and the affected rows you can have there is no rejected rows there is no error also in this particular session ok so now if you go and check the target table this is the target table right india so that&amp;#39;s india if you click on you can see all the india related data will be there even if you have if you see here since i have made it like upper of lower up right so that is why i got this india to the group if i&amp;#39;m going to remove that you will not get it i will show you so that i&amp;#39;m making it so usa yes india underscore usa in in oracle dot datas are case sensitive the same way informatica also case sensitive the group conditions usa and uk you can see here this is what then default group default okay so i&amp;#39;m going to show you this default group you can see another the other than india usa uk all other remaining countries have been loaded into the default group so this is what the router okay now if you look at here so this is a group condition right i&amp;#39;m going to change the group condition instead of upper-half country called india just i&amp;#39;m making country equal to india what will happen here if i go if i if i&amp;#39;m making that country equal to india then it will pass only the the upper case the lower case record right it will go to the default group you can see after creating the session i have i have changed the mapping level so you have to refresh it refresh mapping it has been refreshed that is a symbol here save it and execute again i am going to start workflow since we have given truncate and load it will truncate the existing data and load it again we are seeing here previously we were getting 36 record to india two is coming from india like why because other countries are like lower case money you can see other than this everything it&amp;#39;s like lower case whatever the lower case we have so that will become now in in this default group that will go to the default group right you can see the india india here this group will have only two day two records why because we have written in that way so that is why to say first side we are writing in this way okay upper of country that we are writing since we have written the upper case right so only two record has been loaded into the target table okay in real time this is what we will do single pipeline to multiple pipeline so this is what i have given x is one the router transformation you have to split up this employee into odd number your number this is very simple exercises and then you have to make the higher date is one of the column implies you have to take leap year who has joined on leap year who has joined our non-dpi so that is an exercise another exercise you have you have to take the first off record and second half record to to target table and you have you can split up this into like a duplicate record non-duplicate record also so that you can send it so this is what you can do the exercise if you look at here so this is the properties of the router transformation i hope you are clear about router transformation kindly practice the session and we will meet in the next session with another topic thank you for watching [Music] welcome in this session we will see uh the next transformation is rank transformation so rank transformation is an active transformation and connected one so we know already what is a rank rank it will do the ranking if i want to take top or bottom rank you can go with the rank transformation we have seen already that our aggregator transformation aggregator will give you the maximum minimum sum if i want to know i who is getting this maximum set of vr who is getting this minimum salary or i want to take some latest transaction data for each account something like that i want to take highest salaried employee in each department highest salaried employee in each country okay if i want to do like this ranking then i can go with rank transformation so before that we will see one example then we will go with rank if you consider this is the salary of input data all the employees one employee is getting fifty thousand another employee is getting forty thousand twenty thousand so if i have salary like this for employees selling like this if i want to do ranking based on the employee salary highest to least i will order right i will do the order by on this column normally i will write in sql we will write in this way like rank off over order by order by which column you want to do ranking then you will do the ranking either descending descending means highest to least so this is what rank column so r and k normally this is what we will do in our real time in our sql so something like if i want to do ranking i will use this way rank of over order by which column i want to do the ranking then i will make the order by class the same way here also i will do the ranking i will make order by on the salary from highest to least i can take the salary based on like this highest to least like 70 000 then next to highest will be 60 000 next highest will be 50 000 like this i&amp;#39;m making this so we know already what is this rank rank so i can take rank here if you look at here this is the data if i want to go for rank so this is the highest salaried employee i can make it like a rank spawn next highest will be 2 next highest will be 3 rank next will be 4 5 and 5 why because two employees are getting same salary then i will make five and five next employee will be ranked as seven only why because i have assigned two same rank right next will be seven the next will be eight here and this is also eighth rank employee the next will be ten here eight nine this is ten ten eleven then twelve twelve and twelve so three employees are getting same salary i will make 12 12 12 and next one will be 15 only 15 16 and 17. so how many employees are there so those money rank i will get it so this is called rank this is called rank dense rank what is this dense rank normally dense rank consecutive rank will be assigned for an example next round will be six only six next will be seven and seven the next will be eight nine then i will have ten ten 10 10 11 12 13. so consecutive rank will be assigned this is called dense rank row number is nothing but so serial number will be assigned for row number this is the row number right so row number if i want to take something like unique unique record based on the rank i can go for the row number so this is called rank dense rank and row number by using informatica rank transformation we can do only rank we cannot do directly dense rank by using rank transmission if i want to do dense rank then we have to go with some other logic we can use expression transformation to create a dense rank this one i will explain after some time or you can take it as an exercise you can complete it and also you can do rank on only one column but here it is very simple so no need to write any queries uh it&amp;#39;s very simple it&amp;#39;s for an example i&amp;#39;m getting the salary column it&amp;#39;s coming like input and output if i want to do ranking on the salary just to make it like a rank just you can enable this rank port just to enable it if i want to do group by on the department id the department idea column you can group by it just you can make group height and also you can see the properties top or bottom if i want to do descending order so descending order highest to least then i have to make it like top here so top means highest to least number of rank you have to define how many number so i want to make top five employees in each department then this is the way i can make it as i told earlier we cannot do dense rank by using rank transformation directly instead of rank transformation here we have to use expression transformation to do the dense rank so that i will explain after some time okay now i will take the employees table from the hr schema and load it based on the salary to the target core schema i will take employees table in the source hr schema and i will take t underscore employees table in the target based on the salary i will load it the rank okay we&amp;#39;ll go to the source here source we have already implies table target we have already underscore employees table i can go to the mapping mapping create a mapping m underscore rank underscore transformation so i can have a rank transformation drag and drop implies here drag and drop tms for implies here so this is the rank transformation you can see something like rank right you can keep the rank transformation here so by default you&amp;#39;ll be having the rank index so what is this rank index it&amp;#39;s output port by default will be available rank index is nothing but so this is called rank index okay this number if you are taking this column to the next level you&amp;#39;ll be getting this number okay so if you want you can take it i can take all the columns to the rank double click on the rank transformation it&amp;#39;s very simple i want to do ranking based on the salary i can make rank on only one column if you are enabling any other column based on higher date i want to rank then only one column will be ranked so i can just type i&amp;#39;m doing the rank based on the salary as of now i&amp;#39;m not doing any group by go to the properties you can see it will use cache memory why because it is doing the rank right top or bottom top means highest to the list number of rank i want to make rank for 100 records i want to do the ranking case sensitive string comparison we have already seen that what is case sensitive right so if i&amp;#39;m doing some rank on some string column then case sensitive it will compare okay tracing level data cache is nothing but so this is the data cache so informatica will have all these data into a data cache and this is the index cache so first of all it will arrange it then it will assign the index so that is why it will take data cache and index cache so it will take some time to do the ranking apply here okay click on ok so go to this so i&amp;#39;m taking all the employee to department id here if i want to know the rank index then i can go with this manager id just i&amp;#39;m removing this manager id i&amp;#39;m taking this rank index to the manager id you can ask question okay can i have this rank index to the manager id the column name is different why you are keeping this manager id column with rank index i just want to show you that how the rank index will be populated so that is why i have taken this rank index to manager id is a numeric column it is not necessary to have this column to always output you can leave it blank also no issues it is a valid now you can go to the workflow create an overflow so wf underscore rank underscore transformation you can create one transformation rank transformation click on ok you can click on the session icon keep the session here you can take rank transformation just drag and drop connected double click go to mapping source should be pointing to source target should be pointing to target so just we have to provide all the connections you can truncate target table and you can config object you can go to this stop on errors you can make it like if one error comes you can stop this workflow execution you can fail parent with this task fails if this task fails fail the parent you can enable all these properties you can start workflow so it will be started now you can see the rank transformation is running it will load only 100 records why because we have given top hundred number of rank is hundred only hundred records has been loaded there is no rejected record so throughput is 100 record per second so there is no record no there is no error can go to the target you can see here select star from t underscore employees table you can see all the target data has been loaded now right so based on the salary it is loading you can see from top to bottom it is loading i can make it&amp;#39;s a rank you can see here so one two three four five six seven and seven y seven and seven so here we are getting the same salary seven 7 the next rank will be 9 only rank will get skipped so this is called ranking i want to take only top 10 employees then i can go with top 10 only so i can make only top 10 it will load only top 10 employees for all the records so even if you have millions of record it will not have only top 10 records you can you can start workflow you can see here now it will have only top 10 records so it has been loaded you can see 107 recorded from the source and 10 records only loaded into the target table why because we are given top 10 only you can see the target table now only top 10 records loaded based on the salary right top 10 you can see here the rank index here top 10 so this is the rank i want to make each department i want to take top one so highest salaried employee in each department just to take it like department id group by then go to top one here so each department it will take top one employee it&amp;#39;s very simple so normally in sql you will write some sub queries or we will write some partition by department id order by right so but in informatica it&amp;#39;s very simple you just enable the port so if i have 12 departments you&amp;#39;ll be getting 12 records so this is what you have 12 records so this is the rank transformation you can see here so all the rank index will be now it will be one why because on the top one rank we are loading right so all the top one rank we are loading so each and every department we are loading the top salaried employees okay so this is the rank transformation so how it can be used in real time okay real time project how to be used uh so for an example if i have something data like this i have a transaction date some transaction table i have the transaction date transaction id of the each transaction then account number for the transaction name of the account holder credit or debit so amount balance amount if i want to take top latest file transaction for each account then how will you do latest file transaction for each account like mini statement i have to take our latest one transaction for each account since i am saying each account you have to take group by on the account number and rank on transaction id why because transaction id is unique id will be created so all the numbers are unique right rank on transaction id highest transaction id top one group by account number so i&amp;#39;ll be getting that okay latest transaction for each account so something like this you&amp;#39;ll be getting a queries like this for any interviews are in in real time scenario also so this is what rank will be used so as i exercise you can take dense rank so you can do the dense rank by using the expression transformation you have to create one variable port you have to do a dense rank you can practice that one and you can complete it so kindly practice this if you have any questions you can post so i will answer it uh welcome to the session uh the next session is like sequence generator uh this is the ninth transformation uh sequence generator is like a passive uh transformation okay so far we have seen all the most of the transformations like our active transformation we have seen only one passive transformation we have seen so that is what our expression so now we have another transformation is called sequence generator this is also passive one okay so whenever the sequence generator is nothing but whenever we need auto increment numbers so we can go for this sequence uh generator okay so we can go go with the sequence of numbers like one two three four like this sequence of number you&amp;#39;re going to generate then i can go with the sequence editor okay so auto increment numbers so wherever i need uh the sequence number for like a primary key or any surrogate key or anything then you can go with this sequence generator the sequence generator we have two output port so we have the two output port so one output port will be like one output port will be are the next valve the other one is kerval so this is next well port so this is kernel port so next value and current value okay so this is what the sequence generator will do sequence generator will create a sequence of numbers okay sequence of numbers uh see for an example like one two three four five like this whatever the number you want you can make it like it will incrementally it will generate the sequence number wherever it is uh needed okay you have many properties like uh you have many properties uh in sequence generator these are the properties okay so what is the first one it&amp;#39;s called start value so what is the start value from which value i have to start so that i have to give it in the start value okay start value i&amp;#39;m starting with some eight digit number okay you can you can start with any number like start with one also start to also like four digit number some any digit numbers you can make it that is what start value so you can make it like start value here so just i&amp;#39;m starting with eight digit number okay it will start from this number increment by so it will increment by one one one so one by one it will increment end value it will go up to this number okay one big number like 19 digit number it will go okay then current value so current value is nothing but what is the current value so while starting the value this is the current value you have to mention cycle option okay cycle option means what is the cycle you want to cycle it once it reaches this value yes so that is a cycle option okay reset means you can enable or disable the reset value so that means you are you are enabling the reset value or disabling the reset value reset means for an example today it is starting with number okay starting this number it will go by one by one right it will go by our next number will be uh next to one next number will be two so like this like this it will generate so this is a sequence of numbers it will go with one by one right so sequence of numbers it will go one by one then you can so for an example if you are making like reset option what will happen if you enable this reset option next time also it will start from this number only so this number only okay next time also it will start from this number only right it will generate like this number so it is resetting again tomorrow it&amp;#39;s again next run is running again it will start from this number only this number like this it will generate this is called resetting the option size cycle means if you are not enabling the reset and then you are enabling only cycle then next time it will start with six only okay six seven eight cut it in that way it will go the number of cached values what is the number of cached values for example number of cast values normally zero if you are making like 1000 year next thousand sequence of number will be generated and it will be kept the same catch memory so that it will assign very quickly so that is called a sequence of numbers here okay so wherever you need sequence of number you can generate it so that is what the sql generator to generate the surrogate key or unique key you can use sequence generator what is the surrogate key i will explain unique you know right already unique how the unique value will be generated so that you know i will tell but uh look at here one table i will connect it okay so if you look at here here you have the employee key right employee id and employee first name all these are like from employee id employee to department id i&amp;#39;m going to get from source in order to maintain the history in this table i&amp;#39;m going to use employee key as unique okay why because employee id i cannot make it like a history right if i want to make employee id as a primary key i cannot make it a history i want to i want to place some duplicate value so for that employee key i&amp;#39;m going to use it so that employee key i&amp;#39;m going to generate like something like a primary key values okay so look at here so mapping create so m underscore sequence underscore generator so i am going to take employees table from source but in target i will take employees underscore history table so look at here employees underscore history so i have one more column extra that&amp;#39;s all okay so employees underscore distributed so this is a table i have so i&amp;#39;m going with uh so employees underscore this one source qualifier then i will have this table employees called history from employee id to department id i will have it from here employee key i will make it like sequence generator okay this is the one so a to this one two three right this is a sequence generator i can keep here so this next valve i can make it here next valve okay so i told right sequence generator will have the two values one is next well another one is current valve okay so curve next value next next value to be generated so carver i have the current value click on it go to properties in the property start value will be so i&amp;#39;m going to have start value like four digit number okay increment by one by one end value like this current value should be if the start value is thousand current value or so thousand cycle option is just if i enabled i cannot give if if i&amp;#39;m going to enable the cycle option then i cannot give it for primary key why because primary key will not have repeated value right i should not give and i am not going to reset also number of casuallies i am not catching any values just keeping like 0 apply here ok okay save it now what is this meaning okay so this employee id to everything i&amp;#39;m passing from source this employee key i&amp;#39;m generating by using the sequential like next next sequence number will be generated from here okay so if you look at here i&amp;#39;m going to create some workflow so create wf underscore i am not going to do truncate and load okay i will remove the trinket and load why because since i am going to make a history just i have removed the trunk it unload first i am going to start it it will load all the 107 record right with the four digit number it will generate sequence of number it will generate so it has been generated now you can see so value is generating from here right employee key from thousand it is generating and it is ended with this number one one zero six so if i&amp;#39;m going to again i&amp;#39;m going to run it start workflow it will get succeeded so we are not going to do truncate and load just we are incrementally we are adding the data you can run it it is incrementing right the value is getting incremented the sequence employee key is getting incremented but i&amp;#39;m maintaining the history of the record right if the record is getting changed in the source so here also just i&amp;#39;m inserting it so the same record i can maintain here in this table with the new key okay so this is what in real time would be used to maintain the history and up but we will maintain some history we will use some uh slowly changing dimensions say cdn on that and we will see after some time okay so as of now this is the one thing can we have some another example for the sequence generator yes we can have okay for an example say for an example this is my first record second record from the source this is my record right it&amp;#39;s all the record so these records i have to load every fourth record to target i have to load every fourth record to target so how can i load every fourth record to target so this is a target table so this is one of the interview question they will ask in interview say for an example i have to load every fourth record right so here i have to this record so tomorrow the numbers might be different this number might be different the source incoming employee id might be different so i cannot go with some hardcoded value so for this i&amp;#39;m just generating some sequence of number like this one two three four okay so one two three four one two three four i&amp;#39;m generating for this id i am generating like this and i have something like fourth every fourth record i have to load it right so wherever the next value is coming up like fourth value i have to filter out i have to load it okay so this is something like next val port i&amp;#39;m going to generate this is next val port so this port is next val port i&amp;#39;m going to give from the sequence generator i can give right so what is the start value i can give start value start value is 1 correct increment by 1 end value end value will be 4 should i give cycle option yes i have to give cycle enable ok should i give preset yes you have to reset why why because if it is ending with something like one year okay one year then tomorrow it should not start with two here right so why because we are if you haven&amp;#39;t enabled this reset option tomorrow it ended with again starts with two only okay so don&amp;#39;t do that so like this you can go with so this is start value increment by end value all this okay now look at here i&amp;#39;m going to create a mapping m underscore load every fourth record okay i can take employees table i can take p underscore employees table i have to load every fourth record right so for that i&amp;#39;m i&amp;#39;m placing one&amp;#39;s filter transformation okay i&amp;#39;m placing one filter transformation so this record i have to pass at every fourth record so what i can do i can generate sequence of numbers so you can see here sequence generator right you can place a sequence generator you can drag and drop the next value to here double click on it go to properties the properties you can make start value equal to one increment by one end value will be four okay end value will be four current value and cycle option okay cycle option and reset option that you have to give that&amp;#39;s all so apply okay and this this will have one two three four one two three four right so wherever four comes i have to pass it to the next level only to take this one in the filter transformation you have to take properties you have to take only next well equal to four so next value equal to four you can ask question can i can i take next value to next level if you are taking this next value to target right what will happen here you you will have always four number four why because you are filtering that number only right so that is why no need to take this just along with this source data you are preparing one more column like next value equal to four so just i&amp;#39;m sending it i&amp;#39;m taking to the next level okay now you could look at here i&amp;#39;m going to the workflow so wf underscore the load every fourth record so you can take the load every fourth record target go down truncate target table option okay so look at now this will load every fourth record to target okay so sometimes interview they will ask this question if i want to load every fourth record how can i load look at here now it has been loaded right so every fourth record only it will be loaded okay so control enter you can see this one not three or not seven one eleven right so every fourth record is getting loaded it got completed right now it will be correct correct only alexander diana only look at now second time i&amp;#39;m running it&amp;#39;ll be different now this time it might be different [Music] so next transformation is union transformation okay union is a is an active transformation so this is also an and the connected one okay so it is exactly opposite to router so what router will do router will do router single group to many single too many right router will do single to meeting but this union will do many to single so that means multiple input to single output so that is what the this will do so union is nothing but you are having multiple input single output so single output you are taking multiple group input single group output so this is one group this is another group three groups i&amp;#39;m giving like input so one group first group input second group input so third group input okay the same structured table union right union set operation you should be having same structured one okay set operation should be same structured one so this is union transformation so same structure only have to give so union will have will give you union all in informatica okay so union will act as union all in informatica remember this that means union transformation will not remove duplicate you will get duplicate if i want to remove the duplicate then here you have to put one sorter transformation to remove the duplicate okay i want to remove the duplicate so union will act as a union all right so you have to if i want to remove the duplicate then you have to go with union all so why they are they have given like union all none then you have the starter transformation to create a union but if i want to have union r then i have to go with union transformation only so that is why they have given like union all here by using here you can remove the duplicate by using distinct option you know right in sorter you have distinct so it will come with union or duplicate also it will remove the duplicate so by this combination of union plus sorter will act as a union only union will act as union okay clear what router will do router is exactly opposite to union so router will do like this is router will do single too many right single too many single input you are giving so many output it will it will take so one two three four so again you can take again one two three four okay so like this this is default one if you want to take you can take so this is router right so router is nothing but single too many union is many to single so that is the difference between router and union remember this okay so i will give you this sheet so you can remember this union how the union will work so remember in in duplicate if i want to make remove the duplicate you have to put union plus sorter this is exactly opposite router multiple pipeline input single pipeline output multiple pipeline input single pipeline output okay for an example here i will explain what is this union will do so remember these two structures are same customer table and customer table so joins is different okay two different table you can do the joins same structure table you can do the union okay so same structure table this is union right so this is another table this is another table this is another table but same structure table so you can ask question here this mobile number here it is phone number can i do union yes column name might be different but data type should be same if you have two different data then you can combine it into a single resultant set that will be a union okay that will be a union so you have the customer table you have the location table two different table then you have to go for join but this is union union means it will remove duplicate it will not allow duplicate record for example here i have arun arun here so here also [Music] it will not allow this around this url will not allow okay so only one around this bottle also will not be allowed okay why because already bala is available so arun balarakis will be there then which is from here it will be there that is called union union all means so everything like six plus two everything will be there okay so that is you know okay so i will take one table here yes underscore employees underscore union underscore one so this table i will take look at here this has three records another table is union two same structure only but different records i have okay union two same structure only okay so union two i have you can take this data as flat file also more issues you have these records how many cuts are there you have three so four five six seven eight nine ten if you do union conservation you&amp;#39;ll get all the 10 records okay so 3 plus 7 you&amp;#39;ll get it in informatica also you have the target table t underscore union underscore union this is another table so you can have this source [Music] implies underscore union underscore one so i&amp;#39;m taking first table so second table okay two pipeline you can take are two sources also you can take no issues so instead of two sources we can take two pipeline okay then this is my target table i can bring the union transformation here so look at here union transformation if you drag and drop one input right it will automatically create the output group look at here it is automatically creating the output group right so we are not going to create automatically create i have taken just input group it has already taken that output group so you can take from here to here output but i have to take this also here double click on the union go to groups how many input group are there two two sources are there just click on this add new group that sound click on the add new group apply here click on ok you can if you go down so one more group will be created you can drag and drop all this you can you can connect it okay so control yes it is a valid one okay it&amp;#39;s a valid one you can see here the this is union okay we&amp;#39;re getting the target so this is the one ctrl s you can go to this workflow manager workflows create so wf underscore union underscore transformation okay we can go down trunk a target table option now we&amp;#39;re just giving so first it will do all the 10 records it will not remove duplicate see here union will act as union all now it will not remove duplicate okay see here did not remove duplicate right you will get all the to duplicate record also look at from making that ascending order we are seeing some duplicate record right duplicate means full draw duplicate okay so that is a duplicate it will take okay i if i want to interview they are asking how will you make this unionist remove the duplicate after the union transformation you place sorter transformation sorter here so you can keep double click so go to properties you can enable distinct apply okay now if you look at here so since i made it like sorter right you know already how the sorter will work distinct sorter distinct will remove the duplicator right so anywhere in the pipeline you can use it that is why sort of distinct we have given they have given just you can make now look at here i don&amp;#39;t have any duplicate records see here clear how the union will act as so and also they will ask question so why union is active is an active okay so you are giving some 10 records here 10 records here it is giving you 20 records right it is not removing the duplicate how this will be union will act as active so why because union transformation will change the row idea of the record okay so if any transformation changing the row number or row id of the record so that transformation we will call it as active so that is the reason union is an active transformation so interview they will ask definitely this question why union is inactive so why because union transformation will change the row id so that is why union is an active transformation hello friends welcome to nikeit academy in this session we are going to learn about lookup transformation so lookup transformation is very important transformation in our real time project so most of the mapping we will use this lookup transformation so what is lookup transformation so lookup transformation is to look up some values from other another table so whenever we are creating a logic a real-time project so in the pipeline if i want to look up some value then i can go for the lookup transformation we have two types of lookup transformation one is connected lookup other one is called unconnected lookup in this session we are going to learn about connected lookup in our next session we will learn about uncorrected lookup after that we will learn about when do we go for connected lookup and we go for unconnected lookup so what are the difference between connected and unconnected we will see in later consider this particular example this is the connected lookup example if you look at source we have that table called product i have product id supplier id product information all that related to product information so i have only product id but in target our next level transformation i need to have the product name by passing the product id to the table called product description i will take the product name and description okay to the next level transformation are in the target so in order to take this product id and product name either i can go for joiner or i can go for the lookup so if the table is there the source side i can go for the joiner if the table is there and source our target are if the table is flat file or anywhere in the pipeline if i have the table i can go for the lookup transformation so this is what i look up we can pass or look up a value and then we can join and then get the required details we will see this connector lookup in detail please watch this session without skipping it let us start our session [Music] so this is the 11th transformation so in lookup we have two types of lookup so one is connected lookup another one is unconnected local okay so what is uh lookup this is a lookup is an active transformation and we have connected as well as unconnected lookup transformation we will see the connected lookup first then we will see the uncorrected lookup what is lookup so we know already we have seen that join a transformation the joiner transformation will do the joining right so whenever i need to have the joining column i can go for the uh java transformation the same way the china but joiner need to have the joining column only in the sourcing but in the lookup we can have the the lookup table either it can be might be in the source or target or you can import it newly so that is what the advantage of lookup transformation connected lookup is nothing but it will be on the pipeline okay maybe on the pipeline so you&amp;#39;ll be having something like uh many uh mini port so in lookup we&amp;#39;ll be having input port output port lookup for tundra so we will see one by one and then if you look at here say for an example i have the employees table you are the employees table the same like joiner how we will do the same way look up also we will do think about via the employees table it has employee id so all the columns are there underscore id all the columns are there like up to department id okay so up to department id you have all the columns then here you have the t underscore employees underscore department is another table we have is called a t underscore employees underscore department another table okay so this has like so employee id all this data like employee to department id so then department name you have the department name then you have the location right location all the columns are there i need department name and location agree so what i can do either i can go for the joiner or i can go the loop so this time i&amp;#39;m going with a lookup so this is my source qualifier you know already so how the source qualifier will be working so this is my source qualifier okay this is my source qualifier then you are there you can connect from employee id to department id you can connect so from there you can have all these columns you can have all these columns like all the columns you can have so like this you can have these columns so you can have all these columns by one by one then you can you can keep this like if you want the department name i want i can go with the i have to look up the table right i have to look up the table so this table i can go with the lookup table so this lookup table we will call it as lkp trans okay lkp trans so this table we will make it like departments table we will look up the department&amp;#39;s table okay remember this same like joining same like joining we are looking at the department&amp;#39;s table in this table we have some columns like yeah the department id will be there right so in the lookup table you&amp;#39;ll be having department id then you&amp;#39;ll be having department name the lookup table department name then you&amp;#39;ll be having location id so all these columns will be there here so these columns will be there in the department&amp;#39;s table okay so now i&amp;#39;m going with this column drag and drop this column from here drag and drop this column from here so this column we will take it as a loop column so this lookup column we will call it as in underscore department id department id that we will make like department id then join these two these two columns department id with the department id then you can drag and drop the columns department name and location id clear now so this is what you have to to do the lookup so this is called the connected lookup because the table is there in connection okay so i will take the same designer i will take uh this table uh i&amp;#39;m going to create the same thing so m underscore connected underscore lookup connected underscore lookup you can see here source i&amp;#39;m taking the table same thing i&amp;#39;m repeating okay so whatever we have seen on the day just repeating that we can take p underscore employees underscore department this table i want to do the lookup so from employee id to department id i&amp;#39;m loading here from the department name and location id i&amp;#39;m doing the lookup so how to do the lookup just click on the lookup transformation here lookup table is department stable you can take either table or flat file anything is fine so you can take the flat file sorry table and from here you can pass the department id department id to department id i can i can match it these two double click on it go to condition you can place the condition like department id equal to department id here okay then you can take the department id sorry department name and location both i can drag and drop control yes okay you can arrange it all you can see here so we are passing department id from here if you do not want this manager id you can remove it you can remove it no issues so you can have this department id department name location id condition department id2 department writing okay apply okay yes save it so this is what the connected lookup and go to the workflow manager okay i&amp;#39;m going to the workflow manager so in the workflow manager i&amp;#39;m creating one workflow okay so wf underscore connected lookup underscore one i&amp;#39;m making okay then take this session connected lookup drag and drop whatever we have seen on the date just repeating it okay pointing target to target if you want truncate and load targets you can go to the lookup transformation you can you can do the look upon source player okay save it and then run the start workflow okay start workflow now so it will load the data it got completed you can see here you can see here you can have this table 107 into or not so okay so the data has been loaded now so you can see been loaded you can go and check here in the table so if you have this data has been loaded okay just it has been loaded with employee department name and location so for example i want to know i want to load only purchasing department where can i apply filter so can i apply filter in the lookup so if you are playing the lookup then it will become like a leftover join only so if i want to make sometimes interview they will ask how will you make lookup like inner join you have to make flat some filter here filter apply the filter so keep the filter here drag and drop all the columns from here to here and department name and location id you can keep it here wherever the department name is coming like something like finance department right department name equal to finance purchasing or anything you can make it like purchasing and then you from the filter you have to drag and drop to the next level seem like control yes valid refresh this refresh mapping save it start workflow and see now and then it will load on only six data six record only why because i have only purchasing department six employees are there so that is why it will do the lookup sometimes interview they will ask how will you change the lookup as inner join so this is what see here now six record only it&amp;#39;s loading you can see here now six record is loading right so this is what you can see this value so this is what the connector lookup with single table say for an example i want to do the connector lookup on some two or three tables for an example i have one more table called department underscore lvoc another table okay this table i have to load it i want to load all the department as well as location details also in the table i will go with one more loop lookup okay so connected lookup only connected lookup two tables two lookup tables okay first table i will take it like employees table only source table here i will take employees department and location targeting so normally i will give this as an exercise but here i&amp;#39;m i will do it okay so that you will get idea so how would it be so this table i&amp;#39;m taking so this table you should have created for your exercises like uh source qualifier exercise our joiner exercise you should have created in source i&amp;#39;m going to take locations or you can take it in the runtime also say for an example here first i want to drag and drop this table target table so what are the column needed from employee 82 department id i need all the column name location id state address city and the country so these three column is coming from locations table okay location save so what first i can do first i will do it like departments table lookup so whatever we have done it previously that we have to do so go to this first department stable i&amp;#39;m doing the lookup department&amp;#39;s table okay so look at now so you pass department id from here match this department id and this department id double click on it condition match it here you do not want this manager id you can remove it so you can bring this department name and location id from here you can pass it over here okay then i need state address city and country so from where i can take it this table these three columns i have to look up on the locations table right so how can i do the lookup on location table you have to go with location id so here i have the location id by passing this location id to next transformation i can get it say for an example here click on one more time lookup table click here so this time i will make import import relational table so where from the source is there so yeah the location table is there the source right so that is i&amp;#39;m just connecting with source so i&amp;#39;m just importing from source that locations click on ok so from here i&amp;#39;m passing the location id from here okay you can ask question can i pass from one lookup table to another lookup table yes you can pass it this location id this location id i&amp;#39;m going to match it i&amp;#39;m going to get this street address and the city and country so these three column i&amp;#39;m going to take it okay these three columns i&amp;#39;m going to take it the postal code i will remove and i will i will take it like state i will remove it i will remove i will i will keep all these three columns okay okay look up condition is location id to location so one lookup table to another lookup table so we can go here workflow create so wf underscore connected lookup underscore two lookup tables two tables so you can keep you can make here so only thing is here is so one uh extra thing is in the the lookup you have to provide the connection lookup you have to provide the connection this lookup table is in source site don&amp;#39;t forget this this is the lookup this is also source so this is also source this is also source okay how to give click on ok save it now if you run it will load the data into this table this table as up now we don&amp;#39;t have any data but if you run it it will load the data so you can see here 1007 recorder has been loaded so it will load it with the record details look at here so all the department name location id street address city country everything has been loaded right so this is what connected lookup two tables i hope you are clear on the connected lookup [Music] hello friends welcome to nikkei academy in our previous session we have seen about connected lookup in our today&amp;#39;s session we are going to learn about unconnected lookup in connected lookup the lookup transformation will be in the pipeline in unconnected lookup lookup transformation will not be in the pipeline if you haven&amp;#39;t watched the previous session on lookup connected lookup please watch the session and then continue this session so in our today&amp;#39;s session we are going to have a employee stabilized source department tables lookup table we are going to load department name and location id in the target please watch the session without skipping it let us start our session so connector lookup and current lookup unconnected lookup means it will not be there in the pipeline okay look up table will not be there in the pipeline same thing we will use but we will not put the connection so like this for the table but you need the these details right so definitely you need to bring you need to bring expression transformation so expression transformation is mandatory in this unconnected lookup okay in unconnected lookup expression transformation is mandatory one so i can take the expression transformation you take all the column from sports so up to m up to department id will be having it so department id will be having all the columns so this is what that all right you can go for expression transformation expression so what column you need department name and location id so that column you have to create from here so i&amp;#39;m creating go underscore department name output port you have to create s underscore output so that particular column so here also you need pro underscore location id so that column you have to create it from here location underscore id so here we have write some formula so you can take from employee id to department id right so you can take from here it&amp;#39;s from your employee id up to department id you&amp;#39;ll be having all the columns right up to department underscore id you have all the columns but i need the department name and location so that is what we are doing there so department name and location we will take all the columns okay so now if you look at here i need department name right so i have to write the formula so this is the formula remember so this is the formula what is the formula right colon transformation name then the joining field passing field which field you are passing so that particular field you heard right passing field means the joining field which field you want to join so that particular field you got right so whenever you are writing colon lkp in expression transformation informatica assume that okay you are going to do account so here you have to do the uncorrected lookup for department name colon lkp dot ltp trans the lookup table name from here you are passing the department id from here you are passing the department id so remember for the uncurrent lookup you can you can return only one port you can return only one port so to return a location id i&amp;#39;m going for the one more one more column call on lkp dot lkp trans one department id so one more lookup only i have to go with not on the same lookup so i&amp;#39;m taking the first look up here i am taking the first look up here so first look up will be your lkp trans second lookup will be lkp trans1 so i assume that this is my lkp trans so this is my lkp transform in the lkp trans we have to take same so which table is lkp trans which table i have to look up it&amp;#39;s a department&amp;#39;s table right department stable that is what the lkp transits department department stable only i have to look up so this is same department stable but lkp trans one we have to take same department table lkp transform in this table we can take department id department name then here we have to create our own column okay we have to create our own department id and then location so from here so first table we have to take from here a column from here as department id in the underscore department id so just we have to take this this column we have to create as input port input port we have to create it okay in underscore department so these two we have to join it make the condition of these two and take this fortress return port take this port as little board so here same way you have to pass in underscore okay partners for id you have to take these two columns to join again this time you have to take location ideas written book so for an example here i want the department name so what informatica builder call on lkp assume that it&amp;#39;s unconnected lookup dot lkp trans transmits it will go here it will pass this department id then this department id and the department table department id will match department name it will bring us return and leave it here so only one written boot you can take if you want to take multiple return port then you have to go for the concatenate function okay you can write so if i have only one table you want to take multiple return vote can i do it in the single lookup yes we can do by using the concatenate function so that&amp;#39;s one column we have to take from source and then so in expression we have to split it this is what we will do the unconnected lookup same we will do it in the informatica like m underscore and connected lookup okay so i&amp;#39;m just taking so look at here employees table t underscore employees underscore department table for uncut lookup you need to have expression transformation take the expression so drag and drop the column to here so double click on the expression you create whatever the column you want for next level i want a department name and location id you can make it like output also go underscore department name string and go for some 30 to 40 characters location id also number decimal so make this output port department name it should be old lkp dot lkp trans trans bracket but to pass from here department id okay department same thing you have to take same subnet will not be validated by because we do not have any bitcoins so here also you gotta take this but its lkp transform okay so we are passing the same department id and so this is for lkp trans this is for lkp transform that&amp;#39;s all you can connect it save it it is not valid now you can see here invariant so first lookup you can take the first one okay which table we are looking at departments table just you take the department&amp;#39;s table click on ok so first one you can you can have this department name and there is one you can remove this manager id from here location id from here in underscore department in underscore department ready okay written book department name is you can return it okay that&amp;#39;s all you can the condition you have to join department equal to department then again you have to go for the lookup one more lookup department&amp;#39;s table same department table this time instead of department name you have to keep the location id you have to create one more column that&amp;#39;s in underscore department okay so this is what input port whichever the port you are making like input port that port will be that value will be passed here and return what you take the location id this time so the condition you can make a condition here to part so then right click arrange all then the lookup will come into the front you come in the front so this is what it will be uncorrected on okay and current at home i will make the workflow wf underscore unconnected lookup and go for unconnected logo you try today that look up two tables so lkp trans you can go for source only so trans one also source only click on apply okay save this you can make start or two it has been loaded you can see here same thing will be loaded okay we have truncated and loaded right then same thing it will be loaded you&amp;#39;ll have that null value now maybe because previous one we have removed the null value but here we haven&amp;#39;t removed so if you want to remove then after this expression you can make one filter after this expression you can make one filter okay so that is what you can make the unconnected lookup hello friends welcome to nikkei academy in our today&amp;#39;s session we are going to learn about how to return multiple columns from uncharacter lookup normally we have already seen that unconnected lookup we can return only one written port per lookup say for an example even in the same table we want multiple port we have to use multiple lookup but we have some workaround to return multiple port so that&amp;#39;s what we are going to use please watch the session without skipping it you will be getting lot of stuff here this is the one of the question asked mostly in interview please watch it completely so that you will be getting all the stuff let us see let us see practically so we are using two look up right to look up transformation to return two different columns on the same table so first time we are returning department id second time we are returning location location id department name and location id is it possible with the only one transformation to return department name and location id it is not possible with only one transformation is not possible but we have some workaround so what i&amp;#39;m going to do i&amp;#39;m just copying this copy the same lookup uncorrected lookup multiple return port how to take multiple written book multiple written port is not possible directly so i am going to use 1d lkp trans i am removing this this particular one more lookup and just removing per table only one lookup we can use for this scenario also we can use only two lookup instead of five lookup we can use two lookup so 2 lookup for departments table these 2 one lookup another 3 is for locations right for location we can use only one so only 2 lookup we can complete how to do please look at here so this one same table we are looking up right same table we are getting it so we have some work around for this so what is it we can take department name underscore loc both both of the column in same port how to take see look at here i am going to use so this is i&amp;#39;m selecting from the department&amp;#39;s table right department table so i&amp;#39;m going to choose department underscore id comma is it possible to select a department name and location same column yes it is possible how to take you have to take department name concatenation concatenation location underscore id so location underscore id as this one see here department okay sorry this is a department&amp;#39;s table right department&amp;#39;s table so just i&amp;#39;m selecting it so this is what i&amp;#39;m getting so in one column i&amp;#39;m getting two values right one is uh department name another one is location we can return this but if i return like this i cannot i cannot take only administration i cannot take only location id so to have this segregation i will use some delivery here delimiter okay some delimiter i have used like look till the symbol i&amp;#39;m using is this one so look at here this one i&amp;#39;m using then using that select department ids department id so department id this one has location from this one so this is what i&amp;#39;m using it so this query we are going to use in lookup okay lookup override so how to use it make whenever you are writing this query right so make it like uppercase sometimes uppercase lowercase it will pass some [Music] issues so you can use this as an uppercase controls e i have used so where can i use it so here we have to use go to properties the properties lookup sql override right so here we can use it so this particular query will be executed first column is department id that will be assigned to a department id column second column will be assigned to department name location column single column okay it has single value so if you have many columns you can put all the columns in the tilt symbol we can return it in one single port single port only we can take but that we are taking in this particular column so we can go for more than 30 characters like 100 characters i&amp;#39;m going with clear on this okay so we have to receive this column here which port i can receive can i use this one no i will use one more port called the underscore department underscore location dept underscore name underscore location so this column i will receive it first okay whatever the record board as a variable port why i am using is variable port variable port for intermediate calculation idm passing but this time i will get department name and location okay department name i have to get it right so how to get the department name say for an example this is what you are getting as a result from this i have to get the administration from this i have to get a thousand seven hundred per location by using the tilt symbol i have to write common formula for all the sorry because i cannot use some number of characters for this alone then this is different this is different right so everything is different different that&amp;#39;s what we are used to till the symbol how to get the administration alone from here so you have to use substring sub str substring of the given string right single quotes comma one comma how many characters i have to take 1 2 3 4 5 6 7 8 9 10 11 12 13 14 okay one comma 14 i have to use correct from dual so administration i have taken but is it this formula will work for all the all the data so for this one is correct this one is one comma two right so to identify the tilt symbol i will go for the in string ins tr in string of this one where this tilt symbol is presenting so 15 to place right 15 to place so you have this is 14 to place in order to take this one where is a tilt symbol is present so you can use single quotes tilt symbol it will return like 15 so i have to use so here it is a 14 but it is returning 15. so what i have to do remove this 14 remove this fourteen you paste it here you make minus one okay minus one see here administration has taken correctly we are not doing any hard-coded value so instead of this we are going for any other data say for an example i&amp;#39;m using i t department right so here also i&amp;#39;m using id it will return it correctly so it will take that&amp;#39;s what we have written this formula is for department name okay i will write this formula for department name so instead of hard coding this what i have to use instead of hard coding this what i have to use this port name i have to use this port name you have to use clear okay so for this i have to take thousand seven hundred how can i take thousand seven hundred how can you take thousand seven hundred alone from this so you have to go for sub string of this one so you have to use instead of one you have to find out the tilt symbol correct from each character so that three symbol plus one i have to use not minus one this time plus one that&amp;#39;s all see you have to learn first substring and in string so i have already posted some video on substring and instinct please watch it okay please watch the substring and instrument so you can get it so this is what you can go for the substring and instrument options so this one you have to use it here i will take this one next this one i will use so instead of this i&amp;#39;ll go for this one for this one click on ok now play okay here save see we are used to only one lookup now okay so now i will create a workflow i will show you whether it is loading correctly or not unconnected lookup unconnected lookup multiple return port multiple return vote so i have given here you can see the target i&amp;#39;m just using this target look at now so underscore dept right this table so i&amp;#39;m going to start it will load the data okay it got loaded now you can check it whether it has loaded correctly or not yes it has loaded correctly for all the records right so this is what you have to use multiple written port in single unconnected lookup clear we have to use concatenate function to take that [Music] hello friends welcome to nikeite academy in this session we are going to learn about lookup policy on multiple match we have already seen connected lookup unconnected lookup how to create a lookup and how this lookup will be work everything we have seen now this is one of the interview question what is lookup policy or multiple match so because of this property right informatica look up transformation is called as active transformation it is an active transformation lookup why look up transformation is an active transformation because of this particular property the lookup transformation is called as active transformation so we will see what is that lookup policy or multiple match whenever we are creating a lookup table say for an example from the source we are the department id 30. from the source we are the department id 30 in the lookup table that means our departments table normally we should not have any duplicate value in real time project sometimes we will have some duplicate records it doesn&amp;#39;t have any primary key so that you&amp;#39;ll be having the duplicate value so if you have duplicate value our lookup table will go and check the data corresponding data for 20 we are getting the match and we have one more 20 on the same table so if you are getting multiple match which one i have to consider either i have to take admin or i have to take sales so based on the lookup policy we are to choose from here we do have four different options in the lookup what is that option use any value that&amp;#39;s a default condition use first value so first value means it will take from the last so lookup will create in this way so that&amp;#39;s what it will take sales use last value it will take admin and report error that means session will fail saying that multiple match found so this is the lookup policy and multiple match in addition to this we have one more option is called return all values on multiple match what is this so for one record in the source how many matches we are getting it all the record we will get it so 20 will get admin same 20 you will get sales department so one record in source we are getting multiple record in target so because of this property the lookup transformation is called as active transmission so now let us start our practical session how it will work kindly practice this session if you are getting any doubt or any clarification kindly raise your questions to comments i will respond to it and if you haven&amp;#39;t subscribed my channel so please subscribe the channel so that you&amp;#39;ll be getting all the update so we will see practically now so look up policy on multiple match so i will take this one m underscore lookup policy lookup policy okay i will take so i will take the employee stabilis source p underscore employees department as target so we are going to look up the department id from employees table department id from employee table i have to take department name and location id we know already that okay here we are the employees table so from the employees table select start from employees table we are going to have this department id so department id here okay so for an example department 800 you take so 100 you are having only one record that&amp;#39;s okay it will take 100 means it will take select start from departments departments table so if you are getting only 100 the department that&amp;#39;s working it will take finance department right if you are getting 200 here is it possible first of all in department stable it is not possible departments table is a primary key column it will not have any duplicate value but some table in real time project some table we will have a duplicate value also without primary key so that time so we have to take which one we have to take either finance department or any other different that&amp;#39;s why look up policy are multiple match so this time we are going to look up the flat file why flat file right in the flat file i have added some duplicate data so go here informatica 10.2.0 server infrastructure lkp files local files here i have i kept some departments same thing in the departments i the last record i kept it like duplicate value so 30 here purchasing department correct the same 30 i have made it like insurance department there is multiple match so informatica will confuse now whether i have to take purchasing or i have to take insurance per department name for same 30 right so look at now how can we do with the lookup so from here i will be passing all the columns i&amp;#39;m going to look up so this is lookup here source site so look the flat file is not there so just import import from flat file so you can make this one so this is what you can go for departments click on okay import field name from first line comma this is what so i can go for next column numeric text this is manager id numeric only this is also numeric only listed so connected department id department id match it so since it is a flat file don&amp;#39;t delete any columns okay since it is a platform don&amp;#39;t delete this column if you are deleting then location id the manager id will be placed to location okay don&amp;#39;t do that so which port i can take you go to the properties lookup policy and multiple match so what is on what is the one i&amp;#39;m using now use any value either it will take purchasing or it will take insurance for that department id 30. so i&amp;#39;m taking department name and location id so connect okay so connected we are connected to control yes it&amp;#39;s a valid node okay we have and click valid and go to workflow create a new workflow wf underscore lookup policy lookup policy on multiple match okay lookup policy i&amp;#39;m just creating the session lookup policy connected double click go to mapping so should be pointing to source target should be pointing to target and you can go little bit down you can see lkp translate lookup transformation since it is a flat file we will not have any connections so here one thing we have to make sure that we have to look up source file name we have to make that source file name this is a file name so copy this file name you have to give it here so you don&amp;#39;t forget to mention the extension file extension dot csp comma separated value file if you are not giving dot csv it will not identify the file name so we are given the connections and then truncate target table options just i&amp;#39;m giving click on okay so here i&amp;#39;m going to execute this particular workflow it will truncate the existing data and then it will load it we will see what is the value it has been loaded for 30 what is the value it has been loaded for 30 whether it is taking a chasing department or it is taking insurance department so we will see now go to the core tables you can see here so what is the department id is taking department id 30 is taking purchasing only it is not taking any insurance department correct if i want to take insurance department what i have to use look up policy or multiple match go here i am using use first value use first value means from the last it will take so it will take this time it will take insurance department for same 30 it will take insurance department it will consider us okay these employees from insurance department not from the purchasing department has completed now you can check the same same 30 it is taking for 7 30 it is taking insurance department now right insurance department so that&amp;#39;s what use first value you can go for use last value so that will take the first value first value in the file we have mentioned like purchasing department so now it will go with purchasing use any value means informatica will randomly choose the runtime either purchasing our insurance department if you are using first value it will take last value last well means first value in the order report error it will report it will fail look at here now it&amp;#39;s insurance right since we have taken last value it will take first value that&amp;#39;s a purchasing department it is taking for department id 30. okay i&amp;#39;m going to use report error if multiple matches found it&amp;#39;s a wrong scenario you you fail it you fail the session okay so that we have to do it so what i&amp;#39;m going to do so i&amp;#39;m going to make config object stop another equal to 1 so normally this will be taken for all the session right what is stop another equal to one if any one error will stop the execution control yes i am going to start it normally stop on error and this one we will give it for all the session it will fail now the session will fail saying that multiple match found report error see here it got failed saying that multiple match found in lookup that&amp;#39;s what it is failing then it will not load any data the target okay if multiple match found is failing right so another option is while multiple matches you take all the multiple matches like this so 20 you take admin also 20 you take sales also both you can take it so one record in the source two record in the target that that is what uh active transformation right so how can i take multiple all values is that any values for all values no so where can i find all values while importing the lookup you have to give all wells look at here while importing the lookup so while we are taking the lookup right that time we have written all values on multiple match this option we have to enable return all values on multiple match from flat file so we have to use same flat file departments click on ok same way we have to give all the values and everything click on finish department id department ready you can take this department id this department id match it if you go and check the properties now this properties is on disabled state look at your disabled state so once you have made it like active transformation then after that you cannot change it like a passive that&amp;#39;s why they have disabled it use all values okay this option we cannot select any other option here so once you have used all values that&amp;#39;s all so all the properties other properties are disabled because of source is a flat file lookup is a flat file okay apply here okay here then we can give here see here there is some twist what is that can i use directly from active transformation to target is it possible look at here the blue color you are saying concatenation disallowed on transformation this one is not possible we cannot take it here then how can i take it so we have to take these two columns right can i use expression no expression also we cannot take it so if i&amp;#39;m using expression see here i&amp;#39;m taking the expression here expression also same thing so we cannot use this is a this expression is a passive and this is a one we are getting it see here it is not connecting the department name and location it is not connecting so we cannot use expression also what i can do one workaround is i&amp;#39;m using as jaina transformation you can go for any other different logic i&amp;#39;m using as jaina transformation these column i&amp;#39;m taking from the source these two columns along with the department id i am taking from the from this by using this department id this department id i will join it so department id from source department id from here i will join it but still both are like same only see this department id this department is same only but the giant condition i&amp;#39;m giving it so whenever we are giving like this right whenever you are giving like this double click go to condition so here i&amp;#39;m using department id here i&amp;#39;m using department id whenever we are using like this look at now one changes we have to make so department id location video everything i have given so if i&amp;#39;m going to save this it will not be valid one see here it is not valid on mapping it&amp;#39;s very invalid so why it is invalid we have to check here okay each input of china transformation must originate from different source qualifier joiner or vsam [Music] since we are taking the columns from the same one you cannot take it take it directly so for that i can go and use sorted input option if you are enabling the sorted input then this will become a valid one but while in the runtime it will fail because of we have enabled the sorted input we haven&amp;#39;t used sorter right so that will fail so remove this use one circle transformation use one starter transformation to sort it out so what is that department name location you sorted based on the department id here so you give all the three columns to this three columns same way from the source also we have to give sorted input so remove all the columns use sorter here the source double click here you can go to keyboard you can use the same sorting columns so now you can use all the values from here okay save save this control yes it&amp;#39;s a valid one okay since we are taking this one since we are getting the same department id here right department id column from here department id column from here so you will get all the matches am i correct this department id we haven&amp;#39;t taken from that location sorry from the department&amp;#39;s file we have taken this department id from same column same table this department id we are taken from same table right same table that&amp;#39;s why it will go for cartesian product that&amp;#39;s why we&amp;#39;ll go for cartesian product i will show you so refresh it double click go to mapping since we have removed the lookup the lookup file name has been removed so you can take this dot csv so look at here i have given that file name and starting the workflow since we are taking the department id from the same table it will go for cartesian product so we have to remove the cartesian product how to remove that i will tell you this is one of the interview question they will ask definitely an interview please make sure that see how many record you are getting one not seven or not seven you&amp;#39;re getting three three one six so what we have to do you have to check here so for one record you are getting multiple records this time we are getting 30 insurance 30 purchasing for same record but multiple times we are getting correct multiple times we are getting to remove this multiple duplicate record how can i remove the duplicate record which transformation i can use after this joiner so can we use sorter to remove the duplicate so use sorter distinct to remove the full draw duplicate right so we can use this select see still one more workaround is there that i will tell you we will validate the data and then we&amp;#39;ll see now instead of three three one six it will reduce all the full draw duplicate it will delete since it is going for cartesian product we are getting like this full duplicate i will remove it look up policy on multiple match has been removed so you look at here so for each and every record we are getting only one record so what is this record we are getting for 30th department for one record in source two recording target correct so only two recorded one is for insurance one is for purchasing this is what we are getting it clear but still we have one uh logical error so this is six records right this is six records am i correct for 30 years department we have six six records for this six record we are getting 12 records output 12 records we are getting so source via 107 plus 6 record extra we have to give 1 1 3 but here 1 1 2 only we are getting why 1 1 2 why because it&amp;#39;s a due to we are making in joiner we are making like normal join if you are using likes master outer join master outreach left outer child if you are using master router whatever the record it has been filtered in the inner join that look at now you&amp;#39;ll get that null value record also see this 178 you are not getting now right 178 you didn&amp;#39;t get it see look at your 177 179 that null value record we didn&amp;#39;t get it but now we will get it one one three record see these are all the transformation you can check it and then you can see here now one one three and you can go here and check see based on the scenario we can use it the transformation and we can do the work around clear see for this you should know what is sorter what is joiner what is sorted and put in the joiner everything i hope you are clear on this kindly practice this look up policy on multiple matches kindly practice this then you will get all the stuff very clearly so thank you if you are clear then we will meet our next session [Music] in our previous session we have seen 11 transformation this is a 12th transformation these 12 transformation only frequently used transformation in our real time okay these 12 transformations are frequently used to transformations after that we are going to see five more transformation that is based on the project we will use like a java sql transformation normalizer transaction control stored procedure and all we will use based on the xml we will use these transformations are based on the project the requirement okay here we are the update strategy this update strategy is nothing but so so far we have seen only insert so we are going to insert the data right just we are inserting the data in the target we have truncated and loaded right but in our real time see remember we will for all the tables we will not do truncate and load only uh only staging table we will do the truncated node remember this in real time don&amp;#39;t enable truncated node for all the tables it is not advisable only staging table we have to do that truncate and load what is staging table i will explain the project explanation okay so now we will go for the update strategy update strategy it is an active transformation under connected one is an active under connected one and we will go for the data driven so dd is nothing but the data driven data driven means based on the data it will drive so whether it has to go for insert or update see look at here by using update strategy we can do four different operations one is dd insert okay one is dd insert dd update dd delete dd reject these four operations you can do what is dd dd is nothing but data driven so based on the data it will be drive uh whether it has to go for insert or update or delete or reject either you can use dd insert or you can give zero or one for dd update you can use one you can go for two or three all this so instead of deleting normally we will use these two very rare case very very rare scenario 90 percentage 95 percentage we will use this one only very very rare scenario we will go for dd delete and reject even instead of deleting the hot hot delete it will go for soft delete soft delete means instead of deleting the particular record we will go for the we will inactivate the record by marking some plugin that is what we will go for the update strategy update strategy okay so in order to use update strategy remember you should have a primary key in target table you should have a primary key in target table clear on this okay so now we will i will explain one scenario where we will use the update strategy uh look at this particular table see this is very important uh session in real time whenever you are going for any interviews or real time project so definitely this scenario will come into picture okay so definitely be using this so please watch carefully if you have any questions you can ask me you have source via the customer table you assume that source we are the customer table uh each and every time this is like your oltp system so oltp system each and every uh time you will be getting the data this oltp right oltp you&amp;#39;ll be getting the every day you&amp;#39;ll be getting the customer data in oil ap we have table called dimcast okay dim underscore customer is nothing but dimension table it is nothing but a dimension table okay this dimension table will have okay we have a primary key dimension table will have a primary key and attributes these are all the attributes customer name email mobile number city country all these attributes we have okay what is the scenario see if i have the record say for example very fast today you are running this workflow in our real time project okay very first time you are running this upload the project then we are we want to insert the record into the target we want to insert the record into the target we can insert it see look at here these are our new record today it has been newly customized has been added so what i have to do i have to insert this record into the target table your your data values okay how can i find these are the new record see this customer id is available in the old tv that&amp;#39;s your source table it is available but the same customer id is not available here okay you compare this customer id and this customer id by looking up the column okay you take source customer id and target customer id by lookup okay compare you&amp;#39;ll be getting thousand one in the source you&amp;#39;ll be getting null value against this thousand one if you are not getting any thousand one here that means that is a new record okay that record is not available in target so then you can mark this insert flag is true just we are marking is sorry this one is true we are marking this particular flag is true very first time see all the flag is very first time we are making like true so if it is insert flag is true obviously the update flag is false false means the record is newly going to the target we are not going to do anything okay if it is true then we are inserting the record into the target table just we are inserting it very first time all the record will get inserted into the target table very first time all the record it will get inserted into the target table okay this is the insert plug so what is the formula for insert flag if it&amp;#39;s if it&amp;#39;s not lookup customer id then we are setting up this just true if this flag is true we have to insert the record into the target just we are informing informatica okay these records are available in your source it is not available in target go and insert it okay so next time you are running it next time you are running it what could be the insert flag for all the record you have to set up whether i have to set up the engine plug if i&amp;#39;m going to next time if i&amp;#39;m going to run next time i&amp;#39;m going to run this record is already available here right should i make the insert plug is true if you are making then it will go for insert you&amp;#39;ll get primary key violation to fail okay don&amp;#39;t do that you make it like a false okay it also falls see all the record also you you have to make it like a false if it is false false then the record is matching with this record target record okay source record is matching with target record do i need to do anything if the source and target is matching do i need to insert the record or update the record no okay you assume that next day next day the session is running one particular customer has changed the mobile number next today are some sometimes later okay one particular customer has changed the mobile number you assume that what i have to do i have to compare each and every record one more time this record is record is equal then any of the column is not equal if any of the column is if any of the column is not equal then i have to any one of the column is not equal i have to update it okay i have to update it how can i update if any one of the column is not equal you adapt it so you can go for this record you take this record you take both are equal right both are equal then you are checking whether this is equal yes it is equal this is equal yes this is equal not equal if it is not equal set this update flag is true that means we have to make it like a we have to update the record that we have to make this particular flag is true if the update flag is true we are to update this record based on the customer id keep the customer id and update this mobile number not only this mobile number all the columns will get updated here simply update okay if i&amp;#39;m asking what is the mobile number of previous mobile number of vijay i do not have information right it will simply update the record is called a simple update simply it will update the record this is called a simple update so it will be simply it will get updated i have given this table just to create this table target also you create insert the record into the table to a target don&amp;#39;t insert it okay if the record here it is deleting if you want to delete it here then we have to use dd delete operation that i will tell you okay okay i hope you are clear then we&amp;#39;ll go for the table see here i have the table here source and target table customer update the other table called select star from yes underscore underscore update update strategy i have made it like this is what we are the data so one two three four okay we have four records i will go to the target table target table first of all i will manually trunk it manually trunk it okay if i have any data here i will truncate this table but remember target should have a primary key target should have a primary key okay so look at here select star from t underscore customer underscore update okay this is what we have now we can go for folder and we will create mapping m underscore update underscore strategy update strategy this is called a cd type one what is type one i will tell you a cd type one i will tell you what is this is whatever i have explained so far that is a type one so i will take the source table hr okay you go for hr table and then i will make employees underscore sorry customer underscore update source table source you may have primary key or you may not have okay but in target we should have a primary key target we should have a primary key look at here target we should have a primary key customer id should be a primary key okay so i will go to the mapping here i will take the source table source table i am taking here i will go for the target table t underscore customer underscore update target table i will go for okay so now i will take first of all i will look up the target whether this record should go for insert or update i have to look up for this i will take on expression transformation i will take all the columns i will look up the target now and look up the target again source customer id take the lookup here so far we have looked up only source table right so this is the difference between china and lookup one difference in lookup you can look up anywhere source or target anywhere so you can look up this customer underscore update it has been created against this customer id okay this customer id this customer id we are checking and then we are taking all the remaining say these columns you are taking from customer id to all the columns see these columns from lookup double click go to condition you make customer id equal to customerid clear lookup condition so we are comparing this customer id and this customer id if you are getting null value on this customer id we are making this flag as true if it is equal both are equal then other columns we are checking any of the columns is not equal we are setting up this flag so we have to create two flag one is insert flag other one is update flag so just i&amp;#39;m creating a flag here see these columns from lookup these columns from lookup so i will make lkp underscore so i&amp;#39;ll make lookup columns to just naming sake i will take lkp underscore to denote these columns from lookup is it is it compulsory to make lkp underscore no it&amp;#39;s up to you if you want you can make it lkp underscore okay you can keep the cursor here you can go for add a new port here i can make insert flag or update flag so insert plug is this one we&amp;#39;ll make this one you can go for insert flag [Music] customer id then it is true otherwise it&amp;#39;s false clear any doubt this logic is clear if condition the condition is true what i have to make just i&amp;#39;m making this flag is true okay otherwise false clear validated should be valid okay update flag i&amp;#39;ll go for update flag what is this update flag we go for if same way if this customer id your source customer id equal to target customer id if both are equal and under condition you have to make and if any other column is not equal okay and any of the column what is that column customer name not equal to lookup customer name or anyone right that&amp;#39;s what our email id not equal to this email id are this mobile number not equal to this mobile number are the ct not equal to the city are this country not equal to this country if any one of the column is not equal then i can make it like true otherwise make it like false clear see look at here if if these two are equal if any of the column is not equal any one of the column is not equal then we are making this column is true or false click on ok now click on apply ok so here we are the two flags based on the flag we will route the record whether we have to go for insert or update so we will take the router here we will route the record so insert one group update one group see these this blue color mentioned right these record only should go for insert or update by keeping this flag why we are not taking these columns these columns are not needed only these two flags is enough this flag will route the record from the source whether you should go for insert or update okay we are taking it how many groups should be created here two groups go to group the router create two group one is insert group update group see one group you are creating by default another group will be created first you can go for insert plug okay insert group okay insert group this is for update group you can go for update group okay either should go for insert or update otherwise it will be blocked here the router set will be blocked so which one will be true here you have to make insert flag that should be a true here see whichever the record is flagged with the true here okay it will be flowing to this particular group if it is all the record is false it will be blocked here itself the insert flag itself okay whichever the record is coming like update flag as true it will grow it will go to this particular group clear two groups will be created one group is inside flag other one is update group see insert flag you can directly go and insert in target insert flag you can go and insert directly but update flag update how to use the update strategy in order to update the record you have to use update setters this is what the update strategy this is what the update strategy you can take the update strategy here you can drag and drop all the columns from the update group from the update group then in the update group you can write the logic what is the logic either you can go for 0 sorry 1 or you can go for td update dd update can i write expression here yes you can write it through expression see for an example if the flag equal to something 1 you can go for a dd insert something like that you can write like this r you can go for dd insert so you can write some expression also here in the update strategy but i am not using any expression just i will be removing it okay we are we are validating only dd update dd update r1 i can use so i can take their target table see here i have taken the target table how can i take one more target table should i take one more time yes r you can copy this paste here ctrl c ctrl v ctrl c this target ctrl v here the name will be different if you look at here the name of the target instance will be different here t underscore customer underscore update here t underscore customer underscore update one but if you double click you can see the target table is tn for customer underscore update here the target table is t underscore customer underscore update both the tables are equal only okay so now you can join all the columns here it will be connected you can see it&amp;#39;s a valid one okay it is valid one this is what you&amp;#39;ll go for the update strategy update strategy so now you can arrange it arrange all to be arranged see one more time i will repeat first of all we are taking the source the source should be either inserted or updated how can i find by using this customer id by passing this to a lookup customer id customer addition equal and i&amp;#39;m making that whether the customer id is present or not the customer id is not present in lookup then i will make this flag is true if any one of the column is not equal i will make this flag is true then based on these two flag i will segregate these two groups one is for insert one is for update insert directly i will insert it update through update strategy dd update i will use i will go and update the record for update strategy you need to have one primary key column remember this okay please make a note of it everything any doubt here this is very very important so whenever we are going for next scd types right so a cd type one this is what we will use okay now i will go to the workflow manager to update strategy wf underscore update underscore strategy you take the session and go for update strategy double click on the column go for mapping so should be pointing to source target should be pointing to target but remember you should not truncate target table you should not rank a target table look up look up which one i have to connect target here i can go for the target okay target here so apply okay save this it should be valid now okay it should be valid now okay so now i&amp;#39;m going to run it as of now you will not have any record in the target so now we do not have any record so i&amp;#39;m going to run it all the record should get inserted now all the record get inserted see regard 4 records should go for insert why here zero there is no data for update right that&amp;#39;s what four record from source four record got inserted and zero record for update if you go and check here all the record will get inserted see here all the record has been inserted here in the target okay clear on this okay if i&amp;#39;m going to again i&amp;#39;m going to run the same workflow what should happen what is the statistics here four zero zero why there is no data for insert and update that&amp;#39;s why zero zero so why because source and target is equal that&amp;#39;s what it will not go for any insert update okay now i am going for something like some update i will go for i will source record i&amp;#39;ll go and update say for example this particular record i will go and update this email so manually i&amp;#39;m updating it you can update see who will update in real time the date the customer if the front end tool they are going to update it here it will get updated automatically okay gmail.com so the mobile number i&amp;#39;m writing here okay the mobile number so i&amp;#39;m writing the mobile number here okay i have updated the record click on concatenate sorry commit will be committed so make sure that you are committing the right times after updating it if you don&amp;#39;t commit it it will not be updated so if i&amp;#39;m going to run what could be the statistics here if i&amp;#39;m going to run again what could be the statistics 4 0 1 okay 0 means nothing is nothing is inserted one means one record is getting per update four zero one if you go and check here okay so this is what got updated in the target okay it got updated in the target [Music] welcome on second in our today&amp;#39;s session we are going to learn about java transformation this java transformation is used to implement the java code in our informatica project for some of the real-time applications right real-time scenarios where we will use this java for easy processing the data okay like encryption decryption encoding the data decoding the data like we need to generate some numbers okay transaction even number some numbers some specified to a particular number for this particular identify the each transaction in order to hide that numbers how the numbers are generated so we will go for something like a java transformation in order to create some kind of business numbers some encryption decryption all these numbers right you will go for this java transformation so either java will be used as active or password so what is this active right so whenever you are going to take the java transformation you can say it&amp;#39;s a active transmission or passive so you know active what is active if the java transformation is going to have multiple records in target that will be your active okay so single record multiple records that are the number of record is getting changed the target in the output then it will become active right so passive so we are going to give some passive uh transformation logic then we can take that java is passive so this is what the java so in order to use the java transformation we should know the java concept very well okay we should create at least we should take the java code from the online resources and we are to make use of that java code here just we will take one table so if you look at here i will take this particular i will connect to this repository i will take this particular table so i have that a data like this okay i will take this particular table the source select start from s underscore customer table customer underscore java so i kept for naming convention for to understand easily for java so you look at your customer id customer name customer data birth i have like this and mobile numbers so i have mobile number like this we have for single customer we have different mobile numbers okay so the use case is here i have to split this mobile number into three mobile numbers for this particular customer so one record in source three record in target one record in source four record in target so based on the mobile number i have to split so this is some use case so this can be achieved in expression transformation also okay you can take the expression variable port you can encounter this this particular tilt symbol and then you can go for some looping and then you can get it but it&amp;#39;s very difficult right looping concept in expressions lot of variable port it will take lot of memory also it will take a lot of time but in java we can implement easily so even i have given an expression i have given one logic for a looping concept right i used to call a new sticker the same scenario you can use this java java transformation so this particular data i&amp;#39;m going to load in this bit so this way i&amp;#39;m going to load it so how to load so for single source so like this i&amp;#39;m going to split up based on the mobile number how many mobile numbers are there three mobile numbers i&amp;#39;m going to make it like a three four mobile numbers four records so this is the use case so how to do it so we will do it and then i will truncate the target table so we will load by using the java transformation so if you look at here select start from now the table doesn&amp;#39;t have any data so i will take these two tables as source and target go to source take the import from database hr then click on connect and take the hr the source table i will take this table so target also we have already i have imported this particular table i will go to this mapping i&amp;#39;ll go to this create mapping m underscore java underscore transformation i will create this java transformation okay i will take the source table first take the source table first customer table target table here i can keep this is my java so click on java here so it will ask okay please choose the java transformation type either passive or active so once you have chosen after that you cannot change so for this scenario we have the active so i pick a single record in source multiple recording target so we choose our active so this is an active transformation now so normally what will you do you will drag and drop the port to here right so instead of that double click on this normally you can drag and drop also no issues see this is what the input this is what your output so you can keep the cursor here you can create so how many portions you have four port or you can drag and drop that is also fine so keep the cursor here on the output so create four four targets so what is this three targets are four targets we have to create here i&amp;#39;ll go to the customer id first one is customer id second column is customer name second column is customer name third column is date of birth so you can take data the fourth column is mobile numbers in source we have mobile numbers mobile numbers then what is the data type customer id i can have a as decimal customer name i can have it like a string under characters data birth date and time mobile numbers second string so source via the string you can have it like a string and customer id here so we can have this one is output right so i can have o underscore customer underscore id this is a decimal only you can keep this as decimal then this is like oh underscore customer name customer underscore name who underscore data birth underscore mobile so mobile only not mobile numbers just i will keep us mobile to understand okay this is a single number so this hundred characters data but this date and time i can have the mobile number also string okay 100 characters after that we will convert string into numbers so this is what we will create this particular columns and go to the java code so this is the java code actually you can remove this code these are all comments so based on the column right you will have the full code if you look at here based on the columns you have the full code so all the code will be created like this okay so all the columns will be have created like this we will create a code here so we have to create a code here so that code we should know something like i will give you one exercise for you like encryption decryption you can try that okay here i am going to create so first of all i am going to have a variable called string string variable i will take a strs variable i will store whatever the mobile numbers i&amp;#39;m going to give see whatever the mobile number i&amp;#39;m going to give right so that mobile number i will store it here mobile underscore numbers it&amp;#39;s a string okay that will be stored in str the variable okay it has some tilt symbol right so to take the tilt symbol i will take one temporary variable in the temporary variable i will store this value string null value and i will take that temporary variable called it m so this is temp then i will take a delimiter string to store this delimiter so what is the delimiter i have to store this is the delimiter right so this is the delivery so i have taken all the variables you can take some java code online also so whatever the code you want you can take it say for example i want to store the temp equal to these are all methods str dot string dot split i&amp;#39;m going to split based on the delimiter based on the speed delimiter i&amp;#39;m just splitting that string and storing the value into the temp variable but i have assigned the same variable as null i have split it based on the delimiter you can assign it to delimiter and then you can split or you can directly give the delimiter here then i&amp;#39;m going for the looping so here i will go for for loop for i equal to 0 less than m dot length eng th length and then i plus plus so up to this number i will go and then i will press plus okay then you can open those open curly braces close curly braces okay so you can open and close java we do not have any indentation but in python we have some indentation okay that we need to follow so i will map the columns o underscore customer id equal to customer id from source customer id so that we will use o underscore customer name equal to cust name o underscore dob equal to dob these are all the columns direct uh load so underscore mobile that is a target right or underscore mobile equal to what is that mobile we are splitting the temp variable we are splitted right so temp i i can store so semicolon you have to make this as a loop for this you have to use one method called generate row so generate row is nothing but for each and every record it will create a rose will be generated okay so all the semicolon and everything should be fine so once you are done with this you have to click on compile so compile it it should be compiled okay we are getting some error for i so what is this i whether it is integer or variable cannot find symbol right either you can define here integer i or you can initialize here itself teacher right okay so whenever you&amp;#39;re going to compile so here itself it will compile and then it will say like the compilation should be successful then only it will be valid so if you want to see the full code you can go to the full code the java transformation like this it will import all the packages okay here what are needed so public class so within this what is the variables private public all the methods it will be taken classes and it will create for each and every record they create like this this is the entire code so entire code will be created by informatica itself we are not going to create so exception model also apply okay then take all the four columns into this one target also you can take from here control yes it should be a valid one okay you can go to workflow manager so just i&amp;#39;m creating the workflow to run the java so workflow java underscore transformation java transformation one i will use i will take this particular session java transformation okay so link it so java transformation this is the one right we have created double click on this same thing we have to give the connections if you have any jar file in the project if you have any jar file the java executable file then we have to mention this the class name okay so here we have to mention the jar file so it is taking the default file no issues and apply but in real time project we have to give the java hr file here okay it should be a valid one now i&amp;#39;m going to run it i&amp;#39;m going to start workflow it will load the data into this particular table as of now no data is available it&amp;#39;s getting loaded succeeded two records in source seven records in target so three records for first record four records for uh second record so seven records has been loaded into the target so for this logic just i have created like this say i want to do some encryption decryption for our customer table or any tables so for employee tables i have to create some encryption id for product table or any tables accounts table account number the card number i have to make the card number as encrypted key so make this particular this scenario some exercise say encryption decryption by using java this something like you&amp;#39;ll have you&amp;#39;ll have a encryption decryption logic okay some java code will be there so this import and all informatica will take care you can have some encryption decryption some different logics will be there you can take any one logic okay so this is some method so no need to go for this this method you can go for some other method some other method lot of methods will be there okay top methods will be there you can go for any one method uh you can use it this method okay you can use any any one logic whatever the logic you need okay just to try this if you have any questions you can ask me i will help you out i will give you some code example code for encryption so that you can use that code you can give some account table you can go for the encryption decryption so if somebody else asking that question right you can tell them like for encryption decryption or encoding decoding some something like a conversion or this whatever we have seen for the looping and everything so here we can input take input all the input as one single string then each and every string will have some new line characters right so if new line character says encounter you can put comma here that&amp;#39;s all that&amp;#39;s very simple right so you can use this java transformation to implement this particular logic also either java or expression hello friends welcome to nik iit academy in this session we are going to learn about parameter and variables i have splitted this session into two different parts first part we are going to learn about parameters second part we are going to learn about variables so this is the agenda for today&amp;#39;s session so what is parameter what is variable what is the difference between parameter and variables how to pass a value to a informatica workflow or session through a parameter file how to pass a different value in the next run how to configure the parameter file in the runtime so that we are going to learn with example whatever the concept i have explained in our session i have given in the document it is available in the descriptions you can download the document for more details if you haven&amp;#39;t subscribed my channel so please subscribe the channel you&amp;#39;ll be getting all the updates whenever i&amp;#39;m uploading the new videos let us start our session so parameter and variable is nothing but if you want to pass any value dynamically to the mapping or dynamically to the session we can give us parameter or variable okay so what is parameter and what is variable so everything will be maintained like a dollar symbol if two dollar is there then this is like a mapping level parameter okay this is mapping level uh parameter single dollar is there this is a session level parameter okay uh session level parameter so you you have seen something like dollar pm src file directory right download directory it&amp;#39;s nothing but file directory is nothing but it&amp;#39;s a source file directory you can see here everything we have here we should have if you go here the source is a file then you can see something at dollar pm source file directory this is a single single param single value is there right nothing but dollar pm source file directory equal to the c drive informatica that path will have why they are giving like this it&amp;#39;s a parameterized value instead of particular value so if you are giving a hard coded value what will happen if you are migrating your workflow into other environment then that environment will will not be will not have this particular path okay and also some path some some particular path if you are if you are changing over the period of time you have to change it all over the workflows so instead of that if you are giving us parameters then definitely can able to give so whenever it&amp;#39;s required all right so this is the lpm level it&amp;#39;s a session level properties and so what is the difference between parameter and variables so for example you have the parameter the parameter is nothing but you&amp;#39;ll be having dollar dollar okay dollar is the mapping level parameter right so parameter it will not change during the runtime okay it will not change value during runtime and the session level parameters okay the session level parameters are the parameter if i&amp;#39;m taking for an example dollar dollar on value i&amp;#39;m taking okay so something like i&amp;#39;m taking some value any any values i can take for an example i&amp;#39;m taking dollar dollar salary equal to 1000 or 10 000 or anything so during the run time it will not change okay during the runtime it will not change so this is like parameter so once it has taken the value throughout the decision that value will not change but what is variable during the runtime it will change okay so variable value will change the value will change during the session right may or may not change okay so may change during the session runtime so that means if you are giving some dollar dollar count that value will be changed during the runtime also so this is called uh parameters and variables okay so like if you want to give like something like parameter and variable you can go with so dynamically if you want to pass a value to the mapping then you can go for either the parameter or variable variable is value will change during the runtime the parameter will not change so you can see the variable port also right you can see here i&amp;#39;m going with something like parameter so here i&amp;#39;m going to create a mapping called m underscore parameters okay so m underscore parameter there are two types of parameter values are there one is global parameter global and one is local parameter okay so what two types of parameters what is global parameter global means any session any workflow can read that okay that is a global parameters so any session or any parameter can read that value but the local one is nothing but it&amp;#39;s uh particular to the local to the that particular session you have to mention that folder name session name and everything then that value should be applicable for that session only it is not applicable for other session okay but you are defining a parameter or variable as global then others can use it so that is what the global variable so it can go with something like so this is the local variable and you can see here so look at here i&amp;#39;m going to take one uh so what it will do will automatically take select all the records from the source but i want to dynamically select the values of the source then i can go for something like parameters uh you can first you have to define that what parameter you want you can go here the properties i&amp;#39;m just making that something like even the scale query also you can give it like a parameter but just i&amp;#39;m giving var class only in the parameter just is generating the test here okay generate it and give this where class as dollar dollar something like dollar dollar where i&amp;#39;m just giving us this class is the dollar dollar if i&amp;#39;m not giving any value for this dollar data where the null value will be passed here so click on ok what is downloaded first of all we have to declare it go to mapping to parameter variables add a new parameter or variable it&amp;#39;ll ask you want to have download under where that that one we have used right so just i&amp;#39;m making the data whether it is a variable or parameter so i can take it as a parameter so i do not want to change the value during the runtime mostly we will create a parameter only here so string dollar dollar bar will be like i can give some hundred characters a thousand characters so it&amp;#39;s expression you can make like false okay you can give okay so you can make control yes it should be valid now then go to the workflow so you can create one workflow new workflow so wf underscore parameter okay parameters so i can give session so m underscore parameter click on ok then click on link task double click on the task go to mapping so should be pointing to source target should be pointing to target you can go little bit down you can see trunky target table option right and provide it and then you can apply click ok but we have provided a parameter right you can go to the c drive where we can keep the parameter file we can keep c drive informatica 10.2.0 server and you can go to the param file this is the folder and go inside you can create one file here param so any file name so that file name we have to give here so this is the path right you can see here this is the path in this path we have the param.txt right if you are using any parameter you have to give the properties the properties you can give parameter value right it has one option called the parameter file there you have to mention the parameter file name so click on parameter file name here just to click on ok so what is the file name param dot pht so that file name we have to give here so click on apply okay that&amp;#39;s all so dollar dollar where we have to define here go to parameter file double click on it so you can mention like it&amp;#39;s a global parameter if it is a global then you have to mention like this and dollar dollar value it&amp;#39;s a case sensitive so dollar dollar you have to give like this and you are giving where salary greater than fifteen thousand just making that class then apply ok so what informatica will do informatica while starting it right so if any parameter has been used it will check the parameter file it will go and read the parameter value since we have given us parameters dollar dollar where it will read the data now look at here so this is running informatica is running today i&amp;#39;m giving like dollar dollar where equal to where salary greater than something but tomorrow i may give some different one it got succeeded now you can see it should have taken three records from it if you go to the file the session log you can see so dollar dollar value will be presented here from employees where salary greater than fifteen thousand sorry the parameter value has been passed okay so now you can see go to this workflow parameter file i&amp;#39;m not changing anything in the session level or informatica level just i&amp;#39;m changing the parameter value okay so for example where job underscore id equal to sa underscore so your rf is nothing but sales representative i&amp;#39;m just giving it job id equal to this one so i&amp;#39;m just making like this is the parameter value i haven&amp;#39;t changed anything in the informatica level just i have changed that parameter file here start off below you can see now the characteristics changes right now it is reading 30. in most of the cases in real time the connection value also parameterized the connection value also parameterized so this is like where java id equal to this one okay it&amp;#39;s passing the runtime it is passing the parameter value okay so this is something like parameter global parameter you have to mention like global if you want to give local parameter you have to give this formula for example so global is nothing but all integration service integration service process and able to read that global parameter so you can give this uh this is the one normally we will follow okay something like folder name okay folder name this is entire workflow level folder name dot wf colon or floating are if we will give in this format folder name dot wf underscore name worklet name or session name or you will give this this this formula also the session names if you are giving like this this is the named session right for that particular session you have to give a value so we have to give for an example you want to give for this folder for this folder this session alone you have to something you have to give like this stuff global you have to give folder name will be so this is a folder name right training underscore file so this is a folder name that workflow name so this is our cloning this is session name for this session we are giving right so we have to give session name this session name here like this we have to give and so global like i will remove it now so training underscore five dot this workflow dot session name equal to this one for this we are passing so now you can check whether it is able to read the parameter file okay so 30 records is getting right so this is the this is the way you have to cue for the parameters for some other parameter value you have to give some other name you can give some other name also any path you can mention but you have to tell that information that okay this is the path this is a parameter file but the parameter name the file names vary correctly okay so if you are giving some different name it will get failed even some real time the parameter file also we will create runtime connection value we will create runtime so that is also possible here we are giving like this right but we will create the parameter file runtime also that script will create the file subscript will be there so that script based on the value it will create the file okay use some script to run this particular workflow so look at this is this is one of the wrapper script something like bash script right that&amp;#39;s what you have and this is comment commented out pm home is nothing but infra path so info path and info server info server name you have to give either w uid or prod informatica domain the domain name you have to provide workflow name the you have to give the name formatting a user so what is the user you have to give password you have to give the password or what flow flow name have to give what is the workflow name here so start then dollar pm home that is nothing but so this folder right we you remember that informatica folder so informatica 10.2.0 server infrastruct so here you&amp;#39;ll be having that this is the dollar home path so it will be mentioned in this admin level then you are passing this one pmcmd then start workflow these are all something like variable parameter passing you have to give all this informatica user particular password domain and everything folder wait time or pro name everything you have to give so it will fetch the workflow state uh status so it will take the status whether it is running or succeeded or failed it will take and then if the workflow status equal to zero then it should show you like execution successful alif we are making that execution has failed something like other than 0 we are getting some status like 1 or 0 or something like that 1 or some other value then you may assume that ok this workflow got failed right so this is what you can take this is one of the pmcmd command wrapper script to start the workflow is not to create a parameter file so this is the way we will create normally the parameters and the variables [Music] so debugger is nothing but step by step execution of the flow if i want to know the step by step execution of the flow then i can debug it session say for an example you are a developer you are developing the code and then you are you are completing the mapping and then you want to test it so you are running your session and then reading the data and then loading the data into target table but the session got succeeded but you are not getting expected result then what will you do so normally if the session got failed the session is failed you assume that session is failed if the session is failed you can check the session log so normally you will be checking the session log why it got failed and you&amp;#39;ll be troubleshooting it okay so white card failed and then you will be troubleshooting it if the session got succeeded the session is succeeded and you are you are not getting expected output okay i&amp;#39;m not getting expected output say for example your expected output should be so you&amp;#39;re expecting what a particular column expected output for a particular scenario like some value you are expecting 1000 you assume that one value okay but actual output you are getting is none say actual sorry output you are getting like a null value for this particular column it&amp;#39;s not a error it&amp;#39;s a logical so somewhere you are missing your logic then you have to find out right where you are missing the logic and everything so that you have to go for a debugging so you have to debug it so debugger is nothing but step by step execution of the flow okay step by step execution of the flow of the transformation flow step by step execution of transformation flow so pipeline flow you can say if you look at here so i can debug any session say for an example i want to debug this a cd a cd type too i want to debug it yes i can go for debugging say for an example if you are not getting correct output you can go here go to mapping debugger you can use start debugger so start debugger you can go for next so you you&amp;#39;ll be choosing which session i want to go for like you use an existing session instance use an existing reusable session create a debug session so i want to go for use an existing session instance so that means it is a non-reasonable session this is reusable session so you know already what is non reusable reusable session right you can go to next and then it will take that particular session so this is the session right you can go for next discard target data while running the debugger it will not physically write the data into target table okay while we are running the debugger it will show you what data is populated in that debugger but it will not basically go on right into the target table this database table it will not write okay this is what the debugger okay i can click on finish it will open two window so one is current instance this is what your current instance current instance means so this is one instance this is another instance this is another instance instance by instance instance means transformation once your debugger start running all these are disabled transformations everything is disabled state okay so now i will go with the mapping go to debugger so you can go for next instance that means next instance means next transformation it will move so first time using the next instance you can see here f10 right so next time you can press f10 so you look at here where is the session now session is there in source qualifier so here this is the session source qualifier okay if you look at your row number row id 1 that means first record so first record from source it is taking so you can see your customer id first name mobile number all this you are saying here and then you can this instance is in source qualifier so keep the cursor here you can press f10 here then it will take second first record to a lookup so look up also you are getting lookup key and then this is the customer id one from look up right so we are getting the matching then if it is matching an expression we are getting that new record flag is zero update record flag accounts zero so effectively today&amp;#39;s date end date is 31st so go for next one so it is getting filtered out okay it is getting filter out the row type if you check filtered out then this record is not going for the target okay if you are seeing anything like uh filtered out it the record is not going for the target okay how many targets are there two target right you can check two targets or three target what are the target here see for all the target it will go like a filtered one only so you can check here okay next instance the same way you can go for next next instance so how the data is flowing you can check one by one see here one by one all the record is getting filtered out it&amp;#39;s not moving on to the target so this is what you have to debug it so once everything completed automatically it will come out of the debugging debugger start a shutdown complete okay so this is what it will go for okay now i will go to the another mapping here it will i will show you what is the debugger and how it will be so you remember we have seen transaction control transformation right transaction control to generate the dynamic target right target dynamic target we have used transaction control say for an example this scenario i&amp;#39;m going to take how many transformation areas you&amp;#39;re using source qualifier is one starter is two expression is three transaction control is four so go here the same way debugger so i&amp;#39;ll go for the start debugger go to next you can go for use an existing session instance click on next it will go and bring the sessions so which session you want to use i want to use this transaction control this first session so i will take the first session click on ok discard target data you can click on finish then the same way two window will be opened one is target instance other one is current instance so you can keep just why i have taken this scenario i will tell you since we are using sorter here right so you can see here debugger you can go for next instance you can go for next instance if you are taking next instance row id 1 wires the instance suppose qualifier you can see source qualifier first record it has taken so you you can validate one by one record here so you are clicking on next instance f10 so f10 is means next instance it is not moving on to the sorter it is still available on the is still on the source qualifier you can see here second account has taken from the source qualifier press f10 third record fourth record fifth record like this it is going why it is not moving moving forward why because next transformation is shorter since next transformation is shorter it will not move so why because here we have a sorter will take all the record and then by using all the record it will do the sorting so that&amp;#39;s why this is not moving forward okay so here we have so this is what it will move and then you can you can go for f10 okay so if you have you can ask me so how many records are there one not seven record is there so one not seven times i&amp;#39;m pressing that f10 it is moving you can ask me one doubt okay if i have millions of record will you press million times no so that we will use some something called concept called breakpoints i will tell you what is that so this breakpoint concept is applicable in other like a java python also we have same way here one not six right look at here one not six next to one time i&amp;#39;m pressing f10 one not seven now the instance is available in source qualified if i&amp;#39;m going to press one more time f10 it will move on to the next transformation that is whatever sorter should look at your starter so on what basis it is coming like one not nine it is sorting based on the department id not department you can read it country only we are using the keyboard so based on alphabetical order it is making the country so c then i&amp;#39;m going for the f10 f10 means next instance so how it is moving to the expression okay then here transaction control it&amp;#39;s coming here you can see here it is generating the underscore country underscore canada.csb change flag equal to one that means true and then commit before this expression is making like commit before right so you can make one by one f10 again f10 so this is insert look at your row type is insert means this record will go to the target table okay this record will go to the target so next to the card it is taking where from the starter extra also canada then you can make it like f10 so it is moving right so here it is moving so since previous record this record is saved change flag is zero only previously change flag is one now the oh change flag also zero only it will go with continued transaction previously it was commit before right but now it&amp;#39;s a continued transaction so this is what it will proceed okay if i have millions of record should i go for like this one by one no i want to check only for india i want to check only for india how can i do it so go with debugger so in two ways you can do one is you can go further so i will go for stop debugger will stop it two things one is in the source qualifier you can apply some filter the source qualifier you can apply some filter you can make a bad country called india that that means it will pass only india only you can easily track it otherwise you can go for breakpoints if the source is flat file you can go for breakpoints see here go to mapping debugger edit breakpoints it will ask where you want to place the breakpoints i want to place the breakpoint in expression wherever you want you can keep it no issues i want to keep breakpoint in expression just time making expression transformation go and add the breakpoint so look at here add breakpoint click on condition so what is the condition so i want to make country equal to india right so i can make india country equal to india okay here so i have added on breakpoint you can see here so one stop button will be appear here whenever i&amp;#39;m going to start the debugger stop start debugger to next go to next and use this finish it up so this time you will find one stop button look at your stop button break points it is called breakpoints so what you have to do since you have applied the breakpoint right no need to go for one by one go for directly continue so what is continue it will continue all the record it will break only that india in the expression transformation look at here india expression transformation is it making so record is india right country called india so other countries it will pass so china and england it has been passed canada china england has been passed but now india is coming it has stopped now right so now you can go for f10 so one by one you can move so you can check only for india how the india is moving this is called breakpoints so if if somebody asked when will you use debugger so if i&amp;#39;m not getting expected output in the target if the session got succeeded then i will use the debugger in debugger we have two window one is current instance other one is target instance okay target instance and we are the breakpoints breakpoint is nothing but if i have millions of record i want to test it for a particular record i can use breakpoints i can test that particular record alone okay so this is what you can inform them if the session got failed what will you do yes check the session log okay how will you identify various errors if the while reading itself is failing then you have to check so why this has failed right that&amp;#39;s what you have to backtrack these are all the troubleshooting method continue means other record will continue it will stop in the whatever breakpoints you have applied okay from there you can go for next instance after see for an example one million record is there you want to particularly this particular recorder you have to check it continue all the record you break it only particular record so that&amp;#39;s what breakpoint will be used [Music] so welcome to the session in our today&amp;#39;s session we are going to learn about performance tuning see whenever in the real-time project right whenever we are going for any real-time project so definitely we will be handling huge volume of data so one thing is huge volume the other one is the number of columns will be there okay more number of columns huge volume and complex business logic so in informatica just here we are using maximum 5 to 10 transformations but in real time we&amp;#39;ll be having lot of transformation logic that time it may take some time okay to implement logic for all the records it will take some time to avoid this kind of scenarios we have to do the performance tuning so if we are using normally we are using these transformations in our informatica so definitely we will face performance issue these transformations like if you use aggregator yeah definitely we will face a lot of performance issue so why because aggregator will consider all the record to aggregate the values okay it will take all the records to aggregate the values and sorter yeah this is also it will consider all the records and rank transformation so all that all these transformation right so every record it will compare and it will give you the results so aggregation lot of cache memory will be used sorter rank joiner lookup so everything the catch memory will be used so that&amp;#39;s what if you are using these type of transformations so definitely you&amp;#39;ll be facing performance issue okay so how do you know this this session is running like a performance issue so first of all the session is long running okay the session is running so without so one thing is huge volume you can check the throughput if the throughput is very high and also at the huge volume it is taking a lot of time no issues so let it be running okay but the volume is very less but it&amp;#39;s taking a lot of time to complete and on its long running sessions or we might be going for the timeout issue so timeout issue is nothing but so whenever we are loading the data fetching the data from source and target we are loading so that time you maybe we might be getting the timeout issue that&amp;#39;s what timeout issue our session failed session might be failing due to the timeout issue and cpu consumption is more if you are getting these kind of issues so definitely it&amp;#39;s a performance issue either as a developer we have to find out our informatica admin team they will send you the mail saying that okay these sessions are getting lot of time are taking a lot of cpu so we are converting this particular session and then please take care please implement some performance shooting method okay they will inform us another way so if real-time project if you have many jobs are running okay thousands of workflows sessions are running if your if your customer is saying that okay if your manager is saying that so take any top five sessions top ten sessions and implement the performance shooting method something like a process improvement okay process improvement we so ourselves proactively we have to find with sessions that is taking a lot of time and so what is the reason it is taking a lot of time and we are to find out okay this is what it is taking a lot of time and we have to fix the issue how do i know that these out of these manipulations these sessions are taking a lot of time okay we do have metadata repositories okay so admin repositories in our system we have created right repository in admin schema or in fast schema the same way in real time the informative admin team you can get the repository queries from them so a lot of repository tables are there okay metadata tables are there informatica so we can go to the repository uh tables say for an example whichever the table we are getting like opb so this is the table starting with opp is nothing but it&amp;#39;s a repository tables so opv subject is nothing but so these are all the metadata tables if you look at here select star from this table it will give subject is nothing but folder name see whatever the folder which is available in our repository all the folders it will be displaying here okay you can create so this is the back end query we can execute it and then we can find how many folders are there so what are the different folders and when it has been created all this you can able to see here okay like this we used to have lot of tables we used to have the opb subject task instance run and the workflow so you can take this for plural if you look at here when the workflow has been executed and that you can see here select start from this one if you look at here so this is what you are saying all the subject id is nothing but the folder now folder id then workflow id so by using that workflow id you can find the workflow name and each and every workflow we have a separate employee if you look at here workflow ids what is the server id and when the start time the end time of this workflow and where the workflow log has been created okay then you can find where the workflow log has been created and you can find different informations in this table same way we used to have lot of tables here in our informatica repository i have given one repository query this query will give you the top long running sessions okay so each and every session it will give you the session run time we can find you can take top 10 or top 20 other sessions so if you look at here subject name so by combining multiple tables here so you can take this but this queries in in the unit also we can take it no issues like from these table we have joined and then see whatever the columns needed suggest we have taken that so subject name is a folder name workflow name instance name load date session start time session end time then you can find the duration of this session okay so this is what each and every sessions duration we can calculate if your if your lead saying that okay go and find top long running sessions top 10 long running sessions and try to implement performance tuning method for the test session okay if they are giving you this particular task then you need to talk with your informatica admin team whether a posting has been created or give them this particular query they will execute or you can give the same query to a dba they will execute it and then they will give you the result set so from the result set you can find whichever the session is taking longer time you can take five sessions or ten sessions you can go to the subject and go to the folder okay this is the folder workflow and instance and you can get the session log for all the sessions first of all if i want to implement any performance tuning definitely i have to analyze the session log okay so definitely i need to analyze the session log in order to analyze the session log i need to know which session i have to analyze right so this is what you have to get the details okay by using this below repository query we can get the folder name workflow name session name number of minutes the session has taken to complete so everything we can able to find okay after getting that we have to get the session log either through monitor or we have to get the session lock from unix server okay so once you have taken the session log in the session log you can find session load summary so if the if the particular session is taking longer time to complete so definitely you&amp;#39;ll be having the three threads this load summary you&amp;#39;ll be having the three threads one is called a thread transformation thread and write a thread so read a thread there will be idle time and busy percentage so what is this idle time means idle time means okay it is waiting for the data so it&amp;#39;s being idle and that busy means so it is using all the resource 90 percentage of the resources to read the data so the reader we are getting more than 50 that means a lot of resources it is using bottleneck we&amp;#39;ll say it&amp;#39;s a bottleneck we might be getting timeout based commit point in that target side in our right side why because here the source side it is completely busy occupied then target will be idle idle so it will be idle it is idle for 90 percentage here we see for 90 right here idle for 90 only 10 percentage right so this is what will be having the bc percentage if any if you look at here this is one sample session log in any of the session log if you are finding bc percentage is 100 that means there is a bottleneck where the bottleneck here the reader thread there is a bottleneck while reading we are finding lot of bottleneck issues if you look at here if the bc percentage is 100 then idle time will be zero that means it is not ideal completely it&amp;#39;s busy okay this is a writer threat lighter thread there is no busy percentage zero only okay this is our total id of time why because so here completely occupied right so that&amp;#39;s what we are finding the ideal time okay so this is what you will be finding the in session log you will be finding the bottleneck either source side or transformation side or target side this is called a bottleneck so mostly in our real time project where we will be finding the bottleneck right on the source side okay while reading the data we might be getting the bottleneck the second highest bottleneck in the target set okay it depends on the scenario which depends on the tables so which table you are loading and from which table we are fetching and databases also and server capability also it depends on these factors and the cache memory size and everything its depends first of all if they are asking an interview how do you find performance issues so i will be checking the session log where the bottleneck and finding that okay if the busy percentage is more than 70 percentage there is a bottleneck in the either in the source side or in the target side or in the transformation set okay if it is a source side it for the source side bottleneck so we have to analyze the source query source db source table everything we have to analyze it optimize the source code query so you can use some hints if it is oracle table try to use some hints or in the pre-scale post scale you can write some collect stats before reading the data you can do the collect steps on the source table and optimize the lookup queries if anything and the indexes say the source table doesn&amp;#39;t have indexes the previously create index the post sql we can drop the index and we can go for the partitioning so these are all the techniques we will you will be using in the source side okay we&amp;#39;ll be using in this whole set okay the target segment the target will be using the drop index so if the index is there in the target site and we have to drop the index load the data and after loading the data you have to create the index that we can use it in the pre sql post physical the next one is called partitioning so partitioning is nothing but the parallel processing that we will see now so that you will get clear idea so these are all the methods you&amp;#39;ll be using that to improve the performance okay in the transformation side if it is aggregator then you can use sorted input or incremental aggregation so that concept incremental aggregation i will tell you but you know already what is one way started input right so before rank transformation you can place sorter before joiner you can place the sorter before lookup you can place the sorter that&amp;#39;s what you can improve the performance of the these transformations creating the look of cache and everything okay so you are using these methods still you are not able to implement performance tuning okay you are not able to increase the session runtime okay so how to optimize them we do have two different methods to implement the performance tuning one method is called push down artificial the other method is called partitioning so definitely they&amp;#39;ll be asking an interview so what are the two different methods are there to implement the partitions so one method is called push down optimization the other method is called partitioning what is push down optimization the push down optimization or partitioning these two methods is licensed properties okay so separate license you need to get it for these in order to use these two methods we have to have a separate license the license property so we will call it as push down automation is nothing but pdo so if you consider the push-down optimization is nothing but converting the transformation logic into sql query and pushing down to the source or to the target either to the source or to the target or you can push it to both so that&amp;#39;s what the push down optimization if you look at here so you can take this is the left hand side is source left hand side is source right hand side is the target so here we are the target cycle this is what the source side was stable so this is the transformations we do have transformations here we have the transformations here we have the transformations a lot of transformations are there and here we have the transformations okay here we are the transformations and here this is the target so left hand side you have that source on the right hand side we have the target okay so in between we have the transformation logic see all this will be taken care by informatica so the data is coming to the transformation source qualifier after that the data will be moving one transformation to another transformation right see from one transformation to another transformation the data is moving like this and informatica will take the data from source that is source qualifier it is taking right then cache memory the based on the logic it will transfer the data and then it will load the data into target so this target it will be loading okay this is source side this is target set okay this transformation logic where it will be performed in informatica level okay so instead of doing this transformation into informatica level we can convert the transformation logic into sql queries so if it is expression our filter our aggregator rank or anything you convert that into sql query we are not going to convert informatica will convert it okay convert this logic into sql query and pushing down to the source side or target side so if you look at here so it traverse from source traverse from source and it will push it will try to push try to convert this transformation logic into a scale query so as much as possible it will traverse it will convert informatica will convert it this transformation logic into sql query under pushing down to the source side as the select statement as the select statement this is what push down optimization on the source section okay the same way from the target site it will traverse okay the transformation logic tries to convert these transformation logic into sql query under pushing down to the target section so what is the query here insert so it will convert insert or update okay so either insert or update so it will convert this is what it will go for so you can do both both means like it will try to push the logic from this side on this side so remember if i&amp;#39;m implementing push pushdown optimization then the source or target should be a table it should not be a flat file if source side is flat file if it is a flat file right say for an example this is the source side is flat file we assume that this is the flat file so then i cannot implement piston optimization on the source side if it is a target is flat file then i cannot implement push down notification on the target side but still you can use it on the source side so either one of the source of target it should be a table relational database table so this is what we will implement so where we will implement right so definitely be having one options in the session level just you have to enable that so we should we will not have here we will not have here but in real time we will be having it okay so if you look at here source side push on optimization target push now automatization and full push on automation so if it is a source side it will convert into a select statement okay select statement the integration service generate the select statement as per the transformation logic for each and every transformation it can push into the database we are not going to write any sql query it will automatically write target side yes it will convert this sql query like an insert into insert delete update statements this is what it will pushing down to the target segment so normally daily statement you will not use insert or update will use normally you can go for both so where it will be available if you go to the any session right if you double click on the session you can go to the properties in the properties you can find push down optimization okay this is what the bootstrap optimization you can see here you&amp;#39;ll be having option called source to a source to target to full or you can pass this value as parameter so you want to use a source or target side or full it&amp;#39;s based on the logic bottleneck we can we can give the data the runtime okay this is what you&amp;#39;ll be having it once you have selected so this one then if you go to the properties here so you&amp;#39;ll be seeing the mapping pushdown optimization if you click on this you&amp;#39;ll be having view post optimization if you click on this you can able to see the queries we are not going to write any query informatica will write the queries okay so this is what we have to implement the push down optimization methods so just i have given the separate documents already please go through the document you will get clear idea about personal authorization if somebody asked an interview so what is pushdown appreciation have you implemented in your project yes pushdown optimization is nothing but it&amp;#39;s a converting business logic into a sql queries and pushing down to the source or target or both it&amp;#39;s a one of the performance improvement technique i have used in my project whenever i am facing some issues on the source side or target side as a trial and error method i will implement custom optimization i will check that okay whether it is improving the runtime of the particular session if it is increased then i will use this question optimization if not i will go for the another method called partitioning normally in most of the real time project we will go for partitioning whenever we are finding performance issue so we go for partitioning so partitioning also a licensed property here in our in our informatica we do have partition we will do it practically so partitioning is nothing but we can go to the informatica then we can implement some partitioning so partitioning is nothing but it&amp;#39;s a parallel processing this is a parallel processing so we do have normally we have only one parallel thread like we have read a thread read the data from integration service we have the three threads the handles so reader thread will read the data from source transformation thread will transform the data write a thread will write the data into a target okay this is what we&amp;#39;ll be having the three different threads okay then we&amp;#39;ll be having if you are going to increase the number of thread parallel processing parallel processing in the sense we are increasing the number of thread so instead of having one reader we are going to make three readers that means the same work we are splitting the work to multiple persons right so if one person is taking 30 minutes to complete if the same work given to three three members then it will be reduced to 10 minutes right the same way here also instead of one thread we are going for parallel thread how many number of thread here we are creating three threads three parallel threads first thread it will run second trend it will run parallely the third thread will run valid these three will run parallel simultaneously on the cycle based it will run so fast reader thread will read the data and transformation thread write a thread will wait for that it will read then next instance next cycle next instance the transformation thread will transform the data at the same time the reader thread will read the data second data okay this is what we&amp;#39;ll go for the threads number of threads parallel processing sequential okay this is what the parallel processing the same way reader 2 will read the another data theta 3 will read the another data this is what number of thread so first thing is we need to consider three things whenever we are going to implement partitioning you can also explain the interview so whenever i am going to implement the partitioning i will consider three things one is number of partitions number of partitions means number of thread so here for an example i am using three so this is the default normally we will go for three threads parallel threads you can go for three or four or five so or six or eight or ten also normally we used to go up to ten ten different parallel processing okay not more than 10 if you are going for more than 10 you need to get approval from your informatica admin team why you are going for more than 10 10 parallel trade why because it will take a lot of resourcing okay resource informatica resource it will take so that&amp;#39;s what we should not go normally we&amp;#39;ll go for three threads or four or five okay maximum that&amp;#39;s what we&amp;#39;ll go with some some cases some project they will go for even eight or ten also it&amp;#39;s based on the project and capability of the server okay then we are the partition types what types of partition we are going for and where the partition point is created number of transformation if we have more number of transformation where we are creating the partition point that is also very important okay number of partitions here three threads partition type we have different partition types look at here so one is tp level partitions so if the database table has partitioned then informatica we have to tell that okay database level partition already done okay not in informatica you have to do it on the database level you have to do the partitioning on the database level then you can do it on the informatica level and range partition so if you have the data definite range from 1 to 10 10 to 20 20 to 30. so like this you have a definite range of data some product ids or department id or anything you know only these ranges you will get then go for range partition normally you will go for pass through partitions so this is what default partitioning is called pass through partitions even one single partitions we have the pass-through partitions then we&amp;#39;ll go for round robin hash auto key hash user key so round drafting is nothing but based on the threat availability it will use that round round-robin phases question okay then you&amp;#39;ll be having the hash auto key hash user key so this will use hash algorithm whenever you are using this kind of transformations right so you can go for hash user key it&amp;#39;s a hash algorithm so each and every record will generate information will generate hash key and based on that we will be doing the partitioning okay so now we&amp;#39;ll go for so one type of partition we will see that pass through partition if somebody asked so what is the default partition you will be using in your real-time project you can tell them i will go for pass through partitions normally we are doing a lot of parts through partitions in the real time project okay so if you go here i will open this i will show you how the password partitions will be takes place just i will take one table so you can take anything this time taking the employees okay i will take employees data here so here this is what a session can go here mapping you can see the partitions here or you can go to the partitions how many partitions are there we have the one partition so only one partition is there you can go to edit partition point so this is the edit partition point you cannot delete partition point now why because it&amp;#39;s only one partition point is there you cannot delete it see the re-partition point must be specified at this transformation we cannot delete even one partition point okay at least one partition point should be there is there it is the pass through partitions look at here the pass through partitions okay then you can go to the edit partition point so i want to edit the partition point in three partitions like three partitions are four or five so what is this partition type so based on the scenario how many partitions are there we have three partitions we have either you can go for pass through or key range or you can go for database partition here i am going for the pass through if you know the ranges then you can define the ranges also so just click on ok this is nothing but order of the partitions which partitions we have to order then this is what i can go for hundred percentage okay this is it&amp;#39;s nothing but window size okay then you can go to the transformations the three partitions will be there here so in the sql query are in the source filter so we have to segregate the data based on the partition that we have to give so how can i segregate the data so here i&amp;#39;m going to use some some common method if i have a unique key just i&amp;#39;m giving the unique key here employee is the unique key right so based on the unique key i&amp;#39;m going to do the partitioning so employee id just i&amp;#39;m using the mod function mod of employee id since i&amp;#39;m having three partitions i can use three mod of employee comma three if it is equal to zero then i can use this whatever the record it&amp;#39;s coming employee id we are dividing into three if you are getting remainder is zero it will go into this partition and equal to one equal to one it will go to this particular partition equal to 2 it will go to this partition so it&amp;#39;s mod is nothing but a modulus function i&amp;#39;m i&amp;#39;m making that division of employee id see for example employee is 100 if you are dividing by three so what is the remainder it&amp;#39;s a one right so one so that one that hundred employee it will go to this particular this particular partition only two so whichever the record is coming like employee two remainder then it will go to this partition this partition like this data will be moving from one partition to other partitions so this is what we are going with the partitions okay then i&amp;#39;m going with target no need to do any any changes just you can give all the details then apply okay if i&amp;#39;m going to run this you can see this normally this is what the partition will go for pass through partitions if you look at here parallelly it will run three threads okay if you double click on this normally we will be finding that source and target status right okay surprise that why there is no node or apply node or anything you can see the plus symbol here if you click on the plus symbol you can see the partitions number of partitions the 107 record is equally partitioned in multiple layers right so you can see see you you will ask question here the throughput is very less instead of high so still we are having like one minute to uh one second to execute right so previously one knots on our record also one second now also once again then what is the use so if you have huge volume of data then this will increase okay while starting this is what the throughput after that it will get increased by while session running okay it will get increase automatically so that is what it will be running so even lot of sessions we are doing performance improvement method in our real-time project the session is taking a lot of time okay based on the source data target data we will go for the partitions like this okay so this is what one of the partition method so you can go for the other method is called instead of pass through we can go for some range partitions so what is range partitions here so i can go here so instead of pass through i can go for range partitions click on the partitions here edit partitions so instead of this pass through then go for key range i will delete it to go for key range i&amp;#39;ll add the key range here apply see here if you are applying it you will be having the key here edit keys right so you can go for edit keys which key i want to go for employee id or department id or whatever it might be so you can use some keys so what is the key range you&amp;#39;ll be getting it so employee click on ok start range so employee id from 100 starting from 100 right so you can take from 1 also from 1 to 140 okay from 140 to 180 okay from 182 so you can use this 250 so like this i&amp;#39;m just making that we will check it how the data is flowing so our plane okay it should not get duplicated so make sure that it should not get duplicated okay so no need to mention this this logic here just we have made it like here no need to mention so you can remove this so just we have applied already right the ranges we have applied i don&amp;#39;t need to mention anything here okay so this is what partitions you can apply save it okay then run it if you look at here so this time we are running on the range partitions we are giving some definite range but make sure that the future it should not change the range should not change that&amp;#39;s what you have to go for the range partitions in future if the range is changing go for pass through partition that&amp;#39;s the best method okay normally you go for best method you can see here 40 40 and 27 okay record it has been 107 record has been splitted into multiple layers so this is what you can find the range partitions and if you have any other transformations right say for example i&amp;#39;m using sorter starter transformation sorter as i told if you are using any of this transformation definitely you can go for the hash auto key hash user key if you double click on this you can go to the mapping you can see here here you can see i can go to the partitions you can implement the partition point so partition point is nothing but so here we have one and two right we can add partition points here also just we can add it so i want to add three partitions so here i want to go for hash auto key hash auto key look at here hash auto key three partitions now here what is the partitions pass through partitions source side we have the pass through partitions in the transformation side we have hash auto key so based on that then you can go to the transformations here you can implement the source side we can implement the same type of partitions so here i will go for mod function so equal to 0 this one equal to 1 this one equal to 2 from the source this is the partition point from the source it will take based on this one only but after the source the shorter transformation it will change it based on the hash key so this is what you can improve the performance of the sorter okay this is what you can improve the performance of the sorter also so you can see here a data might be getting changed after the source after the source the reading the data right after that it will be changing so here you can see here data is like here is having that 35 36 3600 but after the hash key generation the target it is changing 29 but everything is same only the number of record will be same okay but here it is based on the hash key the number of record passing through the partition will differ because it&amp;#39;s based on the hash key algorithm okay this is what hash auto key hash user key you can go for okay if interview if they are asking have you implemented any performance tuning method you can tell them yes i have first of all what will you do i will analyze the session log i will find the value of the bottleneck and so if there is a bottleneck on the source side or target side heavily identify so there is a busy percentage right so busy percentage i will find and then i will find vary the bottleneck based on that i will go for the the methods if it is performance degradation on the transformation level then i will go for transformation side some implement some performance tuning if it is on the source side or target side so based on the scenario what i have explained you can explain them so index creation all this you can go for or i can go for push down optimization or partitioning push down optimization is nothing but both strong optimization and the partitioning is the license property whenever i&amp;#39;m finding some issues so trial and error method i will go for first customer notification it&amp;#39;s a license one in not it will not be available in all the project okay some project they will not buy this pushdown automation okay some project only will buy it it&amp;#39;s based on the company okay and here the partitioning is nothing but parallel processing parallel processing we do have different partition types we have to consider three things one is number of partitions partition type partition point just how i have explained here pass through partitions range partition high shutter key so that you can explain them so they will get convinced okay this is what you have to explain the partitioning and performance improvement method and push down optimization in the real time project and also in the interview see this is what you have to explain so just we have we have seen so that you have to explain them very clearly this is what the partitioning always they will just ask what is partitioning have you done the partitioning performance tuning what is meta categories so i will give you this document please go through the document go through the video and practice it so completely practice this this is what they will ask in interview whatever i have explained here if you are explaining them then they will not ask any questions they will get convinced [Music] okay in our today&amp;#39;s session we are going to learn about slowly changing dimension we have seen already what is dimension table dimension table is nothing but it will have a primary key and attributes this is what the dimension table it will have a primary key and attributes we do have many dimension table in our real time project so if you look at here we have customer table product table account table user table employee regions locations time dimension store dimension virtual dimension like this we have lot of different different dimension table so these dimension table we will extract the data from source and we will load it into a target table that&amp;#39;s whatever we will maintain into our data values how are we going to maintain that&amp;#39;s what the slowly changing dimension will have okay in order to maintain this table in target we have different types three types we are we are going to learn about here the first one is type one no history will be maintained that means whenever there is a change in the oltp side so just we are going to change in the target side oil ap side also okay simply we will update it we have already seen this in update strategy session so if you haven&amp;#39;t watched please watch the session practice the session then you can practice the next one okay so this is update second type type 2 type 2 is nothing but history will be maintained so this is what definitely we will use in our real time project history will be maintained that to row level history so entire row will be taken for history so we do have different methods so first method is flag method by using the flag we will maintain the history by using the version we will maintain the history then we have something like effective date and end date so date method by using that method we will maintain the list so that we will see one by one what is that then type three so here the entire row will be maintained for history so it is taking lot of space to avoid that we are going for the column level history okay only recent level history will be maintained okay reason history so last changes only will be available the other changes we will overwrite the data so that&amp;#39;s what we will maintain so we have three different types mainly type 1 type two and type three so type one no history will be maintained type two history will be maintained the two row level so we do have three methods you will see one by one all the method and then type three only reason ht will be maintained column level history will be maintained okay the type one we have already seen just we will i will explain okay consider yes underscore customer is a source table customer table it&amp;#39;s a source table dim underscore customer is the target table or olap side we are maintaining this table so we need to whatever the customer is inserted in the source so a new customer should be inserted into the target if the customer is updating any of the attributes any of the different dimensions that we have to update it here so initially how we will do in type one we will take this particular customer id we will take a lookup customer id we will look up the target customer id and if you are getting null value we will go and insert very first we will go and insert all the record so we will insert it very first time it will be inserted okay second time the job is running you assume that there is no insert and there is no update in this whole site if the job is running then nothing will be happen here okay the target so why because here we have source side we have customer id target side we have the same customer id if both the customer ids are same then we are going to check whether the first name is not equal to this first name mobile number not equal to this mobile number address is not equal to this address zip code and country so all the column we are going to check it if any one of the column value has been changed okay any one of the column value has been changed then we will go for update we will have some flag here instead flag update flag so based on the flag we will take say for an example this mobile number there&amp;#39;s an update there&amp;#39;s an update in the mobile number you assume that here there is an update okay there is an update in the mobile number so what will happen so this customer id and this customer id are equal then first name equal to first name but the mobile number not equal to target mobile number lookup mobile number okay this is what lookup okay if it is not equal we are updating this record in the target okay so we just update we just update this record in the target this is what type one we have already seen that and we have already completed the practical session also in this type okay this is called type one normally type 1 will not have a history that&amp;#39;s why we will not use in our real time project mostly type 2 will use so what is type 2 so if you look at here if you look at the type 2 so we are the source table we are the target table so i am going to check whether this particular record is there in target or not if it is not there i am going to insert the record okay so very first time all the record will get inserted okay all the record will get inserted after that if there is a change if there is a there is a change in the there is a change in the source side mobile number has been changed you assume that this change should be taken as a history the target okay so if one customer is changing any one of the attributes either the first name or mobile number address zip code or country any one of that would be switching getting changed so we will take this particular card okay we will take this particular record if there is an update in order to maintain the history we will take the entire record okay again we will go and insert it here okay we will go and insert it here even though if it is an update here on the source side we will insert it in the target okay so here two records are present for this particular customer right we may get confused which one is the latest one so whenever there is an insert so we will maintain one column called the flag in the flag we will maintain flag equal to one okay we will maintain whenever there is an insert we will make plug equal to one okay so the existing flag right and this flag we have to maintain like as a zero we have to update like a zero so this column we cannot make it like a primary key why because there is a duplicate and i cannot update with this column i cannot update this one so we have two records right so we will maintain in order to maintain the history now to maintain it in order to go for update we will maintain one more key column called surrogate key this is called surrogate key okay this column is called surrogate key su r1 g80 is nothing but in addition to the natural key column that&amp;#39;s a primary key column in order to maintain the history we will maintain one more key column that is called a surrogate key this key column is nothing but it&amp;#39;s a unique column by using sequence generator we will create a surrogate key okay so here we will maintain the key column key values okay like this we will maintain the key values okay this one this record we will maintain as update 0 we will update it like 0 we are updating the flag as 0 here okay so new record we will insert it with the flag equal to 1 okay with the flag equal to 1 for update record we will insert flag equal to one okay we will insert it flag equal to one then update the existing record the flag equal to zero okay again the same record is going to change you assume that the zip code is going to change if there is a change in the record so how which record i have to compare either this record or this record there is a confusion right so always we have to compare with flag equal to one so this record we have to compare okay so that&amp;#39;s why we will take look always look up we will take active record always we will take lookup is a active record look at here if there is a new record in the source new record in the source we will insert the record with the flag equal to one okay if there is an update the source we will insert the record with the flag equal to one and update the existing record with flag equal to zero and always look up the active record look up it&amp;#39;s nothing but flag equal to one so this is what a cd type to flag method we will create a mapping for this then you can able to understand then type to version so how we will maintain in our uh word document all the version right in the project under we will maintain version version equal to one then if there is a change so one plus one you will make two right so the again you are going to change it then three version equal to three that&amp;#39;s what we will maintain right the same way if there is a change very first time if the record is coming up then you will be maintaining version equal to one okay version equal to one if there is a change in the existing record we will maintain one more version one more record here version equal to version plus one existing version plus one if the same record is going to change if the same record is going to change again the previous version two plus one it&amp;#39;s a three three plus one four so like this we will maintain the existing version plus one okay so this is what we will maintain the version so if new record we will maintain version equal to one if there is an update in the record we will maintain version equal to version plus one existing version plus one so that means we will we will never ever update any record so there is no need of update strategy in version method there is no need of update strategy simply insert only but we will check what is the existing version then that person will maintain so for an example this nega is going again the record is going to change then already version is four versus maximum version we will take the lookup we will take maximum version again against this customer id okay customer id we will take the version maximum version so that version we will take and then while inserting it version plus one we will insert it okay so this is what we will go for the version method so flag method question method i hope you are clear on the flag and version method next one is called date method what is date method right so these two method we will we may not know when the data has been changed okay when the data has been changed you you do not have any idea but in version method in the date method right we will maintain this we&amp;#39;ll maintain the column like effective date and end date so what is that effective date for an example today the record you are changing this new record is added into the source then effectively it will be system data so this state will be effective date the end date will be defaulted 999 will maintain so we may not know when the record is going to change okay when the record is going to change we may not know right so that&amp;#39;s why end date we will maintain like nine nine nine if there is a change in the source the source we will insert the record effective date equal to today&amp;#39;s date say for an example today&amp;#39;s date the state is the today&amp;#39;s date and the end date is the 999 and the previous record this same sramanti record here okay this record effective date is same date only but the end date we will mark this system date to end date okay from this date to this date this is an attribute after that from here this is an attribute so what is the which one is the active record the active record is nothing but whichever the date is end date equal to 999 sometimes some project they will use null value some project they will use two triple line some project where they will use four names okay but anyhow it&amp;#39;s a date method this is what we will maintain if there is a new record in the source look at here if there is a new record in the source we will insert the record with the effective date is today&amp;#39;s date end date is defaulted 999 if there is an update if there is an update in the source we will insert the record with the effective date is while inserting this data effective date is today&amp;#39;s date and date is 999 but while updating it we will update the end date equal to system date clear so this is what we will maintain the effective date indeed okay effective date and date look up you have to always look up on the 999 so this is what you will do so i will do this date method so you can practice this version and the flag method or you can check it and the cd type 3 so type 3 is nothing but say for an example this is what the source type 3 is nothing but reason history will be maintained that to column level so whichever the column we have to maintain for history so that column alone we have to make say for an example i want to maintain a history for mobile number history for address it refers zip code history for zip code and everything so that column alone i will maintain here so previous mobile previous address previous report all this okay so this is what i will maintain then if there is a change say for an example this customer is changing the mobile number okay it&amp;#39;s changing the mobile number you assume that there is a change in the mobile number so what we will do we will take this mobile number already existing mobile number ctrl c and ctrl b here we will we will copy and paste it here that means we will maintain as a history here then this mobile number we will insert it here we will update it here that&amp;#39;s what we will do there if there is an update so one record in source one record in target if there is an update also the same record we will go for the history if again the record is going to change this will go to this one previous mobile number then this one comes in the actual mobile number but here the previous to previous mobile number will get updated okay this is what we will maintain the scd type three only column level history will maintain okay so i hope you are clear on this what is std type one type two type three and type four sorry type three any doubt in this any doubt on this method flag method question date and type three okay i hope you are clear we will take so the mapping if you look at here so if you have practiced if you have already practiced the update strategy then this is very very easy okay it&amp;#39;s very very easy the extension to that particular update strategy only this a cd type 2 history so look at here i&amp;#39;m taking the source table and target table select star from yes underscore customer underscore cd okay so this is the table we have in our source and the target we have t underscore customer underscore svd underscore date is another another table initially we will truncate the table okay so we will manually truncate this table truncate table table name but after that you should not truncate it so this is what we will go for t underscore customer underscore a cd underscore date okay this is a table we will go for we will take these two tables for source and target so you&amp;#39;ll be expecting lot of questions from this so if you have any questions please ask me and by today itself you have to practice all this okay we can go to mapping m underscore customer a cd type to gate method okay date method i will do it other method you can do it std type 3 i have already posted video on youtube so please watch it you can able to complete the source i&amp;#39;m going to take the source table the source i&amp;#39;m going to import it hr and i can take the source table yes underscore customer underscore a cd on the target we will take this target table e underscore customer underscore a cd date in this table so make sure that target has primary key okay so this is called surrogate key here and we have one more key column this is not key column in order to maintain the history we are maintaining one more column okay effective date indeed then go to mapping it&amp;#39;s very simple only the same like update strategy only so we will take yes underscore customer underscore sct stable source table okay and then t underscore customer underscore svd date so this is the target table okay so this is a target table and i&amp;#39;m going to take expression transformation here so i take expression transformation so we have to find whether this particular record will go for insert or update just i have taken expression transformation then look up on the target right so we have to look up on the target against this customer id so take the lookup here on the target table so this is a target table choose a target table and look up again source customer id okay this customer id and this customer id we have to match it the condition customer id customer id and just i have matched it so from the lookup i will take all the columns okay so these columns are from lookup customer customer key i need to take i do not want this effective date and date just you can take from customer key you can take all this column okay so we will we will create two port one is insert flag update flag so as we have seen in our update step update strategy transformation we will take new record flag so this flag we will set this flag whenever new record the source new record flag this is update record flag so we will set this particular flag whenever there is an update in the source update record flag then we have here the expression in the expression we will have the new record is nothing but okay if a f is null of professional love customer id okay if it&amp;#39;s no love look up customer id these columns the one is nothing but lookup so if you want if you want you can make it like lkp underscore also so if you if you&amp;#39;re thinking that no need then leave it if it&amp;#39;s looked up customer key as customer if it&amp;#39;s not look up customer id then i&amp;#39;m making like true otherwise i&amp;#39;m making like false so this is what we have to maintain this is what we have to maintain for new record new record flag okay validated it should be validated then update record flag so when we will update when we will set this particular flag if if this customer id okay this customer id equal to this customer look up customer id and and first name first name not equal to this first name or any one of the column we are going for r okay mobile number are not equal to lookup mobile number or address not equal to lookup address or zip code not equal to this code or country not equal to this country okay if it is any one of the column is not equal then we&amp;#39;ll maintain like true otherwise false okay so validate it it should be valid then click on ok that&amp;#39;s all then we will maintain two more columns one is called effective date end date so effective date we can mention like date and time column effective date is nothing but your system date only state you can maintain assistant okay this is date here and end it end date will be your default date so it&amp;#39;s also date and time you can go for date and time is nothing but so we will maintain like this two underscore date of 12 31st 999 single quotes comma single quotes mm dd yyyy format 4y okay this is what we&amp;#39;ll maintain okay validated apply okay here so we have to take next level source column along with customer key so we have to take customer key also next level why because customer key is the primary key column which by using this key only we are going to update that that&amp;#39;s why we will take customer key other lookup column no need to take okay we i&amp;#39;m going to take a router transformation to route it so look at here i&amp;#39;m going to route it so take all the source column and customer key also and we will take all these four columns okay so these four columns only these two columns are driving columns like new record flag update record flag so here the target if you look at here this target double click on the router go to groups i&amp;#39;ll create a two group one is insert group one is another update group so this is what insert a group this is what update group update group okay so in insert group we will insert it will make new record group new record flag is true so this is like update record flag is true okay apply okay here then the insert we will insert it customer id up to here we will make it an insert then effective date and date and customer key so what is the customer key we have customer keys nothing but we&amp;#39;ll maintain so key values right so here we will maintain the sequence of numbers so for this we have to use sequence editor so we will use sequence enter one two three this is sequence editor take the sequential tree here so take next value to this customer key double click on this to properties so from start value one we have to maintain we should not enable the cycle option since it is a primary key and we should not enable the reset also okay so control yes then so this one we have completed this logic we have completed okay new record we have inserted effective date is today&amp;#39;s date and it is 999 okay we will update it for update logic we have to insert and update so take this ctrl c ctrl v two times okay so we&amp;#39;ll maintain two times here so for update we will insert right we&amp;#39;ll go for insert so for update insert we will directly insert it okay we&amp;#39;ll insert it here we have to take it from the update group okay remember we have to take it from update group so this is update insert effective date indeed so look at here effective date ended then the same sequence we have to maintain take this next value from the same sequence number from here also here also so why because here if you look at here already five records has been inserted for update insert also we have to maintain the same key only if you are maintaining a different key for update insert then again it will go for one only it&amp;#39;s a primary it will fail so that&amp;#39;s why we are going to use same sequence generator so we have to use same sequential for this okay update update so this is update update right for update update we have to update the date column so we will take update strategy here by using the key value by using this key value we will update by using the key value say for example this record here right by using this key value we will update end date equal to effective date okay take effective date from here okay double click on this update strategy go to update so we can main mention that dd update that is one a dd update is one then by using this customer key with this primary key we will update so this end date equal to effective date that&amp;#39;s all okay that&amp;#39;s all you will mean so this is for insert logic this is for insert insert this is sorry this is update insert this is update update okay so here in lookup we have to look up only active record right we have to look up always active record so go to lookup sql override so here just click on generate sql sql will be generated you can mention that where class where so look at here where end date equal to so this is what the end date right where end date equal to two underscore date of 12 31st 999 single quotes then we will make mm slash td slash yybaby 4y single quotes close it so this is what you will maintain for we will always take only the active record so this query will be executed on target schema in the course schema right so if you want you can execute it from here also so from here also you can execute you can check it whether this query is executing fine or not yes it is executing so we can make it like so control yes apply here so one more time i will tell you first we are looking up the target table okay we are looking up the target table against customer id this customer id we are matching it and here we are creating new record flag update record pack if you snill off customer that look up customer id then we are making this flag as true then this flag if all the columns are equal if any one of the if key columns are equal if customer id equal to the source customer id equal to target customer id okay then any one of the column is not equal we are setting up this flag if the new record flag is set up then we will go and insert the flag inside the record the update update record is set up then we will go and update the record in the target okay so for update logic we will insert it and update it for update we will go for by using the key value we have to update effective date equal to this end date okay any doubt on this so now we will create a workflow we will create a workflow and then we will see the result look at here i will create a workflow wf underscore scd date okay date type to date method so i will take this mapping so remember we should not to truncate and load the target so fail parent of this task fails go to mapping so should be pointing to source target should be pointing to target connection go to target here also maintain like a target don&amp;#39;t go for truncate and load look up also we have to provide the connection so here just we will maintain apply okay save now i am going to execute the particular workflow before executing check check here there is no data in the target so i&amp;#39;m going to start the workflow very first time all the record will get inserted into the target table so record got inserted if you look at here the statistics five record in the source so got inserted five records this is insert logic so this is update insert update update okay so if you look at here effective date will be today&amp;#39;s date effective date will be today&amp;#39;s date end date will be 999 this is what it will maintain okay if i&amp;#39;m going to execute the same workflow again i&amp;#39;m going to execute it should not insert any record it should not update any record in the target see here only it is reading the data five records from source it is not going for any insert in insert update insert update update okay it&amp;#39;s not going wrong okay so i&amp;#39;m going to update any record in the source so manually i&amp;#39;m going to update say yes underscore customer underscore update so look at here this record i&amp;#39;m going to update it the source side any one record i&amp;#39;m going to update so here not update this is sdd this table i&amp;#39;m going to update this mobile number i&amp;#39;m going to update it okay this mobile number i am going to update it so i have updated the mobile number okay so commit it so we have to maintain the history for this right so it should be 5 0 1 1 look at here 5 0 for new record so this one for update insert update update so you can see here so this mobile number so previous mobile number this is new mobile number okay this is the new mobile number okay seven zero one double zero eight zero so this is what it has updated right so you can see here so effective date so since we are we have updated on the same date it is making two days date so this is what so which one is active record so these are the active record so by looking up these five records only should be taken for the consideration clear on this any doubt so this is what scd types okay type one i have already we have already completed type two flag method version method you just try it type three i have already posted in youtube so type two you can complete it okay so practice all the types if you have any doubt then please let me know [Music] you

Transcript for:Informatica PowerCenter Notes

Transcript for:
Informatica PowerCenter Notes