Recipes of Data Warehouse and Business Intelligence

Staging Area: How to verify the reference day of the data source

The Micro ETL Foundation

• The Micro ETL Foundation is a set of ideas and solutions for Data Warehouse and Business Intelligence Projects in Oracle environment.

• It doesn’t use expensive ETL tools, but only your intelligence and ability to think, configure, build and load data using the features and the programming language of your RDBMS.

• This recipes is another easy example based on the slides of Recipes 1 and 2 of Data Warehouse and Business Intelligence.

• Copying the content of the following slides with your editor and SQL Interface utility, you can reproduce this example.

• The solution presented here is the check of the reference day of a data source. This is another little component to permit us to achive the complete control of the Staging Area loading.

The problem

• In the «Recipe 3» we have seen the check about the rows number. We have ensured this: You have received, for example, 943 rows from the data source file. You have loaded 943 rows in the Staging Area table.

• This is not enough.• Now we need to ensure that the reference day (DAY_KEY) of the data is

correct.• Suppose that you start the loading of account balances file now:

wednesday at 01:00. You expect that the reference day of the balances is yesterday, tuesday. The loading day is today, the expected day is today-1.

• If now is monday, and there is no process during week-end, the expected day is today-3, friday

What to check

• At the end of the loading, we need to control that it is gone all ok. We need to ensure that in this loading day, the expected day is correct. To have this safety, we must:

1. Build a configuration table.2. Initialize the configuration table with the expected day for every

loading day3. Create a log table4. Fill the log table at the end of the load of Staging Area.

• Without this check we risk to reload the same day if the source system had some problems or it has not been able to overwrite the previous file.

The daily configuration table

• IO_COD is the same of the configuration table created in «Recipes2». It is unique.

• DAY_KEY is the loading day.• DY_TXT is the day of the week• WD_FLG is the flag of working day.• FROM_EDAY_KEY/TO_EDAY_KEY are the

expected range of day. For daily data source file they will have the same value.

• To have a range can be useful for monthly data files that contains all the days of the month.

• Now we see how to configure 3 years.

DROP TABLE STA_IODAY_CFT;CREATE TABLE STA_IODAY_CFT( IO_COD VARCHAR2(12), DAY_KEY NUMBER, DY_TXT VARCHAR2(3), WD_FLG NUMBER, FROM_EDAY_KEY NUMBER, TO_EDAY_KEY NUMBER );

The load of daily table

• The load of this table may be done only one time. • In this example we configure a daily loading (excluding week-end) from a year in

the future (sysdate+365) and 2 years in the past.• Configure holidays according to your country calendar. Here are configured only

Christmas Day, New Year's Day and Independence Day.

INSERT INTO STA_IODAY_CFT (IO_COD,DAY_KEY,DY_TXT,WD_FLG,FROM_EDAY_KEY,TO_EDAY_KEY)WITH X AS ( SELECT TO_CHAR((SYSDATE+365)-LEVEL,'YYYYMMDD') AS DAY_KEY ,TO_CHAR((SYSDATE+365)-LEVEL,'dy') DY_TXT FROM DUAL CONNECT BY LEVEL <= (365*3)),Y AS ( SELECT TO_CHAR((SYSDATE+365)-LEVEL,'YYYYMMDD') AS DAY_KEY ,FIRST_VALUE(TO_CHAR((SYSDATE+365)-LEVEL,'YYYYMMDD')) OVER (ORDER BY TO_CHAR((SYSDATE+365)-LEVEL,'YYYYMMDD') ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS EDAY_KEY FROM DUAL WHERE TO_CHAR((SYSDATE+365)-LEVEL,'dy') NOT IN ('sun','sat') AND SUBSTR(TO_CHAR(SYSDATE+365-LEVEL,'YYYYMMDD'),5) NOT IN ('0101','0704','1225') CONNECT BY LEVEL <= (365*3) ORDER BY 1 DESC)SELECT 'employees1',X.DAY_KEY,X.DY_TXT,(CASE WHEN (EDAY_KEY IS NULL) THEN 0 ELSE 1 END),Y.EDAY_KEY,Y.EDAY_KEYFROM XLEFT OUTER JOIN Y ON (X.DAY_KEY=Y.DAY_KEY)ORDER BY 1,2 DESC;

The daily log table

• This table must contain a row for every data source file.

• The DAY_KEY is the loading day• The others day columns are the

expected reference day and the effective reference day.

• The RET_COD contains the result code of the check and will be ‘OK ‘ or ‘NOT OK’.

DROP TABLE STA_IODAY_LOT;CREATE TABLE STA_IODAY_LOT( IO_COD VARCHAR2(12) NOT NULL, SOURCE_COD VARCHAR2(80) NOT NULL, DAY_KEY NUMBER, FROM_EDAY_KEY NUMBER, TO_EDAY_KEY NUMBER, FROM_DAY_KEY NUMBER, TO_DAY_KEY NUMBER, RET_COD VARCHAR2(30), STAMP_DTS DATE);

The load of the log table

• We use the test case of the «Recipe 2» and the data loaded into STA_EMPLOYEES1_STT table. (so it has an old day_key)

• We calculate the min and max day_key from the Staging Area table.• We compare the received day(s) with the expected day(s)

INSERT INTO STA_IODAY_LOT ( IO_COD, SOURCE_COD, DAY_KEY, FROM_EDAY_KEY, TO_EDAY_KEY, FROM_DAY_KEY, TO_DAY_KEY, RET_COD, STAMP_DTS) WITH X AS ( SELECT SOURCE_COD ,TO_CHAR(SYSDATE,'YYYYMMDD') DAY_KEY ,MIN(DAY_KEY) FROM_DAY_KEY ,MAX(DAY_KEY) TO_DAY_KEY FROM STA_EMPLOYEES1_STT GROUP BY SOURCE_COD)SELECT A.IO_COD,X.SOURCE_COD,X.DAY_KEY,A.FROM_EDAY_KEY,A.TO_EDAY_KEY,X.FROM_DAY_KEY,X.TO_DAY_KEY,(CASE WHEN (A.FROM_EDAY_KEY+A.TO_EDAY_KEY=X.FROM_DAY_KEY+X.TO_DAY_KEY) THEN 'OK' ELSE 'NOT OK' END),SYSDATEFROM STA_IODAY_CFT ALEFT OUTER JOIN X ON (A.DAY_KEY = X.DAY_KEY)WHERE A.IO_COD = 'employees1'AND A.DAY_KEY = TO_CHAR(SYSDATE,'YYYYMMDD');COMMIT;

Conclusion

Email - massimo_cenci@yahoo.itBlog (italian/english) - http://massimocenci.blogspot.it/

With only two tables, we have reached the control of the reference day of the source data without ETL tools.This is the philosophy of Micro ETL Foundation.

We are at the end of this recipe. The configuration and the log tables are: