Oracle 19c: Real-Time and High Frequency Statistics Collection · oracle 19c: real-time statistics & high-frequency statistics collection david kurtz accenture enkitec group ukoug

ORACLE 19C:

REAL-TIME

STATISTICS &

HIGH-FREQUENCY

STATISTICS

COLLECTION

DAVID KURTZ

ACCENTURE ENKITEC GROUP

UKOUG TECHFEST2019

Copyright © 2019 Accenture All rights reserved. |

WHO AM I

Accenture Enkitec Group

Performance tuning

• PeopleSoft ERP

• Oracle RDBMS

Book

• www.go-faster.co.uk

• blog.psftdba.com

Oak Table

Copyright © 2019 Accenture All rights reserved. | 3

AGENDA

2 new 19c features

• Real-Time Statistics

• High-Frequency Automatic Optimizer Statistics Collection

Unfortunately, at the moment, these features are only available on Engineered Systems


The optimizer is a piece of software that works out how to execute your SQL statements.

• Oracle 2.3, 1979, rule-based optimizer

• Cost-Based Optimizer introduced Oracle 7.0.12, 1992

• Since then

• DBMS_STATS package replaced ANALYZE

• Lots of automatic behaviour

• Histograms to describe skew

• Maintenance Window

• Optimizer Dynamic Sampling

• Statistic Preferences (11gR2)

• Cardinality/Statistics Feedback (11gR2/12c)

• Adaptive Plans (12c)

• Hash based NDV calculation (12c)

LET'S TALK ABOUT THE OPTIMIZER

• Cost-Based Optimizer

• Volumetric statistics about tables and indexes inform a mathematical model that costs execution plans and finds the cheapest.

• Implicit assumption:

• Execution plan cost α Execution duration

• If you provide inaccurate statistics to the optimizer, then you cannot expect it to produce the correct answer.

• With every release we have seen efforts to improve the optimizer cost model and improve the quality of the statistics

• The optimizer works better with accurate statistics.


Dave Ensor

"the grand old man of performance optimization methodology in the Oracle world" – Jonathan Lewis

6

ENSOR PARADOX

• "The only time that it is safe to gather statistics is when to do so would make no difference."

• What happens when you have gathered statistics?

• The plan might change

• Or the plan might not change

• What happens if you do not gather statistics?

• The plan might still change because the data has changed.

• Literal or bind values have moved outside the range of the column statistics, or into a region of different skew.

• Why do we gather optimizer statistics?

• So that the plans remain the same!


REAL-TIME STATISTICS

7


PURPOSE OF REAL-TIME STATISTICS

Oracle Documentation:

• 10.3.3.3 19c SQL Tuning Guide, Part V Optimizer Statistics, Optimizer Statistics Concepts,

• How the Database Gathers Optimizer Statistics, On-line Statistics Gathering, Real-Time Statistics

"Online statistics … aim to reduce the possibility of the optimizer being misled by stale statistics.

• 12c introduced online statistics gathering for CREATE TABLE AS SELECT statements and direct-path inserts.

• 19c introduces real-time statistics, which extend online support to conventional DML statements.

• Because statistics can go stale between DBMS_STATS jobs, real-time statistics helps the optimizer generate more optimal plans.

• Real-time statistics augment rather than replace traditional statistics…must continue to gather statistics regularly using DBMS_STATS"

https://docs.oracle.com/en/database/oracle/oracle-database/19/tgsql/optimizer-statistics-concepts.html#GUID-769E609D-0312-43A7-9581-3F3EACF10BA9

https://docs.oracle.com/en/database/oracle/oracle-database/19/tgsql/optimizer-statistics-concepts.html#GUID-769E609D-0312-43A7-9581-3F3EACF10BA9


TABLE MONITORING

Tracks inserts, updates and deletes, as well as whether the table has been truncated, in memory.

• Introduced in 10g

• Enabled by default since 11g

Used to determined whether statistics are stale

• Drives statistics collection during the maintenance window

Periodically persisted to disk

• by SMON, approximately every 15 minutes

• can be flushed out manually with dbms_stats.flush_database_monitoring_info


WHAT HAPPENS DURING DML?

"When a DML operation is currently modifying a table, Oracle Database dynamically computes values for the most essential statistics."

• There is no call to DBMS_STATS visible in trace.

• 'OPTIMIZER STATISTICS GATHERING' operation in execution plan

• The changes to the table are tracked in memory and are flushed to the data dictionary later along with other monitoring data.

• Or they can be flushed with dbms_stats.flush_database_monitoring_info.

• On-line statistics visible in data dictionary views

• DBA_TAB_STATISTICS

• DBA_TAB_COL_STATISTICS

• Additional rows with note: 'STATS_ON_CONVENTIONAL_DML'


DEMONSTRATING REAL-TIME STATISTICS


DEMONSTRATING REAL-TIME STATISTICS

A series of simple tests that

• Build, populate, and then make some more changes

• Then we look at the statistics and some execution plans

This will

• reveal the behaviour of real-time statistics

• and suggest some limitations and risks


TEST: POPULATING A SIMPLE TABLE IN CONVENTIONAL PATH MODE

connect / as sysdbaalter system flush shared_pool;

connect oe/oedrop table t purge;create table t(a number NOT NULL,a2 number NOT NULL,b varchar2(2000),CONSTRAINT t_pk PRIMARY KEY (a));

insert into twith n as (select rownum n from dual connect by level <= 1000)select n.n, ceil(sqrt(n.n)), TO_CHAR(TO_DATE(n.n,'j'),'jsp')from n;commit;

insert into twith n as (select 1000+rownum n from dual connect by level <= 10000)select n.n, ceil(sqrt(n.n)), TO_CHAR(TO_DATE(n.n,'j'),'jsp')from n;commit;

connect / as sysdbaexec dbms_stats.flush_database_monitoring_info;connect oe/oe

A2 starts in the range 1-32, and

later 1-105, but is highly skewed


REAL-TIME STATISTICS BEING GATHEREDON CONVENTIONAL DML

SQL_ID dv6z6pa38uuyk, child number 0

-------------------------------------

insert into t with n as (select 1000+rownum n from dual connect by

level <= 10000) select n.n, ceil(sqrt(n.n)),

TO_CHAR(TO_DATE(n.n,'j'),'jsp') from n

Plan hash value: 761049541

---------------------------------------------------------------------------------------------------------------------

| Id | Operation | Name | E-Rows |E-Bytes| Cost (%CPU)| E-Time | OMem | 1Mem | Used-Mem |

---------------------------------------------------------------------------------------------------------------------

| 0 | INSERT STATEMENT | | | | 2 (100)| | | | |

| 1 | LOAD TABLE CONVENTIONAL | T | | | | | | | |

| 2 | OPTIMIZER STATISTICS GATHERING | | 1 | 13 | 2 (0)| 00:00:01 | | | |

| 3 | VIEW | | 1 | 13 | 2 (0)| 00:00:01 | | | |

| 4 | COUNT | | | | | | | | |

| 5 | CONNECT BY WITHOUT FILTERING| | | | | | 2048 | 2048 | 2048 (0)|

| 6 | FAST DUAL | | 1 | | 2 (0)| 00:00:01 | | | |

---------------------------------------------------------------------------------------------------------------------


BUT JUST BECAUSE THERE ARE GATHERED DOESN'T MEAN THEY ARE STORED

Simple Query of Statistics Views

Table Num

Name MON Rows

----- --- --------

T YES

Table Num

Name Rows BLOCKS NOTES

----- -------- ---------- -------------------------

T

Column Num

Name DATA_TYPE Vals LOW_VALUE LOW_V HIGH_VALUE HI_V

------ ---------- -------- ---------------- ---------------- -------------------------------- --------------------------------

A NUMBER

A2 NUMBER

B VARCHAR2

no rows selected

ALL_TABLES

ALL_TAB_STATISTICS

ALL_TAB_COLUMNS

ALL_TAB_COL_STATISTICS



• If you don't have normal statistics, you won't get real-time statistics.


TEST: STATISTICS AUTOMATICALLY GATHERED ON INITIAL DIRECT-PATH LOAD

create table t

(a number NOT NULL

,a2 number NOT NULL

,b varchar2(2000)

,CONSTRAINT t_pk PRIMARY KEY (a));

insert /*+APPEND*/ into t

with n as (select rownum n from dual connect by level <= 1000)

select n.n, ceil(sqrt(n.n)), TO_CHAR(TO_DATE(n.n,'j'),'jsp')

from n;

commit;

insert into t

with n as (select 1000+rownum n from dual connect by level <= 10000)


from n;


WITHOUT FLUSHING MONITORING INFOWE HAVE ADDITIONAL COLUMN STATS

Table NumName MON Rows----- --- --------T YES 1000

Table NumName Rows BLOCKS NOTES----- -------- ---------- -------------------------T 1000 5

Column NumName DATA_TYPE Vals LOW_VALUE LOW_V HIGH_VALUE HI_V------ ---------- -------- ---------------- ---------------- ------------------------------------- -------------------------------------A NUMBER 1000 C102 1 C20B 1000A2 NUMBER 32 C102 1 C121 32B VARCHAR2 1000 6569676874 eight 74776F2068756E647265642074776F two hundred two

Table Column NumName Name Distinct LOW_VALUE LOW_V HIGH_VALUE HI_V HISTOGRAM----- ------ -------- ---------------- ---------------- ------------------------------------- ------------------------------------- -------------NOTES-------------------------T A 1000 C102 1 C20B 1000 NONESTATS_ON_LOAD

T A2 32 C102 1 C121 32 NONESTATS_ON_LOAD

T B 1000 6569676874 eight 74776F2068756E647265642074776F two hundred two NONESTATS_ON_LOAD

T A C102 1 C3020A53 10982STATS_ON_CONVENTIONAL_DML

T B 6569676874 eight 74776F2074686F7573616E642074776F20687 two thousand two hundred twenty-seven56E64726564207477656E74792D736576656E

STATS_ON_CONVENTIONAL_DML

T A2 C102 1 C20206 105STATS_ON_CONVENTIONAL_DML


COLUMN STATS FOR CONVENTIONAL DMLBUT NUMBER OF ROWS ON TABLE WRONG

SQL_ID dbxq2k3458mkn, child number 0

-------------------------------------

select count(*) from t where a2 <= 42


------------------------------------------------------------------------------------------------------------------------------

| Id | Operation | Name | Starts | E-Rows |E-Bytes| Cost (%CPU)| E-Time | A-Rows | A-Time | Buffers | Reads |

------------------------------------------------------------------------------------------------------------------------------

| 0 | SELECT STATEMENT | | 1 | | | 3 (100)| | 1 |00:00:00.01 | 79 | 5 |

| 1 | SORT AGGREGATE | | 1 | 1 | 13 | | | 1 |00:00:00.01 | 79 | 5 |

|* 2 | TABLE ACCESS FULL| T | 1 | 425 | 5525 | 3 (0)| 00:00:01 | 1764 |00:00:00.01 | 79 | 5 |

------------------------------------------------------------------------------------------------------------------------------

…

Predicate Information (identified by operation id):

---------------------------------------------------

2 - filter("A2"<=42)

Note

-----

- dynamic statistics used: statistics for conventional DML


FLUSH THE DATABASE MONITORING INFOMATION

I could wait for the monitoring information to flush naturally.Or I can flush it manually:

exec dbms_stats.flush_database_monitoring_info;


HAVING FLUSHED MONITORING INFORMATIONALSO HAVE ADDITIONAL TABLE STATS!Table Num

Name MON Rows

----- --- --------

T YES 1000

Table Num


----- -------- ---------- -------------------------

T 1000 5

T 11000 76 STATS_ON_CONVENTIONAL_DML

Column Num


------ ---------- -------- ---------------- ---------------- ------------------------------------- -------------------------------------

A NUMBER 1000 C102 1 C20B 1000

A2 NUMBER 32 C102 1 C121 32

B VARCHAR2 1000 6569676874 eight 74776F2068756E647265642074776F two hundred two

Table Column Num

Name Name Distinct LOW_VALUE LOW_V HIGH_VALUE HI_V HISTOGRAM

----- ------ -------- ---------------- ---------------- ------------------------------------- ------------------------------------- -------------

NOTES

-------------------------

T A 1000 C102 1 C20B 1000 NONE

STATS_ON_LOAD

T A2 32 C102 1 C121 32 NONE

STATS_ON_LOAD

T B 1000 6569676874 eight 74776F2068756E647265642074776F two hundred two NONE

STATS_ON_LOAD

T B 6569676874 eight 74776F2074686F7573616E642074776F20687 two thousand two hundred ninety-two

56E64726564206E696E6574792D74776F


T A2 C102 1 C20206 105


T A C102 1 C3020A1F 10930



COLUMN STATS FOR TABLE TOO

SQL_ID 4w1zhvzyh0hb9, child number 0

-------------------------------------

select /*B*/ count(*) from t where a2 <= 42


---------------------------------------------------------------------------------------------------------------------

| Id | Operation | Name | Starts | E-Rows |E-Bytes| Cost (%CPU)| E-Time | A-Rows | A-Time | Buffers |

---------------------------------------------------------------------------------------------------------------------

| 0 | SELECT STATEMENT | | 1 | | | 22 (100)| | 1 |00:00:00.01 | 78 |

| 1 | SORT AGGREGATE | | 1 | 1 | 13 | | | 1 |00:00:00.01 | 78 |

|* 2 | TABLE ACCESS FULL| T | 1 | 4680 | 60840 | 22 (0)| 00:00:01 | 1764 |00:00:00.01 | 78 |

---------------------------------------------------------------------------------------------------------------------

…

Note

-----


Added comment to force parse of new

statement, otherwise would get same

plan as before from library cache



• Real-time column statistics really are real-time because they are immediately visible to the optimizer

• Real-time table statistics do not become visible until the table monitoring information has been flushed.

• Creation of real-time statistics does not invalidate cursors.

• Assuming application requires to parse new SQLs, there are several opportunities for execution plans to change as state of statistics change

• Stale Statistics – under estimate rows

• Real-time column statistics – worse under estimate

• Real-time table statistics – over estimate


TEST: STATISTICS (INCLUDING HISTOGRAM) EXPLICITLY GATHERED BEFORE CONVENTIONAL DML

create table t(a number NOT NULL,a2 number NOT NULL,b varchar2(2000),CONSTRAINT t_pk PRIMARY KEY (a));

insert into twith n as (select rownum n from dual connect by level <= 1000)select n.n, ceil(sqrt(n.n)), TO_CHAR(TO_DATE(n.n,'j'),'jsp')from n;commit;

exec dbms_stats.gather_table_stats('OE','T' -,method_opt=>'FOR ALL COLUMNS SIZE AUTO FOR COLUMNS A2 SIZE 254');

insert into twith n as (select 1000+rownum n from dual connect by level <= 10000)select n.n, ceil(sqrt(n.n)), TO_CHAR(TO_DATE(n.n,'j'),'jsp')from n;commit;


ADDITIONAL COLUMNS STATISTICS ON DMLAND HISTOGRAM ON COLUMN A2Table Num

Name MON Rows

----- --- --------

T YES 1000

Table Num


----- -------- ---------- -------------------------

T 1000 5

Column Num


------ ---------- -------- ---------------- ---------------- ------------------------------------- -------------------------------------

A NUMBER 1000 C102 1 C20B 1000

A2 NUMBER 32 C102 1 C121 32


Table Column Num


----- ------ -------- ---------------- ---------------- ------------------------------------- ------------------------------------- -------------

NOTES

-------------------------

T A 1000 C102 1 C20B 1000 NONE

T A2 32 C102 1 C121 32 FREQUENCY


T A2 C102 1 C20206 105


T B 6569676874 eight 74776F2074686F7573616E642074776F20687 two thousand two hundred eighty-nine

56E64726564206569676874792D6E696E65


T A C102 1 C3020A51 10980



HISTOGRAM HAS TAKEN PRECEDENCE OVER REAL-TIME COLUMN STATISTICS


-------------------------------------



---------------------------------------------------------------------------------------------------------------------


---------------------------------------------------------------------------------------------------------------------


| 1 | SORT AGGREGATE | | 1 | 1 | 13 | | | 1 |00:00:00.01 | 73 |


---------------------------------------------------------------------------------------------------------------------

…

Note

-----


All the values in histogram <= 32

So optimizer thinks it will get all

1000 rows in table


HAVING FLUSHED MONITORING INFORMATION NOW HAVE REAL-TIME TABLE STATISTICSTable Num

Name MON Rows

----- --- --------

T YES 1000

Table Num


----- -------- ---------- -------------------------

T 1000 5


Column Num


------ ---------- -------- ---------------- ---------------- ------------------------------------- -------------------------------------

A NUMBER 1000 C102 1 C20B 1000

A2 NUMBER 32 C102 1 C121 32


Table Column Num


----- ------ -------- ---------------- ---------------- ------------------------------------- ------------------------------------- -------------

NOTES

-------------------------

T A 1000 C102 1 C20B 1000 NONE

T A2 32 C102 1 C121 32 FREQUENCY


T B 6569676874 eight 74776F2074686F7573616E642074776F20687 two thousand two hundred forty-four

56E6472656420666F7274792D666F7572


T A2 C102 1 C20206 105


T A C102 1 C302094E 10877



HISTOGRAM HAS TAKEN PRECEDENCE OVER REAL-TIME COLUMN STATISTICS


-------------------------------------



---------------------------------------------------------------------------------------------------------------------


---------------------------------------------------------------------------------------------------------------------


| 1 | SORT AGGREGATE | | 1 | 1 | 13 | | | 1 |00:00:00.01 | 79 |

|* 2 | TABLE ACCESS FULL| T | 1 | 11000 | 139K| 22 (0)| 00:00:01 | 1764 |00:00:00.01 | 79 |

---------------------------------------------------------------------------------------------------------------------…

Note

-----


The optimizer knows there are

11000 rows in the table, but the

histogram still says they are all

in the range 1-32, even though

the column max value is 105



Histograms takes precedence over column statistics.

• The proportion of rows returned will be calculated from histogram without reference to column statistics.

• Real-time statistics never generate statistics that would require scanning the table.

• Don't generate histograms, or update the number of distinct values

• Heavily skewed data may still present challenges even with real-time statistics

On-Line statistics (on direct-path load) can also gather histograms

• _optimizer_gather_stats_on_load_hist=TRUE

• This undocumented parameter is set in Autonomous Data Warehouse

Collect on-line statistics on every direct path load

• _optimizer_gather_stats_on_load_all=TRUE

• Also set in Autonomous Data Warehouse


Oracle Default:

• histograms created automatically where skew and usage

• set_global_prefs

• method_opt => 'for all columns size auto'

• set_table_prefs

• method_opt => 'for all columns size auto, for columns XXX size 1'

http://coug.us/wp/wp-content/uploads/2019/03/MColgan_Best_Practices_for_Manging_statistics.pdf

31

SHOULD WE COLLECT HISTOGRAMS BY DEFAULT?

Jonathan Lewis:

• Do not gather histograms

• Unless Frequency or Top-Frequency

• Get rid of all "size repeat" in 12c+

• Change default

• set_global_prefs

• method_opt => 'for all columns size 1'

• set_table_prefs

• method_opt => 'for all columns size 1, for columns XXX size N'

• Write scripts for "unusual" histograms

https://cdn.ymaws.com/ukoug.org/resource/resmgr/scotland_19/jonathan_lewis.pdf

http://coug.us/wp/wp-content/uploads/2019/03/MColgan_Best_Practices_for_Manging_statistics.pdf

https://cdn.ymaws.com/ukoug.org/resource/resmgr/scotland_19/jonathan_lewis.pdf


TEST: FULL EXPLICIT STATISTICS

create table t

(a number NOT NULL

,a2 number NOT NULL

,b varchar2(2000)





from n;

commit;

insert into t



from n;

commit;


exec dbms_stats.gather_table_stats('OE','T' –

,method_opt=>'FOR ALL COLUMNS SIZE AUTO FOR COLUMNS A2 SIZE 254');


STATISTICS COLLECTION DELETES REAL-TIME STATISTICSTable Num

Name MON Rows

----- --- --------

T YES 11000

Table Num


----- -------- ---------- -------------------------

T 11000 76

Column Num


------ ---------- -------- ---------------- ---------------- ------------------------------------- -------------------------------------

A NUMBER 11000 C102 1 C3020B 11000

A2 NUMBER 105 C102 1 C20206 105

B VARCHAR2 11000 6569676874 eight 74776F2074686F7573616E642074776F20687 two thousand two hundred two

56E647265642074776F

Table Column Num


----- ------ -------- ---------------- ---------------- ------------------------------------- ------------------------------------- -------------

NOTES

-------------------------

T A 11000 C102 1 C3020B 11000 NONE

T A2 105 C102 1 C20206 105 FREQUENCY

T B 11000 6569676874 eight 74776F2074686F7573616E642074776F20687 two thousand two hundred two NONE

56E647265642074776F

Not surprising. Fresh statistics

do not need to be augmented

by real-time statistics


UP TO DATE FREQUENCY HISTOGRAM PRODUCES ACCURATE ESTIMATE ON SKEWED DATA


-------------------------------------



---------------------------------------------------------------------------------------------------------------------


---------------------------------------------------------------------------------------------------------------------


| 1 | SORT AGGREGATE | | 1 | 1 | 4 | | | 1 |00:00:00.01 | 78 |


---------------------------------------------------------------------------------------------------------------------


TEST: LONE DELETES ARE NOT TRACKED BY REAL-TIME STATITICS

create table t

(a number NOT NULL

,a2 number NOT NULL

,b varchar2(2000)





from n;

commit;

delete from t where a > 800;

commit;



NO REAL-TIME STATISTICS AFTER DELETE ONLY

Table Num

Name MON Rows

----- --- --------

T YES 1000

Table Num


----- -------- ---------- -------------------------

T 1000 5

Column Num


------ ---------- -------- ---------------- ---------------- ------------------------------------- -------------------------------------

A NUMBER 1000 C102 1 C20B 1000

A2 NUMBER 32 C102 1 C121 32


Table Column Num


----- ------ -------- ---------------- ---------------- ------------------------------------- ------------------------------------- -------------

NOTES

-------------------------

T A 1000 C102 1 C20B 1000 NONE

STATS_ON_LOAD

T A2 32 C102 1 C121 32 NONE

STATS_ON_LOAD


STATS_ON_LOAD


TEST: UPDATES ARE TRACKED BY REAL-TIME STATISTICS

create table t

(a number NOT NULL

,a2 number NOT NULL

,b varchar2(2000)





from n;

commit;

update t

set a2 = ceil(sqrt(a+1000))

, b = 'z'||TO_CHAR(TO_DATE(a+1000,'j'),'jsp')

where a <= 500;

commit;


A2 goes from 1-32 to 23-39

B is prefixed with z


REAL-TIME STATISTICS AFTER UPDATE

Table Num

Name MON Rows

----- --- --------

T YES 1000

Table Num


----- -------- ---------- -------------------------

T 1000 5


Column Num


------ ---------- -------- ---------------- ---------------- ------------------------------------- -------------------------------------

A NUMBER 1000 C102 1 C20B 1000

A2 NUMBER 32 C102 1 C121 32


Table Column Num


----- ------ -------- ---------------- ---------------- ------------------------------------- ------------------------------------- -------------

NOTES

-------------------------

T A 1000 C102 1 C20B 1000 NONE

STATS_ON_LOAD

T A2 32 C102 1 C121 32 NONE

STATS_ON_LOAD


STATS_ON_LOAD

T B 6569676874 eight 7A6F6E652074686F7573616E6420746872656 zone thousand three hundred eleven

52068756E6472656420656C6576656E


T A2 C102 1 C126 37


Increase in column maximum

values have been detected

Increase in column minimum

value not detected


TEST: UPDATE THEN DELETE IS TRACKED BY REAL-TIME STATISTICS


insert /*+APPEND*/ into twith n as (select rownum n from dual connect by level <= 1000)select n.n, ceil(sqrt(n.n)), TO_CHAR(TO_DATE(n.n,'j'),'jsp')from n;commit;

update tset a2 = ceil(sqrt(a+1000)), b = 'z'||TO_CHAR(TO_DATE(a+1000,'j'),'jsp')where a <= 500;commit;

delete from t where a > 800;commit;



REAL-TIME STATISTICS UPDATE THEN DELETE

Table Num

Name MON Rows

----- --- --------

T YES 1000

Table Num


----- -------- ---------- -------------------------

T 1000 5


Column Num


------ ---------- -------- ---------------- ---------------- ------------------------------------- -------------------------------------

A NUMBER 1000 C102 1 C20B 1000

A2 NUMBER 32 C102 1 C121 32


Table Column Num


----- ------ -------- ---------------- ---------------- ------------------------------------- ------------------------------------- -------------

NOTES

-------------------------

T A 1000 C102 1 C20B 1000 NONE

STATS_ON_LOAD

T A2 32 C102 1 C121 32 NONE

STATS_ON_LOAD


STATS_ON_LOAD

T B 6569676874 eight 7A6F6E652074686F7573616E642074776F206 zone thousand two hundred thirty-five

8756E64726564207468697274792D66697665


T A2 C102 1 C126 37


Reduction in number

of rows detected

No real-time column statistics on

A because it was not updated.


REAL-TIME STATISTICS AFTER UPDATE THEN DELETE

SQL_ID 1fs1r0dh9pg95, child number 0

-------------------------------------

select count(*) from t where a > 800


--------------------------------------------------------------------------------------------------------------------


--------------------------------------------------------------------------------------------------------------------


| 1 | SORT AGGREGATE | | 1 | 1 | 4 | | | 1 |00:00:00.01 | 2 |

|* 2 | INDEX RANGE SCAN| T_PK | 1 | 160 | 640 | 2 (0)| 00:00:01 | 0 |00:00:00.01 | 2 |

--------------------------------------------------------------------------------------------------------------------

…

Note

-----


Still estimates 160 rows because

20% of range of 1-1000, out of 800 rows.

800*.2=160


TEST: TRUNCATE DOES NOT REMOVE STATISTICS

create table t

(a number NOT NULL

,a2 number NOT NULL

,b varchar2(2000)





from n;

truncate table t;

insert into t



from n;

commit;



REAL-TIME STATISTICS AFTER TRUNCATE AND INSERTTable Num

Name MON Rows

----- --- --------

T YES 1000

Table Num


----- -------- ---------- -------------------------

T 1000 5


Column Num


------ ---------- -------- ---------------- ---------------- ------------------------------------- -------------------------------------

A NUMBER 1000 C102 1 C20B 1000

A2 NUMBER 32 C102 1 C121 32


Table Column Num


----- ------ -------- ---------------- ---------------- ------------------------------------- ------------------------------------- -------------

NOTES

-------------------------

T A 1000 C102 1 C20B 1000 NONE

STATS_ON_LOAD

T A2 32 C102 1 C121 32 NONE

STATS_ON_LOAD


STATS_ON_LOAD

T A2 C102 1 C20206 105


T A C102 1 C302095C 10891


T B 6569676874 eight 74776F2074686F7573616E642074776F20687 two thousand two hundred three

56E64726564207468726565


10000 rows √


REAL-TIME STATISTICS AFFECTED BY STATISTICS ON TRUNCATED DATA

SQL_ID 65dtcj7k2w7c8, child number 0

-------------------------------------

select count(*) from t where a < 501


--------------------------------------------------------------------------------------------------------------------


--------------------------------------------------------------------------------------------------------------------


| 1 | SORT AGGREGATE | | 1 | 1 | 13 | | | 1 |00:00:00.01 | 3 |

|* 2 | INDEX RANGE SCAN| T_PK | 1 | 459 | 5967 | 2 (0)| 00:00:01 | 0 |00:00:00.01 | 3 |

--------------------------------------------------------------------------------------------------------------------

…

Note

-----


Query returns no rows because data in

range 1-1000 was truncated.

Statistics suggest A in range 1-11000

So 500/11000*10000=454


A FEW TRAPS

It appears that deletes are not tracked by real-time statistics – surely a bug

• Updates are tracked

• Deletes after Updates are tracked

Truncate doesn't delete statistics

• So you can get statistics on truncated data mixed with real-time statistics on new data


TEST: PREDICATE OUTSIDE COLUMN RANGE STATISTICS –UNIFORM DISTRIBUTION

create table t

(a number NOT NULL

,a2 number NOT NULL

,b varchar2(2000)


create index t_a2 on t(a2);




from n;

commit;

select /*A*/ count(b) from t where a >= 900;

insert into t



from n;

commit;


select /*C*/ count(b) from t where a >= 1900;

exec dbms_stats.gather_table_stats('OE','T',method_opt=>'FOR ALL COLUMNS SIZE AUTO FOR COLUMNS A2 SIZE 1');

select /*D*/ count(b) from t where a >= 1900;


STATISTICS ON LOAD

SQL_ID b614cm44zbh7n, child number 0

-------------------------------------

select /*A*/ count(b) from t where a >= 900


---------------------------------------------------------------------------------------------------------------------


---------------------------------------------------------------------------------------------------------------------


| 1 | SORT AGGREGATE | | 1 | 1 | 26 | | | 1 |00:00:00.01 | 6 |


---------------------------------------------------------------------------------------------------------------------


REAL-TIME STATISTICSTable Num

Name MON Rows

----- --- --------

T YES 1000

Table Num


----- -------- ---------- -------------------------

T 1000 5


Column Num


------ ---------- -------- ---------------- ---------------- ------------------------------------- -------------------------------------

A NUMBER 1000 C102 1 C20B 1000

A2 NUMBER 32 C102 1 C121 32


Table Column Num


----- ------ -------- ---------------- ---------------- ------------------------------------- ------------------------------------- -------------

NOTES

-------------------------

T A 1000 C102 1 C20B 1000 NONE

STATS_ON_LOAD

T A2 32 C102 1 C121 32 NONE

STATS_ON_LOAD


STATS_ON_LOAD

T B 6569676874 eight 74776F2068756E647265642074776F two hundred two


T A2 C102 1 C12E 45


T A C102 1 C21455 1984


Max column value of A is 1984,

actually 2000


STALE STATISTICSWITH REAL-TIME STATISTICS

SQL_ID 32cu1x428w7sn, child number 0

-------------------------------------

select /*C*/ count(b) from t where a >= 1900


---------------------------------------------------------------------------------------------------------------------


---------------------------------------------------------------------------------------------------------------------


| 1 | SORT AGGREGATE | | 1 | 1 | 29 | | | 1 |00:00:00.01 | 14 |


---------------------------------------------------------------------------------------------------------------------

…

Note

-----


Estimate slightly off because maximum column

value is recorded as 1984 when actually 2000

85/1984*2000=86

Good estimate on a column without skew.

This is the problem that real-time statistics

solves very well.


FRESH STATISTICS

SQL_ID 2jv5mfvk6qswb, child number 0

-------------------------------------

select /*D*/ count(b) from t where a >= 1900


---------------------------------------------------------------------------------------------------------------------------------------


---------------------------------------------------------------------------------------------------------------------------------------


| 1 | SORT AGGREGATE | | 1 | 1 | 33 | | | 1 |00:00:00.01 | 4 |

| 2 | TABLE ACCESS BY INDEX ROWID BATCHED| T | 1 | 101 | 3333 | 3 (0)| 00:00:01 | 101 |00:00:00.01 | 4 |

|* 3 | INDEX RANGE SCAN | T_PK | 1 | 101 | | 2 (0)| 00:00:01 | 101 |00:00:00.01 | 2 |

---------------------------------------------------------------------------------------------------------------------------------------

The conventional statistics produce a more accurate

estimated number of rows, and consequently we use

an index scan rather than a full scan


TEST: STATISTICS AND REAL-TIME STATISTICS ON GLOBAL TEMPORARY TABLES (FROM 12C)

create global temporary table t

(a number NOT NULL

,a2 number NOT NULL

,b varchar2(2000)

,CONSTRAINT t_pk PRIMARY KEY (a))

on commit preserve rows;




from n;

commit;

insert into t



from n;

commit;



ON-LINE BUT NOT REAL-TIME STATISTICS ON GLOBAL TEMPORARY TABLE

Table Num

Name MON Rows

----- --- --------

T NO

Table Num

Name Rows BLOCKS NOTES SCOPE

----- -------- ---------- ------------------------- -------

T SHARED

T 1000 5 SESSION

Column Num


------ ---------- -------- ---------------- ---------------- ------------------------------------- -------------------------------------

A NUMBER

A2 NUMBER

B VARCHAR2

Table Column Num


----- ------ -------- ---------------- ---------------- ------------------------------------- ------------------------------------- -------------

NOTES

-------------------------

T A 1000 C102 1 C20B 1000 NONE

T A2 32 C102 1 C121 32 NONE



NO REAL-TIME STATISTICS ON GLOBAL TEMPORARY TABLES

Concept of real-time statistics doesn't fit with GTTs

Statistics on GTTs have always been problematic.

• Statistics gather on data in one session can be used in other sessions

• 19 introduces session level statistics on GTTs

• Stored in memory, exposed via x$kxttstets

GTTs designed for short-lived storage of data

• If you wait for a database process to flush real-time statistics to disk

• Is the session using the GTTs likely still there?

• Which session's statistics will be stored in memory and flushed?

• Where they being flushed to? And you would have to have conventional statistics

So, it just doesn't happen


LOCK STATISTICS BEFORE DML

create table "OE".t

(a number NOT NULL

,a2 number NOT NULL

,b varchar2(2000)


insert /*+APPEND*/ into "OE".t



from n;

commit;

exec dbms_stats.lock_table_stats('OE','T');

update "OE".t set b = UPPER(TO_CHAR(TO_DATE(a,'j'),'jsp')) where rownum<=101;

commit;



LOCKING STATISTCS BEFORE DMLPREVENTS REAL-TIME STATISTICS

Table Num


----- -------- ---------- ------------------------- -------

T 1000 5 SHARED

Column Num


------ ---------- -------- ---------------- ---------------- ------------------------------------- -------------------------------------

A NUMBER 1000 C102 1 C20B 1000

A2 NUMBER 32 C102 1 C121 32


Table Column Num


----- ------ -------- ---------------- ---------------- ------------------------------------- ------------------------------------- ------------

NOTES

-------------------------

T A 1000 C102 1 C20B 1000 NONE

STATS_ON_LOAD

T A2 32 C102 1 C121 32 NONE

STATS_ON_LOAD


STATS_ON_LOAD


LOCKING STATISTCS BEFORE DMLPREVENTS REAL-TIME STATISTICS

Locking statistics prevents

• collection of statistics in schema/database-wide operations

• collection of real-time statistics on either direct or conventional-path load.

Locking statistics is used because

• No explicit statistics, perhaps rely on dynamic sampling

• Nothing for real-time statistics to augment

• Manage statistics collection manually, or perhaps in the application.

• No point confusing the issue with real-time statistics.


Despite potential for problems, I think real-time statistics are useful enhancement.

However, if you really want, you can disable this behaviour.

Two undocumented parameters

• Can be set at system, session, or statement level

• optimizer online stats gathering for conventional DML

• _optimizer_gather_stats_on_conventional_dml

= TRUE

• use optimizer statistics gathered for conventional DML

• _optimizer_use_stats_on_conventional_dml

= TRUE

58

DISABLING REAL-TIME STATISTICS

Documented hints

• Applied to DML statements

• NO_GATHER_OPTIMIZER_STATISTICS

• GATHER_OPTIMIZER_STATISTICS

• Doesn't override_optimizer_gather_stats_on_conventional_dml = FALSE

There is no table preference for Real-Time Statistics

• It is not possible to control them at table level


Supress collection of statistics on initial bulk load of table

• CREATE TABLE … AS SELECT …

• INSERT /*+APPEND*/ … SELECT …

• (into an empty table)

Introduced in Oracle 12.1

59

NO_GATHER_OPTIMIZER_STATISTICS

Supress collection of real-time statistics during conventional DML

Introduced in Oracle 19.1


NO_GATHER_OPTIMIZER_STATISTICS HINT


insert /*+APPEND NO_GATHER_OPTIMIZER_STATISTICS*/ into twith n as (select rownum n from dual connect by level <= 1000)select n.n, ceil(sqrt(n.n)), TO_CHAR(TO_DATE(n.n,'j'),'jsp')from n;commit;

insert into twith n as (select 1000+rownum n from dual connect by level <= 10000)select n.n, ceil(sqrt(n.n)), TO_CHAR(TO_DATE(n.n,'j'),'jsp')from n;



NO STATISTICS AT ALL

If you don't have normal statistics then you don't get real-time statistics

Table Num

Name MON Rows

----- --- --------

T YES

Table Num


----- -------- ---------- ------------------------- -------

T SHARED

Column Num


------ ---------- -------- ---------------- ---------------- ------------------------------------- -------------------------------------

A NUMBER

A2 NUMBER

B VARCHAR2

no rows selected


NO_GATHER_OPTIMIZER_STATISTICS HINT


insert /*+APPEND*/ into twith n as (select rownum n from dual connect by level <= 1000)select n.n, ceil(sqrt(n.n)), TO_CHAR(TO_DATE(n.n,'j'),'jsp')from n;commit;

insert /*+NO_GATHER_OPTIMIZER_STATISTICS*/ into twith n as (select 1000+rownum n from dual connect by level <= 10000)select n.n, ceil(sqrt(n.n)), TO_CHAR(TO_DATE(n.n,'j'),'jsp')from n;



STATISTICS ON LOAD ONLY

Table Num

Name MON Rows

----- --- --------

T YES 1000

Table Num


----- -------- ---------- ------------------------- -------

T 1000 5 SHARED

Column Num


------ ---------- -------- ---------------- ---------------- ------------------------------------- -------------------------------------

A NUMBER 1000 C102 1 C20B 1000

A2 NUMBER 32 C102 1 C121 32


Table Column Num


----- ------ -------- ---------------- ---------------- ------------------------------------- ------------------------------------- -------------

NOTES

-------------------------

T A 1000 C102 1 C20B 1000 NONE

STATS_ON_LOAD

T A2 32 C102 1 C121 32 NONE

STATS_ON_LOAD


STATS_ON_LOAD



• In general, more accurate statistics will result in fewer problems if not better performance.

• Skewed data has always been a challenge

• It is still going to be a challenge with real-time statistics

• However, there is no substitute for maintaining appropriate explicit statistics

• Including during application execution if necessary!


HIGH-FREQUENCY AUTOMATIC OPTIMIZER STATISTICS COLLECTION


HIGH-FREQUENCY AUTOMATIC STATISTICS COLLECTION

• Scheduled PL/SQL Procedure: dbms_stats.gather_st_job_continuous_proc

• Very similar to automatic statistics collection job that runs in the maintenance window (auto optimizer stats collection).

• Not enabled by default. You have to turn it on.

• Applies to whole database.

• Key differences are limited run time, so it can be run more frequently.

• Only one instance can run concurrently. So only one CPU.

• It will run during the maintenance window

• Uses SYS.STATS_TARGET$ to drive processing

• Stalest tables within each object type first

• This also is used to prevent the two jobs running concurrently?

• Can interact with real-time statistics.


Cursor Invalidation

• Caused by collecting statistics

• Cursors might disappear quite quickly after a process completes as HFASC occurs

• Might increase parse overhead

CONSEQUENCES AND THREATS

Statistics History

• Old statistics and histograms saved to history when new collected.

• Increased volume of statistics history retained.

• Increased purge overhead.


LIMITED CONTROL WITH PREFERENCES

You can

• enable/disable job,

• set Execution Frequency,

• set Maximum Run-time,

• Omit tables by locking statistics

EXEC DBMS_STATS.SET_GLOBAL_PREFS('AUTO_TASK_STATUS','ON');

EXEC DBMS_STATS.SET_GLOBAL_PREFS('AUTO_TASK_INTERVAL','900');

EXEC DBMS_STATS.SET_GLOBAL_PREFS('AUTO_TASK_MAX_RUN_TIME','3600');


READ PREFERENCES TO GET CURRENT SETTINGS

SET LINESIZE 100 pages 99

COLUMN auto_task_status HEADING 'Auto Task|Status' FORMAT a10

COLUMN auto_task_max_run_time HEADING 'Auto Task|Max Run Time' FORMAT a15

COLUMN auto_task_interval HEADING 'Auto Task|Interval' FORMAT a10

SELECT DBMS_STATS.GET_PREFS('AUTO_TASK_STATUS') AS auto_task_status

, DBMS_STATS.GET_PREFS('AUTO_TASK_MAX_RUN_TIME') AS auto_task_max_run_time

, DBMS_STATS.GET_PREFS('AUTO_TASK_INTERVAL') AS auto_task_interval

FROM dual;

Auto Task Auto Task Auto Task

Status Max Run Time Interval

---------- --------------- ----------

OFF 3600 900

Default Settings


DBA_AUTO_STAT_EXECUTIONS

with x as (

SELECT x.*

, end_time-start_time diff

, start_time-(LAG(end_time,1) over (order by start_time)) start_lag

FROM DBA_AUTO_STAT_EXECUTIONS x

ORDER BY x.start_time

)

select opid, origin, status

, ((extract( day from start_lag )*24+

extract( hour from start_lag ))*60+

extract( minute from start_lag ))*60+

extract( second from start_lag ) start_lag

, start_time, end_time

, ((extract( day from diff )*24+

extract( hour from diff ))*60+

extract( minute from diff ))*60+

extract( second from diff ) secs

, completed, failed, timed_out, in_progress

from x

Where start_time >= sysdate-1/24

/


DBA_AUTO_STAT_EXECUTIONS

Report on each automatic statistics collection processStart Exec Timed In

OPID ORIGIN STATUS Lag START_TIME END_TIME Secs COMPLETED FAILED Out Progress

----- -------------------- ----------- -------- ----------------------------------- ----------------------------------- -------- --------- ------ ------ --------

8405 AUTO_TASK COMPLETED 313.873 01-JUL-19 08.16.09.885689 AM -05:00 01-JUL-19 08.16.22.277274 AM -05:00 12.392 45 1 0 0





























8468 AUTO_TASK IN PROGRESS 60.339 01-JUL-19 09.09.35.537039 AM -05:00 01-JUL-19 09.09.35.537039 AM -05:00 .000 66 0 0 1


SYS.STATS_TARGET$COLUMN Description

START_TIME Timestamp when statistics gathering started

END_TIME Timestamp when statistics gathering ended

STALENESS Staleness as a percentage derived from statistics and counts of table modifications statistics, but

limited to ±100

LEAST/GREATEST combination of ROWCNT, INSERTS, INSERTS+DELETE+UPDATES,

ROWCNT+INSERTS-DELETES/(ROWCNT+.01))

Also Derived from DBMS_STATS_INTERNAL.IS_STALE and DBMS_STATS_INTERNAL.BITCLR

OSIZE Size on object in bytes from DBMS_STATS_INTERNAL.GET_TABLE_BLOCK_COUNT or

DBMS_STATS_INTERNAL.SEGMENT_NUMBER_BLOCKS

OBJ# Object number – see SYS.OBJ$

TYPE# Object type – see DBA_OBJECTS or table above

FLAGS ?

STATUS 0=awaiting processing

1=in process

2=complete

?=timed out (because there is a timeout column on DBA_AUTO_STAT_EXECUTIONS

4=failed

SID SID of HFASC process

SERIAL# Serial# of HFASC process

PART# Partition/Sub-partition number

BO# Base Object ID for partitioned object


SYS.STATS_TARGET$

select

s.START_TIME, s.END_TIME

,s.STALENESS

,s.OSIZE

,s.OBJ#, name, NVL(subname,'<NULL>') subname

,s.TYPE#

,decode(s.type#, 0, 'NEXT OBJECT', 1, 'INDEX', 2, 'TABLE', 3, 'CLUSTER',

19, 'TABLE PARTITION', 20, 'INDEX PARTITION',

34, 'TABLE SUBPARTITION', 35, 'INDEX SUBPARTITION',

'UNDEFINED') object_type

,s.FLAGS, s.STATUS

,s.SID, s.SERIAL#

,s.PART#, s.BO#

from sys.stats_target$ s

left outer join sys.obj$ o on o.obj# = s.obj#

Where start_time >= sysdate-10/1440

order by start_time

/


SYS.STATS_TARGET$

START_TIME END_TIME STALENESS OSIZE OBJ#

----------------------------------- ----------------------------------- ---------- ---------- -------

NAME SUBNAME TYPE# OBJECT_TYPE FLAGS STATUS SID

------------------------------ ---------------------------------------- ----- ------------------ ----- ------ -----

SERIAL# PART# BO#

------- ---------- ----------

04-SEP-19 05.45.20.430796 PM +00:00 04-SEP-19 05.45.20.529349 PM +00:00 .4 131072 11576

WRM$_DATABASE_INSTANCE <NULL> 2 TABLE 288 2 17

28450

04-SEP-19 05.45.20.599045 PM +00:00 04-SEP-19 05.45.20.854072 PM +00:00 .5 262144 670

COL_USAGE$ <NULL> 2 TABLE 288 2 17

28450

04-SEP-19 05.45.20.887099 PM +00:00 04-SEP-19 05.45.20.961224 PM +00:00 .4 65536 88757

WRM$_PDB_IN_SNAP WRM$_PDB_IN_SNAP_1884370030_2326 19 TABLE PARTITION 352 2 17

28450 120 11594

04-SEP-19 05.45.20.937960 PM +00:00 04-SEP-19 05.45.20.958548 PM +00:00 -99 819200 88759

WRM$_PDB_IN_SNAP_PK WRM$_PDB_IN_SNAP_1884370030_2326 20 INDEX PARTITION 576 2 11

32191 120 11596

04-SEP-19 05.45.20.982490 PM +00:00 04-SEP-19 05.45.21.213654 PM +00:00 .4 589824 88701

WRH$_SYSMETRIC_HISTORY WRH$_SYSMETRIC_HISTORY_1884370030_2326 19 TABLE PARTITION 352 2 17

28450 321 11343

04-SEP-19 05.45.21.149670 PM +00:00 04-SEP-19 05.45.21.210769 PM +00:00 -99 819200 88703

WRH$_SYSMETRIC_HISTORY_INDEX WRH$_SYSMETRIC_HISTORY_1884370030_2326 20 INDEX PARTITION 576 2 11

32191 321 11345


CONCLUSIONS


BOTH FEATURES ARE EXADATA ONLY

19c Feature available on

• Oracle Database Enterprise Edition on Engineered Systems (EE-ES)

• Oracle Database Exadata Cloud Service (ExaCS)

• Database Licensing Information User Manual, 1.3 Permitted Features, Options, and Management Packs by Oracle Database Offering, Performance - Table 1-8

I think both features are primarily aimed at Autonomous Transaction Processing Databases

• Not ADW because you mostly direct path load, and don't update HCC objects

• It will also apply to Exadata.

• For example, they will apply to a classical ERP on Exadata

• But why not other platforms? It would help them also.

However, you can reproduce all the behaviour if you set

• alter system set "_exadata_feature_on"=TRUE scope=spfile;

• Obviously, you would NEVER do that in any other than a play database, but it does show that the feature is there and it works.

https://myoffice.accenture.com/personal/david_kurtz_accenture_com/Documents/Documents/word/Licensing%20Information,%201.3%20Permitted%20Features,%20Options,%20and%20Management%20Packs%20by%20Oracle%20Database%20Offering,%20Performance%20-%20Table%201-8



• Helps to avoid plan regressions caused by using stale data

• Clearly works well with uniformly distributed data

• Augments existing statistics with simple metrics:

• Row counts

• Min/max values

• Nothing that requires a scan of all the data

• No Histograms – doesn't handle skewed data well

• Based on table monitoring

CONCLUSIONS

HIGH-FREQUENCY AUTOMATIC STATISTICS COLLECTION

• It is a continuous maintenance window

• Recollects statistics that are stale

• Starting with most stale within object type

• Could it repeatedly get stuck on large but stale tables?


E-mail:

• [email protected]

78

QUESTIONS

mailto:[email protected]