Dbfs Mount

HistoryOracle Support - 04 March 2012 16:11:10 GMT+05:30 [ODM Solution/Action Plan]i) check the account status

SQL> select account_status from dba_users where username='<dbfs username>';

ii) Get the profile for dbfs user

SQL> select username,profile from dba_users where username='<dbfs username>';

iii) Alter the user profile to unlimited

SQL>ALTER PROFILE DEFAULT LIMIT PASSWORD_LIFE_TIME UNLIMITED;

iv) change the password of dbfs user

SQL>ALTER USER <dbfs username> IDENTIFIED BY <current DBFS password>';

v) now stop and start the dbfs resource and check whether it is mounting or not

Oracle Support - 04 March 2012 16:08:18 GMT+05:30 [ODM Proposed Solution Justif]check SAP

Oracle Support - 04 March 2012 16:08:02 GMT+05:30 [ODM Proposed Solution(s)]check SAP

Oracle Support - 04 March 2012 16:07:49 GMT+05:30 [ODM Cause Justification]dbfs user account enters the GRACE period for the password and giving warning message.

INFO : ERROR 28002 - ORA-28002: the password will expire within 6 days

Oracle Support - 04 March 2012 16:06:59 GMT+05:30 [ODM Cause Determination]dbfs.out~~~~~~~~~[4dd27940 03/04/12 05:00:58.012007 ./LcdfThreadPool.h:275 ] DEBUG: Retrying after some time .110. 1330855258 10139000[41d68940 03/04/12 05:00:59.870164 LcdfDBPool.cpp:188 ] ERROR: Failed to create session pool ret:1[41d68940 03/04/12 05:00:59.870204 LcdfDBPool.cpp:381 ] INFO : OCI_SUCCESS_WITH_INFO[41d68940 03/04/12 05:00:59.870225 LcdfDBPool.cpp:384 ] INFO : ERROR 28002 - ORA-28002: the password will expire within 6 days

[41d68940 03/04/12 05:00:59.870241 LcdfDBPool.cpp:250 ] DEBUG: Clean up OCI session pool...[41d68940 03/04/12 05:00:59.873070 LcdfSfs.cpp:89

OS logs

~~~~~~~~Mar 4 05:00:51 aodpdb02 snmpd[5538]: looks like a 64bit wrap, but prev!=newMar 4 05:00:59 aodpdb02 dbfs_client[31368]: Fail to set up database connection.

Oracle Support - 04 March 2012 07:55:31 GMT+05:30 [ODM Issue Clarification]Unable to mount the dbfs mount point

[oracle@aodpdb02 script]$ sh mount-dbfs.sh startmount-dbfs.sh mounting DBFS at /dbfs_direct from database dbmORACLE_SID is dbm2spawning dbfs_client command using SID dbm2Start -- OFFLINE

Oracle Support - 04 March 2012 16:04:49 GMT+05:30 [Call - Inbound]Call - Inbound----------------------------Customer contact name (if not same as in SR header): Verify preferred method of contact: PHONE OR My Oracle Support customer portal

Phone#: 9052409998Email: [email protected]

Comments for this update: Pavan called and told that the issue has been resolved after resetting the password of dbfs user.He has confirmed to close the SR

ACTION PLAN--------------------

Thank you.PrabuOracle Global Customer Support

Oracle Support - 04 March 2012 16:02:01 GMT+05:30 [Call - Inbound]

Oracle Support - 04 March 2012 16:01:38 GMT+05:30 [ODM Action Plan]It looks that dbfs user account enters the GRACE period for the password and giving warning message.

You need to alter the user password and password life to unlimited to avoid this issue in future

ACTION PLAN:-~~~~~~~~~~~~~~

i) check the account status

SQL> select account_status from dba_users where username='<dbfs username>';

ii) Get the profile for dbfs user

SQL> select username,profile from dba_users where username='<dbfs username>';

iii) Alter the user profile to unlimited

SQL>ALTER PROFILE DEFAULT LIMIT PASSWORD_LIFE_TIME UNLIMITED;

iv) change the password of dbfs user

SQL>ALTER USER <dbfs username> IDENTIFIED BY <current DBFS password>';

v) now stop and start the dbfs resource and check whether it is mounting or not

RegardsPrabu

Oracle Support - 04 March 2012 16:00:23 GMT+05:30 [ODM Data Collection]Name--------=== ODM Data Collection ===

dbfs.out~~~~~~~~~[4dd27940 03/04/12 05:00:58.012007 ./LcdfThreadPool.h:275 ] DEBUG: Retrying after some time .110. 1330855258 10139000[41d68940 03/04/12 05:00:59.870164 LcdfDBPool.cpp:188 ] ERROR: Failed to create session pool ret:1[41d68940 03/04/12 05:00:59.870204 LcdfDBPool.cpp:381 ] INFO : OCI_SUCCESS_WITH_INFO[41d68940 03/04/12 05:00:59.870225 LcdfDBPool.cpp:384 ] INFO : ERROR 28002 - ORA-28002: the password will expire within 6 days

[41d68940 03/04/12 05:00:59.870241 LcdfDBPool.cpp:250 ] DEBUG: Clean up OCI session pool...[41d68940 03/04/12 05:00:59.873070 LcdfSfs.cpp:89

OS logs~~~~~~~~Mar 4 05:00:51 aodpdb02 snmpd[5538]: looks like a 64bit wrap, but prev!=newMar 4 05:00:59 aodpdb02 dbfs_client[31368]: Fail to set up database connection.

FileName----------------

FileComment----------------------

[email protected] - 04 March 2012 15:46:36 GMT+05:30 [Update from Customer]Upload to gtcr successful for the file messages.txt.

[email protected] - 04 March 2012 15:46:11 GMT+05:30 [Update from Customer]Upload to gtcr successful for the file dbfs.out.

Oracle Support - 04 March 2012 15:24:58 GMT+05:30 [Notes]Generic Note------------------------

Hi,

I have reviewed the SR and uploaded logs.

We need to get the tracing of dbfs_client mount

I see that you are using wallet configuration for dbfs.

Please see the below given example to mount the dbfs with tracing enabled

su - oracle

export ORACLE_SID=<dbfs db sid>export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1cd $ORACLE_HOME/bindbfs_client <dbfs user>@ -otrace_file=/tmp/dbfs.out -otrace_level=1 -otrace_size=0 /<dbfs mount>

after entering the password,it will just hang there itself.

If its hangs there,please open another new terminal and check the below

++ ls -l /<dbfs mount>

++ cat /etc/mtab

++ upload /tmp/dbfs.out

++ latest OS logs -> /var/log/messages

RegardsPrabu

Oracle Support - 04 March 2012 15:13:00 GMT+05:30 [Notes]Generic Note------------------------Hi,

The SR has been transferred to me.I am reviewing the SR and will update you as soon as possible.

RegardsPrabu


Oracle Support - 04 March 2012 14:41:54 GMT+05:30 [Notes]Generic Note------------------------UPDATE========Going through similar SRs with us .

[email protected] - 04 March 2012 14:36:25 GMT+05:30 [Update from Customer]Hi Suppot,

Please find the output below..

[oracle@aodpdb02 bin]$ ./crsctl stat res -t--------------------------------------------------------------------------------NAME TARGET STATE SERVER STATE_DETAILS--------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------dbfs_mountONLINE ONLINE aodpdb01ONLINE OFFLINE aodpdb02OFFLINE OFFLINE aodpdb03OFFLINE OFFLINE aodpdb04ora.DATA_AODP.dgONLINE ONLINE aodpdb01ONLINE ONLINE aodpdb02ONLINE ONLINE aodpdb03ONLINE ONLINE aodpdb04ora.DBFS_DG.dgONLINE ONLINE aodpdb01ONLINE ONLINE aodpdb02ONLINE ONLINE aodpdb03ONLINE ONLINE aodpdb04ora.LISTENER.lsnrONLINE ONLINE aodpdb01ONLINE ONLINE aodpdb02ONLINE ONLINE aodpdb03ONLINE ONLINE aodpdb04ora.RECO_AODP.dgONLINE ONLINE aodpdb01ONLINE ONLINE aodpdb02ONLINE ONLINE aodpdb03ONLINE ONLINE aodpdb04ora.asmONLINE ONLINE aodpdb01 StartedONLINE ONLINE aodpdb02ONLINE ONLINE aodpdb03 StartedONLINE ONLINE aodpdb04 Startedora.gsdOFFLINE OFFLINE aodpdb01OFFLINE OFFLINE aodpdb02OFFLINE OFFLINE aodpdb03OFFLINE OFFLINE aodpdb04ora.net1.networkONLINE ONLINE aodpdb01ONLINE ONLINE aodpdb02ONLINE ONLINE aodpdb03ONLINE ONLINE aodpdb04ora.onsONLINE ONLINE aodpdb01ONLINE ONLINE aodpdb02ONLINE ONLINE aodpdb03ONLINE ONLINE aodpdb04ora.registry.acfsONLINE ONLINE aodpdb01ONLINE ONLINE aodpdb02ONLINE ONLINE aodpdb03ONLINE ONLINE aodpdb04--------------------------------------------------------------------------------

Cluster Resources--------------------------------------------------------------------------------ora.LISTENER_SCAN1.lsnr1 ONLINE ONLINE aodpdb03ora.LISTENER_SCAN2.lsnr1 ONLINE ONLINE aodpdb04ora.LISTENER_SCAN3.lsnr1 ONLINE ONLINE aodpdb01ora.aodpdb01.vip1 ONLINE ONLINE aodpdb01ora.aodpdb02.vip1 ONLINE ONLINE aodpdb02ora.aodpdb03.vip1 ONLINE ONLINE aodpdb03ora.aodpdb04.vip1 ONLINE ONLINE aodpdb04ora.biprd.biprd1.svc1 ONLINE ONLINE aodpdb01ora.biprd.biprd2.svc1 ONLINE ONLINE aodpdb02ora.biprd.biprd3.svc1 ONLINE ONLINE aodpdb03ora.biprd.biprd4.svc1 ONLINE ONLINE aodpdb04ora.biprd.db1 ONLINE ONLINE aodpdb01 Open2 ONLINE ONLINE aodpdb02 Open3 ONLINE ONLINE aodpdb03 Open4 ONLINE ONLINE aodpdb04 Openora.cvu1 ONLINE ONLINE aodpdb03ora.dbm.db1 ONLINE ONLINE aodpdb01 Open2 ONLINE ONLINE aodpdb02 Open3 ONLINE ONLINE aodpdb03 Open4 ONLINE ONLINE

Oracle Support - 04 March 2012 14:33:11 GMT+05:30 [Notes]Generic Note------------------------UPDATE=========All the resource crsctl stat res -t output

Oracle Support - 04 March 2012 14:30:03 GMT+05:30 [Notes]Generic Note------------------------UPDATE===========

Can you try mounting the /dbfs_direct

mount /dbfs_direct ( as non-root user )


ld

FileName----------------

FileComment----------------------echo $LD_LIBRARY_PATH[oracle@aodpdb02 bin]$ echo $LD_LIBRARY_PATH/u01/app/oracle/product/11.2.0/dbhome_1/lib


Doc File

FileName----------------

FileComment----------------------[oracle@aodpdb02 bin]$ ./crsctl stat resource dbfs_mount -t--------------------------------------------------------------------------------NAME TARGET STATE SERVER STATE_DETAILS--------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------dbfs_mountONLINE ONLINE aodpdb01ONLINE OFFLINE aodpdb02ONLINE OFFLINE aodpdb03ONLINE OFFLINE aodpdb04

b) Please send the output of <GRID_HOME>/crs/script/mount-dbfs.sh check[oracle@aodpdb02 script]$ ./mount-dbfs.sh checkCheck -- OFFLINE

==========================================================================================================================================================================================================================================c) Can you start mounting again with <GRID_HOME>/crs/script/mount-dbfs.sh start[oracle@aodpdb02 script]$ sh mount-dbfs.sh startmount-dbfs.sh mounting DBFS at /dbfs_direct from database dbmORACLE_SID is dbm2spawning dbfs_client command using SID dbm2Start -- OFFLINE

====================================================================================================================================================================================================================

======================

/var/log/messages

--------------------Mar 4 02:22:05 aodpdb02 DBFS: /u01/app/11.2.0/grid/crs/script/mount-dbfs.sh mounting DBFS at /dbfs_direct from database dbmMar 4 02:22:06 aodpdb02 ntpd[5630]: synchronized to LOCAL(0), stratum 10Mar 4 02:22:07 aodpdb02 DBFS: ORACLE_SID is dbm2Mar 4 02:22:07 aodpdb02 DBFS: spawning dbfs_client command using SID dbm2Mar 4 02:22:07 aodpdb02 DBFS: Password:Mar 4 02:22:07 aodpdb02 dbfs_client[12064]: Fail to set up database connection.Mar 4 02:22:08 aodpdb02 DBFS: Start OFFLINEMar 4 02:22:18 aodpdb02 DBFS: cleaning up DBFS using fusermount and kill on dbfs_client and mount.dbfsMar 4 02:23:17 aodpdb02 ntpd[5630]: synchronized to 168.189.30.14, stratum 2Mar 4 02:44:52 aodpdb02 dbfs_client[30552]: Fail to set up database connection.Mar 4 02:49:10 aodpdb02 sreport.sh: Starting new sos report at Sun Mar 4 02:49:10 EST 2012Mar 4 02:49:12 aodpdb02 sreport.sh: Saving sos report /var/log/cellos/sosreports in /dev/mapper/VGExaDb-LVDbSys1Mar 4 02:49:12 aodpdb02 sreport.sh: Report completed from starting date Sun Mar 4 02:49:10 EST 2012Mar 4 02:52:07 aodpdb02 dbfs_client[3854]: Fail to set up database connection.Mar 4 03:00:52 aodpdb02 DBFS: /u01/app/11.2.0/grid/crs/script/mount-dbfs.sh mounting DBFS at /dbfs_direct from database dbmMar 4 03:00:52 aodpdb02 DBFS: ORACLE_SID is dbm2Mar 4 03:00:52 aodpdb02 DBFS: spawning dbfs_client command using SID dbm2Mar 4 03:00:52 aodpdb02 DBFS: Password:Mar 4 03:00:53 aodpdb02 dbfs_client[10129]: Fail to set up database connection.Mar 4 03:00:53 aodpdb02 DBFS: Start OFFLINEMar 4 03:01:04 aodpdb02 DBFS: cleaning up DBFS using fusermount and kill on dbfs_client and mount.dbfsMar 4 03:06:46 aodpdb02 DBFS: mount-dbfs.sh mounting DBFS at /dbfs_direct from database dbmMar 4 03:06:47 aodpdb02 DBFS: ORACLE_SID is dbm2

==========================================================================================================================================================================================================================================[oracle@aodpdb02 script]$ sh mount-dbfs.sh stopmount-dbfs.sh: line 165: mount-dbfs.sh: command not foundfilesystem /dbfs_direct not currently mounted, no need to stop

==========================================================================================================================================================================================================================================

]$ sh mount-dbfs.sh startmount-dbfs.sh mounting DBFS at /dbfs_direct from database dbmORACLE_SID is dbm2spawning dbfs_client command using SID dbm2Start -- OFFLINE

=====================================================

=====================================================================================================================================================================================crsctl start resource dbfs_mount

[oracle@aodpdb02 bin]$ ./crsctl start resource dbfs_mountCRS-2672: Attempting to start 'dbfs_mount' on 'aodpdb04'CRS-2672: Attempting to start 'dbfs_mount' on 'aodpdb02'CRS-2672: Attempting to start 'dbfs_mount' on 'aodpdb03'CRS-2674: Start of 'dbfs_mount' on 'aodpdb02' failedCRS-2679: Attempting to clean 'dbfs_mount' on 'aodpdb02'CRS-2674: Start of 'dbfs_mount' on 'aodpdb04' failedCRS-2679: Attempting to clean 'dbfs_mount' on 'aodpdb04'CRS-2674: Start of 'dbfs_mount' on 'aodpdb03' failedCRS-2679: Attempting to clean 'dbfs_mount' on 'aodpdb03'CRS-2681: Clean of 'dbfs_mount' on 'aodpdb03' succeededCRS-2681: Clean of 'dbfs_mount' on 'aodpdb02' succeededCRS-2681: Clean of 'dbfs_mount' on 'aodpdb04' succeededCRS-4000: Command Start failed, or completed with errors.==========================================================================================================================================================================================================================================

Oracle Support - 04 March 2012 14:06:26 GMT+05:30 [Notes]Generic Note------------------------UPDATE==========Going through all that.Please wait

[email protected] - 04 March 2012 14:03:32 GMT+05:30 [Update from Customer]Upload to gtcr successful for the file strace.out.

[email protected] - 04 March 2012 14:03:22 GMT+05:30 [Update from Customer]Upload to gtcr successful for the file SR_ODM_ACTION_PLAN.docx.

[email protected] - 04 March 2012 14:02:36 GMT+05:30 [Update from Customer]Hi Suport,

Pleased find uploded files as per the ODM Action Plan.

Thanks,Pavan

Oracle Support - 04 March 2012 13:33:12 GMT+05:30 [Notes]Generic Note------------------------ACTION PLAN===============

a) Please send the output of

crsctl stat resource dbfs_mount -t

b) Please send the output of <GRID_HOME>/crs/script/mount-dbfs.sh check

c) Can you start mounting again with <GRID_HOME>/crs/script/mount-dbfs.sh start

d) Can you please upload /var/log/messages output during this time ?

e) Please stop it again >/crs/script/mount-dbfs.sh stop

f) Please start again using >/crs/script/mount-dbfs.sh start

g) Can you also start using crsctl

crsctl start resource dbfs_mount

h) Can you do the tail of scriptagent_oracle.log ( please upload the output )

i) Can you try mounting the /dbfs_direct

mount /dbfs_direct ( as non-root user )

Send the output please.

j) Can you check the LD_LIBRARY_PATH is set or not on NODE 1

echo $LD_LIBRARY_PATH

k) Can you make sure that the environment has :

$ export LD_LIBRARY_PATH=$ORACLE_HOME/lib

Then run the mount again with strace also :

$ strace -fae -o /tmp/strace.out dbfs_client ...

l ) Can we trace this using ?

How to trace DBFS when any failure happens [ID 1320683.1]

Please send the output of trace.

Oracle Support - 04 March 2012 13:11:00 GMT+05:30 [ODM Research]How to clean-up failed DBFS configuration (Doc ID 1312236.1)Accessing Dbfs Using Oracle Wallet Fails With Input/output error (Doc ID 1358194.1) How to trace DBFS when any failure happens [ID 1320683.1]Cluster Resource DBFS in not starting (Doc ID 1401303.1)Dbfs File System became unmounted due to mount-dbfs.sh timeout on Exadata Database Machine (Doc ID 1385927.1) Configuring DBFS on Oracle Database Machine (Doc ID 1054431.1)Errors when mounting a DBFS store using fstab due to missing libraries (Doc ID 1053008.1)

Oracle Support - 04 March 2012 13:04:12 GMT+05:30 [Notes]Generic Note

------------------------He informed , even reboot has not resolved the issue.

Oracle Support - 04 March 2012 13:03:41 GMT+05:30 [Notes]Generic Note------------------------UPDATE===========Called +91 9052409998 spoke to Pavan and informed that SR will be updated with action plan

Oracle Support - 04 March 2012 13:01:43 GMT+05:30 [Call Back - Customer]Call Back Customer------------------------------------Customer contact name (if not same as in SR header):

Phone#: +91 9052409998

Reason for Call Back? The customer called requesting call back

[email protected] - 04 March 2012 13:01:35 GMT+05:30 [Update from Customer]Hi Support,

As per the ODM action Plan we have rebooted the server, but the dbfs file system was not mounted..

Thanks,Pavan


Oracle Support - 04 March 2012 12:37:49 GMT+05:30 [Notes]Generic Note------------------------OWC Summary=============

Problem started after customer restarted the database.

The issue is only on node 2 .

DBBS is mounting successfully on Node 1 but this node shows OFFLINE

a) We checked TNS_ADMIN variable on both the nodes . It is not set

b) We tried to start using

crsctl start resource dbfs_mount

It was ONLINE on OTHER nodes but not on this node it was showing OFFLINE

c) We checked it using crsctl stat ( status )

It was again showing offline

We tried to stop and start but was not successful

d) We also tried to start using

>/crs/script/mount-dbfs.sh

But , it was still not mounting on Node 2

e) We checked listener and also the tnsping ( they are fine )

f) The difference is seen in following

cat /etc/mtab

dbfs-dbfs_user@:/ /dbfs_direct fuse rw,nosuid,nodev,max_read=1048576,default_permissions,allow_other,user=oracle 0 0

cat /etc/fstab/sbin/mount.dbfs#/@fsdb.local /dbfs_direct fuse rw,user,direct_io,allow_other,wallet,noauto 0 0

Node 2. txt

cat /etc/mtabNo entry for dbfs in mtab

Where as

cat /etc/fstabshows/sbin/mount.dbfs#/@fsdb.local /dbfs_direct fuse rw,user,direct_io,allow_other,wallet,noauto 0 0

g) /var/log/messages shows

DBFS: Fail to connect to database server

h) We also tried

fusermount -u /dbfs_direct"

and then /crs/script/mount-dbfs.sh start Was not successful

i) We checked scriptagent_oracle.log

SUSPECTING password issue ( not sure now ) Because It asks for password and next line shows

DBFS: Fail to connect to database server

Presently waiting for customer to reboot and then proceed further

Oracle Support - 04 March 2012 11:10:08 GMT+05:30 [Notes]Generic Note------------------------

UPDATE============

The

node1.txt=====================Shows

cat /etc/mtab

dbfs-dbfs_user@:/ /dbfs_direct fuse rw,nosuid,nodev,max_read=1048576,default_permissions,allow_other,user=oracle 0 0

cat /etc/fstab/sbin/mount.dbfs#/@fsdb.local /dbfs_direct fuse rw,user,direct_io,allow_other,wallet,noauto 0 0

Node 2. txt

cat /etc/mtabNo entry for dbfs in mtab

Where as

cat /etc/fstabshows/sbin/mount.dbfs#/@fsdb.local /dbfs_direct fuse rw,user,direct_io,allow_other,wallet,noauto 0 0

ACTION PLAN===================Please reboot this node and let us know the result We will resume OWC againsteps to power down or reboot an Exadata storage cell without affecting ASM:1188080.1

[email protected] - 04 March 2012 11:04:21 GMT+05:30 [Update from Customer]Upload to gtcr successful for the file node2.txt.

[email protected] - 04 March 2012 11:04:10 GMT+05:30 [Update from Customer]Upload to gtcr successful for the file node1.txt.

Oracle Support - 04 March 2012 09:27:01 GMT+05:30 [Notes]Generic Note------------------------Use Internet Explorer .Make sure popup blocker is disabled .Visit strtc.oracle.comThe conference id is 9726215The conference key is 16882554

Oracle Support - 04 March 2012 09:24:05 GMT+05:30 [Call - Outbound]Call - Outbound

------------------------------Discussing with Pavan at +91 9052409998

[email protected] - 04 March 2012 09:19:38 GMT+05:30 [Call - Inbound]Call - Inbound----------------------------Customer contact name (if not same as in SR header): Verify preferred method of contact: PHONE OR My Oracle Support customer portal

Phone#: +91 9052409998Email: [email protected]

Comments for this update: The customer called in requesting for urgent call back on this SR.

ACTION PLAN--------------------Contacting Engineer

Thank you.AsthaOracle Global Customer Support


Oracle Support - 04 March 2012 09:10:56 GMT+05:30 [Notes]Generic Note------------------------UPDATE==========a) Please provide the result of complete action plan

b) /usr/local/lib/libclntsh.so.11.1 ( If this is there , the link between dbhome_1/lib/libclntsh.so.11.1 and /usr/lib seems missing )

c) Please compare the dbhome_1/lib/libclntsh.so.11.1 to other node ( ls -all ) Is there any link available on that node ?


a)Previously its working fine,b)only on one node

./crsctl stat res -t

dbfs_mountONLINE ONLINE aodpdb01ONLINE OFFLINE aodpdb02ONLINE ONLINE aodpdb03ONLINE ONLINE aodpdb04

=======================================================================

ls -all /u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1-rwxr-xr-x 1 oracle oinstall 52378028 Jan 27 2011 /u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1==============================================================================================fstab

/sbin/mount.dbfs#/@fsdb.local /dbfs_direct fuse rw,user,direct_io,allow_other,wallet,noauto 0 0

==================================================================

/var/log/messages

Mar 3 21:22:36 aodpdb02 auditd[5101]: Audit daemon rotating log filesMar 3 21:52:48 aodpdb02 DBFS: /u01/app/11.2.0/grid/crs/script/mount-dbfs.sh mounting DBFS at /dbfs_direct from database dbmMar 3 21:52:49 aodpdb02 DBFS: ORACLE_SID is dbm2Mar 3 21:52:49 aodpdb02 DBFS: spawning dbfs_client command using SID dbm2Mar 3 21:52:49 aodpdb02 DBFS: Password:Mar 3 21:52:49 aodpdb02 dbfs_client[5383]: Fail to set up database connection.Mar 3 21:52:50 aodpdb02 DBFS: Start OFFLINEMar 3 21:53:00 aodpdb02 DBFS: cleaning up DBFS using fusermount and kill on dbfs_client and mount.dbfsMar 3 22:07:37 aodpdb02 auditd[5101]: Audit daemon rotating log files

Oracle Support - 04 March 2012 08:13:44 GMT+05:30 [Notes]Generic Note------------------------UPDATE=========The problem seems related to locating the library files here .

SOME QUESTIONS================a) Was it working previously ?b) Is the problem only node specific and happening on all the nodesc) Can you mount the mount points using mount -a ( as root user ) d) Are there any entries in /etc/fstab related to DBFS

ACTION PLAN----------------------

a) Do you have this file /usr/local/lib/libclntsh.so.11.1 ? ( To check if file is present )

b) Can you list the output of ls -all /u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1

c) Are you seeing all services up and running using crs ?

d) Can you follow the article and let us know the result

Errors when mounting a DBFS store using fstab due to missing libraries (Doc ID 1053008.1)

e) Can you try export LD_LIBRARY_PATH=$ORACLE_HOME/lib and then see if that

helps ?

We might take strace if above does not resolve the case.

f) We might check the script output of

Configuring DBFS on Oracle Database Machine (Doc ID 1054431.1)

Oracle Support - 04 March 2012 08:05:30 GMT+05:30 [ODM Research]Configuring DBFS on Oracle Database Machine (Doc ID 1054431.1)Errors when mounting a DBFS store using fstab due to missing libraries (Doc ID 1053008.1) Bug 9309554: ERRORS WHEN MOUNTING A DBFS STORE USING FSTAB DUE TO MISSING LIBRARIESBug 10337293: TEMPORARILY UNABLE TO MOUNT DBFSeventually, after just waiting some time the mounts were able to work.

Oracle Support - 04 March 2012 07:55:57 GMT+05:30 [ODM Issue Verification]Verified the issue in the log file as noted in SR Itself

Oracle Support - 04 March 2012 07:48:56 GMT+05:30 [ODM Research]Configuring DBFS on Oracle Database Machine (Doc ID 1054431.1)Errors when mounting a DBFS store using fstab due to missing libraries (Doc ID 1053008.1)

Oracle Support - 04 March 2012 07:46:13 GMT+05:30 [Call - Outbound]Call - Outbound------------------------------Called +91-9052409998 spoke to Pavan who informed this is Half Rack Exadata environment .He just restarted the database using shutdown immediate command and after it came up , this problem is seen now.Database is up and running .Informed Pavan , the problem will be analyzed and SR will be updated with action plan.

[email protected] - 04 March 2012 07:45:31 GMT+05:30 [Update from Customer]Database Servers: Exadata SoftwareVersion & PatchesExadata Image - Version: 11.2.2.2Database Servers: Grid InfrastructureVersion & PatchesVersion 11.2.0.2.0BP3 - 10387939Database Servers: Oracle RDBMS Version& PatchesVersion 11.2.0.2.0BP3 - 10387939Storage Servers Exadata Software andPatchesExadata Image - Version: 11.2.2.2


We getting the below error while mounting dbfs_client in production

/u01/app/oracle/product/11.2.0/dbhome_1/bin/dbfs_client: error while loading shared libraries: libclntsh.so.11.1: cannot open shared object file: No such file or directory

Thanks,Pavan+91-9052409998

[email protected] - 04 March 2012 07:34:07 GMT+05:30 [Administration]#### SEVERITY 1 REQUEST TEMPLATE #####

Can you easily recover from, bypass or work around the problem?

Does your system or application continue normally after the problem occurs?

Are the standard features of the system or application still available; is the loss of service minor?

Is this a Data Corruption issue?

Do you have a Complete Loss of Service?

Is this a Test or Production Environment?

Are you unable to proceed with implementation and the go live date is in less than 3 days?

Please provide a description of the business impact:

ALL PHONE NUMBERS SHOULD INCLUDE THE COUNTRY CODE AND AREA CODE.

Customers Management name:Customer's Management 24x7 contact number: Primary Customer contact nameCurrent Customer 24x7 voice phone number Current Customer Email address: Current Customer Pager/Fax number: Secondary Contact Name: Secondary Contact Phone number: Secondary Contact Email:

Comments for this update:

ACTION PLAN--------------------

[email protected] - 04 March 2012 07:33:20 GMT+05:30 [Customer Problem Description]Problem Description: The customer will update SR with Problem Description

Documents

Dbfs Mount