interview que -ORACLE RAC (11g + 12c)


!! ORACLE RAC (11g + 12c)!!
















--------------------------------------------------------------------------
                            ORACLE RAC (11g + 12c)
--------------------------------------------------------------------------


*. Mention what is cluster?
    A cluster is referred to a group of independent, but connected servers that behaves as a single system.
       
*.What is RAC?
    RAC stands for Real Application cluster.
    It is a clustering solution from Oracle Corporation that ensures high availability of databases by providing instance failover, media failover features.
    Oracle RAC is a cluster database with a shared cache architecture that overcomes the limitations of traditional shared-nothing and shared-disk approaches to provide a highly scalable and available database solution for all the business applications.
    Oracle RAC provides the foundation for enterprise grid computing.

*.    What is Oracle RAC One Node?
    Oracle RAC one Node is a single instance running on one node of the cluster while the 2nd node is in cold standby mode. If the instance fails for some reason then RAC one node detect it and restart the instance on the same node or the instance is relocate to the 2nd node incase there is failure or fault in 1st node. The benefit of this feature is that it provides a cold failover solution and it automates the instance relocation without any downtime and does not need a manual intervention. Oracle introduced this feature with the release of 11gR2 (available with Enterprise Edition).

*. What is RAC and how is it different from non RAC databases?
    Oracle Real Application clusters allows multiple instances to access a single database, the instances will be running on multiple nodes. 
    In Real Application Clusters environments, all nodes concurrently execute transactions against the same database.
    Real Application Clusters coordinates each node's access to the shared data to provide consistency and integrity.'

*. What are the advantages of RAC (Real Application Clusters)?
    Reliability - if one node fails, the database won't fail'
    Availability - nodes can be added or replaced without having to shutdown the database
    Scalability - more nodes can be added to the cluster as the workload increases

*. What is Cache Fusion?
    Oracle RAC is composed of two or more instances. When a block of data is read from datafile by an instance within the cluster and another instance is in need of the same block, it is easy to get the block image from the instance which has the block in its SGA rather than reading from the disk. To enable inter instance communication Oracle RAC makes use of interconnects. The Global Enqueue Service (GES) monitors and Instance enqueue process manages the cache fusion.

*. How do we verify that RAC instances are running?
    SQL>select * from V$ACTIVE_INSTANCES;
    The query gives the instance number under INST_NUMBER column,host_:instancename under INST_NAME column.

*.Which is the "MASTER NODE" in RAC?
    The node with the lowest node number will become master node and dynamic remastering of the resources will take place.
    To find out the master node for particular resource, you can query v$ges_resource for MASTER_NODE column.
    To find out which is the master node, you can see ocssd.log file and search for "master node number".
    when the first master node fails in the cluster the lowest node number will become master node.   
   
*. Give few examples for solutions that support cluster storage?
    ·ASM (automatic storage management),
    ·Raw disk devices,
    ·Network file system (NFS),
    ·OCFS2 and
    ·OCFS (Oracle Cluster Fie systems).
   
*. What are Oracle Cluster Components?
    1.Cluster Interconnect (HAIP)
    2.Shared Storage (OCR/Voting Disk)
    3.Clusterware software
    4.Oracle Kernel Components

*. What are Oracle RAC Components?
    VIP, Node apps etc.
   
*.What are Oracle Kernel Components?
    Basically Oracle kernel need to switched on with RAC On option when you convert to RAC, that is the difference as it facilitates few RAC bg process like LMON,LCK,LMD,LMS etc.
   
   
*. What are Oracle database background processes specific to RAC?
    Oracle RAC is composed of two or more database instances. They are composed of Memory structures and background processes same as the single instance database.Oracle RAC instances use two processes GES(Global Enqueue Service), GCS(Global Cache Service) that enable cache fusion.Oracle RAC instances are composed of following background processes:
    ACMS—Atomic Controlfile to Memory Service (ACMS)
    GTX0-j—Global Transaction Process
    LMON—Global Enqueue Service Monitor
    LMD—Global Enqueue Service Daemon
    LMS—Global Cache Service Process
    LCK0—Instance Enqueue Process
    RMSn—Oracle RAC Management Processes (RMSn)
    RSMN—Remote Slave Monitor
    To ensure that each Oracle RAC database instance obtains the block that it needs to satisfy a query or transaction, Oracle RAC instances use two processes, the Global Cache Service (GCS) and the Global Enqueue Service (GES). The GCS and GES maintain records of the statuses of each data file and each cached block using a Global Resource Directory (GRD). The GRD contents are distributed across all of the active instances.
   
*. How to turn on RAC?
    # link the oracle libraries
    $ cd $ORACLE_HOME/rdbms/lib
    $ make -f ins_rdbms.mk rac_on
    # rebuild oracle
    $ cd $ORACLE_HOME/bin
    $ relink oracle
   
   
*. Disk architechture in RAC?
    SAN (Storage Area Networks) - generally using fibre to connect to the SAN
    NAS (Network Attached Storage) - generally using a network to connect to the NAS using either NFS, ISCSI

*. What are the Oracle Clusterware key components?
    Oracle Clusterware has two key components Cluster Registry OCR and Voting Disk.

*.    What is Voting Disk and OCR?
    • Voting Disk :
    Oracle RAC uses the voting disk to manage cluster membership by way of a health check and arbitrates cluster ownership among the instances in case of network failures. The voting disk must reside on shared disk.
    A node must be able to access more than half of the voting disks at any time.
    For example, if you have 3 voting disks configured, then a node must be able to access at least two of the voting disks at any time. If a node cannot access the minimum required number of voting disks it is evicted, or removed, from the cluster.

    • Oracle Cluster Registry (OCR) :
    The cluster registry holds all information about nodes, instances, services and ASM storage if used, it also contains state information ie they are available and up or similar.
    The OCR must reside on shared disk that is accessible by all of the nodes in your cluster.
   
   
*.What are the administrative tasks involved with voting disk? [428681.1]
    Following administrative tasks are performed with the voting disk :
    1) Backing up voting disks        ./ocrconfig -export /backup/ocrvot_bkp/ocr.dmp or  dd if=/dev/raw/raw2 of=/backup/ocrvot_bkp/voting.dmp
    2) Recovering Voting disks
    3) Adding voting disks             ./ocrconfig -add /dev/raw/raw3 OR  ./crsctl add css votedisk /dev/raw/raw5 -force
    4) Deleting voting disks
    5) Moving voting disks
   
# show backups
ocrconfig -showbackup

# to change the location of the backup, you can even specify a ASM disk
ocrconfig -backuploc <path|+asm>

# perform a backup, will use the location specified by the -backuploc location
ocrconfig -manualbackup

# perform a restore
ocrconfig -restore <file>

# delete a backup
orcconfig -delete <file>

Note: there are many more option so see the ocrconfig man page ## add/relocate the ocrmirror file to the specified location
ocrconfig -replace ocrmirror '/ocfs2/ocr2.dbf'

## relocate an existing OCR file
ocrconfig -replace ocr '/ocfs1/ocr_new.dbf'

## remove the OCR or OCRMirror file
ocrconfig -replace ocr
ocrconfig -replace ocrmirror
   
*. Can you add voting disk online? Do you need voting disk backup?
    Yes,  as per documentation, if you have multiple voting disk you can add online, but if you have only one voting disk , by that cluster will be down as its lost you just need to start crs in exclusive mode and add the votedisk using
    crsctl add votedisk <path>
   
   
*. What are the major RAC wait events?
    In a RAC environment the buffer cache is global across all instances in the cluster and hence the processing differs.The most common wait events related to this are gc cr request and gc buffer busy

    • GC CR request :the time it takes to retrieve the data from the remote cache
    Reason: RAC Traffic Using Slow Connection or Inefficient queries (poorly tuned queries will increase the amount of data blocks requested by an Oracle session. The more blocks requested typically means the more often a block will need to be read from a remote instance via the interconnect.)

    • GC BUFFER BUSY: It is the time the remote instance locally spends accessing the requested data block.
   
*. What do you do if you see GC CR BLOCK LOST in top 5 Timed Events in AWR Report?
    This is most likely due to a fault in interconnect network.
    Check netstat -s
    if you see "fragments dropped" or "packet reassemblies failed" , Work with your system administrator find the fault with network.
   
*.What is GRD?
    GRD stands for Global Resource Directory. The GES and GCS maintains records of the statuses of each datafile and each cahed block using global resource directory.This process is referred to as cache fusion and helps in data integrity.

*.What is ACMS?
    ACMS stands for Atomic Controlfile Memory Service.In an Oracle RAC environment ACMS is an agent that ensures a distributed SGA memory update(ie)SGA updates are globally committed on success or globally aborted in event of a failure.

*. What is SCAN listener?
    A scan listener is something that additional to node listener which listens the incoming db connection requests from the client which got through the scan IP, it got end points configured to node listener where it routes the db connection requests to particular node listener.

    SCAN IP can be disabled if not required. However SCAN IP is mandatory during the RAC installation. Enabling/disabling SCAN IP is mostly used in oracle apps environment by the concurrent manager (kind of job scheduler in oracle apps).
   
*. What is SCAN? (11gR2 feature)
-->
    Single Client Access Name (SCAN) is s a new Oracle Real Application Clusters (RAC) 11g Release 2
    feature that provides a single name for clients to access an Oracle Database running in a cluster. The benefit
    is clients using SCAN do not need to change if you add or remove nodes in the cluster.

    SCAN provides a single domain name via (DNS), allowing and-users to address a RAC cluster as-if it were a
    single IP address. SCAN works by replacing a hostname or IP list with virtual IP addresses (VIP).

    Single client access name (SCAN) is meant to facilitate single name for all Oracle clients to connect to the cluster
    database, irrespective of number of nodes and node location. Until now, we have to keep adding multiple address
    records in all clients tnsnames.ora, when a new node gets added to or deleted from the cluster.

    Single Client Access Name (SCAN) eliminates the need to change TNSNAMES entry when nodes are added to or removed from the
    Cluster. RAC instances register to SCAN listeners as remote listeners. Oracle recommends assigning 3 addresses to SCAN,
    which will create 3 SCAN listeners, though the cluster has got dozens of nodes.. SCAN is a domain name registered to at
    least one and up to three IP addresses, either in DNS (Domain Name Service) or GNS (Grid Naming Service). The SCAN must
    resolve to at least one address on the public network. For high availability and scalability, Oracle recommends configuring
    the SCAN to resolve to three addresses.tly used in oracle apps environment by the concurrent manager (kind of job scheduler in oracle apps).

*.  What is difference between RAC ip addresses ?

    Public IP adress : is the normal IP address typically used by DBA and SA to manage storage,
                       system and database. Public IP addresses are reserved for the Internet.
    Private IP address : is used only for internal clustering processing (Cache Fusion) (aka as interconnect).
                       Private IP addresses are reserved for private networks.
    VIP : is used by database applications to enable fail over when one cluster node fails. The purpose for having VIP is so
        client connection can be failover to surviving nodes in case there is failure           
       
*. What is split brain?

    When database nodes in a cluster are unable to communicate with each other, they may continue to process and modify the
    data blocks independently. If the same block is modified by more than one instance, synchronization/locking of the data
    blocks does not take place and blocks may be overwritten by others in the cluster. This state is called split brain.
   
*. What is cache fusion?
    In a RAC environment, it is the combining of data blocks, which are shipped across the interconnect from remote database caches (SGA) to the local node, in order to fulfill the requirements for a transaction (DML, Query of Data Dictionary).
   
*.    What is the difference between Crash recovery and Instance recovery?
    When an instance crashes in a single node database on startup a crash recovery takes place. In a RAC enviornment the same recovery for an instance is performed by the surviving nodes called Instance recovery.

*.What is an interconnect network?(private )
    An interconnect network is a private network that connects all of the servers in a cluster. The interconnect network uses a switch/multiple switches that only the nodes in the cluster can access.

*. How can we configure the cluster interconnect?
    · Configure User Datagram Protocol (UDP) on Gigabit Ethernet for cluster interconnects.
    · On UNIX and Linux systems we use UDP and RDS (Reliable data socket) protocols to be used by Oracle Clusterware.
    · Windows clusters use the TCP protocol.
*.  What is VIP IP used for?
    It returns a dead connection IMMIDIATELY, when its primary node fails. Without using VIP IP, the clients have to wait around 10 minutes to receive ORA-3113: “end of file on communications channel”. However, using Transparent Application Failover (TAF) could avoid ORA-3113.

*. What is the purpose of Private Interconnect?
    Clusterware uses the private interconnect for cluster synchronization (network heartbeat) and daemon communication between the the clustered nodes. This communication is based on the TCP protocol.
    RAC uses the interconnect for cache fusion (UDP) and inter-process communication (TCP). Cache Fusion is the remote memory mapping of Oracle buffers, shared between the caches of participating nodes in the cluster.
   
*. What are SCAN components in a cluster?
    1.SCAN Name
    2.SCAN IPs (3)
    3.SCAN Listeners (3)
*. What is FAN?
    Fast application Notification as it abbreviates to FAN relates to the events related to instances,services and nodes.This is a notification mechanism that Oracle RAc uses to notify other processes about the configuration and service level information that includes service status changes such as,UP or DOWN events.Applications can respond to FAN events and take immediate action.
   
*. What is TAF?
    TAF (Transparent Application Failover) is a configuration that allows session fail-over between different nodes of a RAC database cluster.
    Transparent Application Failover (TAF). If a communication link failure occurs after a connection is established, the connection fails over to another active node. Any disrupted transactions are rolled back, and session properties and server-side program variables are lost. In some cases, if the statement executing at the time of the failover is a Select statement, that statement may be automatically re-executed on the new connection with the cursor positioned on the row on which it was positioned prior to the failover.

    After an Oracle RAC node crashes—usually from a hardware failure—all new application transactions are automatically rerouted to a specified backup node. The challenge in rerouting is to not lose transactions that were "in flight" at the exact moment of the crash. One of the requirements of continuous availability is the ability to restart in-flight application transactions, allowing a failed node to resume processing on another server without interruption. Oracle's' answer to application failover is a new Oracle Net mechanism dubbed Transparent Application Failover. TAF allows the DBA to configure the type and method of failover for each Oracle Net client.
    TAF architecture offers the ability to restart transactions at either the transaction (SELECT) or session level.   
   
*. What are the characteristics controlled by Oracle services feature?
    The characteristics include a unique name, workload balancing, failover options, and high availability.
   


   
   
Process Name            Functionality
crsd                    •The CRS daemon (crsd) manages cluster resources based on configuration information that is stored in                          Oracle Cluster Registry (OCR) for each resource. This includes start, stop, monitor, and failover                           operations. The crsd process generates events when the status of a resource changes.
cssd                    •Cluster Synchronization Service (CSS): Manages the cluster configuration by controlling which nodes                          are members of the cluster and by notifying members when a node joins or leaves the cluster. If you                          are using certified third-party clusterware, then CSS processes interfaces with your clusterware to                          manage node membership information. CSS has three separate processes: the CSS daemon (ocssd), the CSS                            Agent (cssdagent), and the CSS Monitor (cssdmonitor). The cssdagent process monitors the cluster                         and provides input/output fencing. This service formerly was provided by Oracle Process Monitor                         daemon (oprocd), also known as OraFenceService on Windows. A cssdagent failure results in Oracle C                           lusterware restarting the node.
diskmon                    •Disk Monitor daemon (diskmon): Monitors and performs input/output fencing for Oracle Exadata Storage                           Server. As Exadata storage can be added to any Oracle RAC node at any point in time, the diskmon                            daemon is always started when ocssd is started.
evmd                    •Event Manager (EVM): Is a background process that publishes Oracle Clusterware events
mdnsd                    •Multicast domain name service (mDNS): Allows DNS requests. The mDNS process is a background process                          on Linux and UNIX, and a service on Windows.
gnsd                    •Oracle Grid Naming Service (GNS): Is a gateway between the cluster mDNS and external DNS servers. The                             GNS process performs name resolution within the cluster.
ons                        •Oracle Notification Service (ONS): Is a publish-and-subscribe service for communicating Fast                             Application Notification (FAN) events
oraagent                •oraagent: Extends clusterware to support Oracle-specific requirements and complex resources. It runs                             server callout scripts when FAN events occur. This process was known as RACG in Oracle Clusterware                             11g Release 1 (11.1).
orarootagent            •Oracle root agent (orarootagent): Is a specialized oraagent process that helps CRSD manage resources                             owned by root, such as the network, and the Grid virtual IP address
oclskd                    •Cluster kill daemon (oclskd): Handles instance/node evictions requests that have been escalated to CSS
gipcd                    •Grid IPC daemon (gipcd): Is a helper daemon for the communications infrastructure
ctssd                    •Cluster time synchronisation daemon(ctssd) to manage the time syncrhonization between nodes, rather                         depending on NTP   
   


*.  Under which user or owner the process will start?
    (http://db.geeksinsight.com/2012/09/28/11g-grid-infrastructure-rac-crs-startup-sequence/)

    Component                                        Name of the Process                    Owner
   
    Oracle High Availability Service                ohasd                                init, root
    Cluster Ready Service (CRS)                        Cluster Ready Services                root
    Cluster Synchronization Service (CSS)            ocssd,cssd monitor, cssdagent        grid owner
    Event Manager (EVM)                                evmd, evmlogger                        grid owner
    Cluster Time Synchronization Service (CTSS)        octssd                                root
    Oracle Notification Service (ONS)                ons, eons                            grid owner
    Oracle Agent                                    oragent                                grid owner
    Oracle Root Agent                                orarootagent                        root
    Grid Naming Service (GNS)                        gnsd                                root
    Grid Plug and Play (GPnP)                        gpnpd                                grid owner
    Multicast domain name service (mDNS)            mdnsd                                grid owner
   


   
*. What enables the load balancing of applications in RAC?
    Oracle Net Services enable the load balancing of application connections across all of the instances in an Oracle RAC database.

*. LOAD BALANCING in RAC:-
    The Oracle RAC system can distribute the load over many nodes this feature called as load balancing.

    There are two methods of load balancing
    1.Client load balancing
    2.Server load balancing

    1.Client Load Balancing
        Client Load Balancing distributes new connections among Oracle RAC nodes so that no one server is overloaded with connection requests and it is configured at net service name level by providing multiple descriptions in a description list or multiple addresses in an address list. For example, if connection fails over to another node in case of failure, the client load balancing ensures that the redirected connections are distributed among the other nodes in the RAC.

        Configure Client-side connect-time load balancing by setting LOAD_BALANCE=ON in the corresponding client side TNS entry.

                    TESTRAC =
                    (DESCRIPTION =
                    (ADDRESS_LIST=
                    (LOAD_BALANCE = ON)
                    (ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1-VIP)(PORT = 1521))
                    (ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC2-VIP)(PORT = 1521))
                    )
                    (CONNECT_DATA = (SERVICE_NAME = testdb.selectstarfrom.com))
                    )
                   
        2.Server Load Balancing
            Server Load Balancing distributes processing workload among Oracle RAC nodes. It divides the connection load evenly between all available listeners and distributes new user session connection requests to the least loaded listener(s) based on the total number of sessions which are already connected. Each listener communicates with the other listener(s) via each database instance’s PMON process.

            Configure Server-side connect-time load balancing feature by setting REMOTE_LISTENERS initialization parameter of each instance to a TNS name that describes list of all available listeners.

                        TESTRAC_LISTENERS =
                        (DESCRIPTION =
                        (ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1)(PORT = 1521)))
                        (ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC2)(PORT = 1521))))
                        )
                       
                Set *.remote_listener= TESTRAC_LISTENERS’ initialization parameter in the database’s shared SPFILE and add TESTRAC_LISTENERS’ entry to the TNSNAMES.ORA file in the Oracle Home of each node in the cluster.

    Once you configure Server-side connect-time load balancing, each database’s PMON process will automatically register the database with the database’s local listener as well as cross-register the database with the listeners on all other nodes in the cluster. Now the nodes themselves decide which node is least busy, and then will connect the client to that node.

*.AILOVER in RAC:-
    The Oracle RAC system can protect against failures caused by O/S or server crashes or hardware failures. When a node failure occurs in RAC system, the connection attempts can fail over to other surviving nodes in the cluster this feature called as Failover.

    There are two methods of failover
    1. Connection Failover
    2. Transparent Application Failover (TAF)
   
        1. Connection Failover
        If a connection failure occurs at connect time, the application failover the connection to another active node in the cluster. This feature enables client to connect to another listener if the initial connection to the first listener fails.
   
        Enable client-side connect-time Failover by setting FAILOVER=ON in the corresponding client side TNS entry.

                            TESTRAC =
                            (DESCRIPTION =
                            (ADDRESS_LIST=
                            (LOAD_BALANCE = ON)
                            (FAILOVER = ON)
                            (ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1-VIP)(PORT = 1521))
                            (ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC2-VIP)(PORT = 1521))
                            )
                            (CONNECT_DATA = (SERVICE_NAME = testdb.selectstarfrom.com))
                            )
                           
    If LOAD_BALANCE is set to on then clients randomly attempt connections to any nodes. If client made connection attempt to a down node, the client needs to wait until it receives the information that the node is not accessible before trying alternate address in ADDRESS_LIST.

    2. Transparent Application Failover (TAF)
            If connection failure occurs after a connection is established, the connection fails over to other surviving nodes. Any uncommitted transactions are rolled back and server side program variables and session properties will be lost. In some case the select statements automatically re-executed on the new connection with the cursor positioned on the row on which it was positioned prior to the failover.

                        TESTRAC =
                        (DESCRIPTION =
                        (LOAD_BALANCE = ON)
                        (FAILOVER = ON)
                        (ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1-VIP)(PORT = 1521))
                        (ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1-VIP)(PORT = 1521))
                        (CONNECT_DATA =
                        (SERVICE_NAME = testdb.selectstarfrom.com)
                        (FAILOVER_MODE = (TYPE = SELECT)(METHOD = BASIC)(RETRIES = 180)(DELAY = 5))
                        )
                        )
   

*. What is the difference between CRSCTL and SRVCTL?
    crsctl manages clusterware-related operations:
    Starting and stopping Oracle Clusterware
    Enabling and disabling Oracle Clusterware daemons
    Registering cluster resources

    srvctl manages Oracle resource–related operations:
    Starting and stopping database instances and services
    Also from 11gR2 manages the cluster resources like network,vip,disks etc
   
   
*.What is hangcheck timer?
    The hangcheck timer checks regularly the health of the system. If the system hangs or stop the node will be restarted automatically.
    There are 2 key parameters for this module:
    -> hangcheck-tick: this parameter defines the period of time between checks of system health. The default value is 60 seconds; Oracle recommends setting it to 30seconds.
    -> hangcheck-margin: this defines the maximum hang delay that should be tolerated before hangcheck-timer resets the RAC node.

   
*. State the initialization parameters that must have same value for every instance in an Oracle RAC database?
    Some initialization parameters are critical at the database creation time and must have same values.Their value must be specified in SPFILE or PFILE for every instance.The list of parameters that must be identical on every instance are given below:
    ACTIVE_INSTANCE_COUNT
    ARCHIVE_LAG_TARGET
    COMPATIBLE
    CLUSTER_DATABASE
    CLUSTER_DATABASE_INSTANCE
    CONTROL_FILES
    DB_BLOCK_SIZE
    DB_DOMAIN
    DB_FILES
    DB_NAME
    DB_RECOVERY_FILE_DEST
    DB_RECOVERY_FILE_DEST_SIZE
    DB_UNIQUE_NAME
    INSTANCE_TYPE (RDBMS or ASM)
    PARALLEL_MAX_SERVERS
    REMOTE_LOGIN_passWORD_FILE
    UNDO_MANAGEMENT
   
   
*. How to find location of OCR file when CRS is down?
    If you need to find the location of OCR (Oracle Cluster Registry) but your CRS is down.
    On Linux: /etc/oracle/ocr.loc
    On Solaris: /var/opt/oracle/ocr.loc


   
   
   
COMMANDS : ----------------------------------->



COMMAND                                                     PERPOSE                             USER
#crsctl stop crs                                            STOP crs                             root
#crsctl disable crs
#crsctl enable crs
#crsctl start crs
$crs_stat or
$crsctl stat res -t ------> 11gr2
$crsctl query crs activeversion
$crsctl query crs softwareversion [node_name]
$crsctl start crs
$crsctl stop crs
#/etc/init.d/init.crs start
#/etc/init.d/init.crs stop
#/etc/init.d/init.cssd stop
#/etc/init.d/init.cssd start
$crsctl enable crs
$crsctl disable crs
#/etc/init.d/init.crs enable
#/etc/init.d/init.crs disable
$crsctl check crs
$crsctl check cluster [-node node_name]
$crsctl check cssd
$crsctl check crsd
$crsctl check evmd
$crsctl check css votedisk
$crsctl query css votedisk
$crsctl add css votedisk <PATH>
$crsctl delete css votedisk <PATH>
$crsctl start resources
$crsctl stop resources
$crs_stat
$crs_register
$crs_unregister
$crs_start
$crs_stop
$crs_getperm
$crs_profile
$crs_relocate
$crs_setperm

Dual Environment CRSCTL Commands:

    crsctl add resource
    crsctl add type
    crsctl check css
    crsctl delete resource
    crsctl delete type
    crsctl get hostname
    crsctl getperm resource
    crsctl getperm type
    crsctl modify resource
    crsctl modify type
    crsctl setperm resource
    crsctl setperm type
    crsctl start resource
    crsctl status resource
    crsctl status type
    crsctl stop resource
   
cemulto -n        ---CLUSTER NAME
gsdctl             ---global service demon
oifcfg getif    ---oracle interphse
    oicfg setif -global <interface name>/<subnet>:public
    oicfg setif -global <interface name>/<subnet>:cluster_interconnect

olsnodes -n/l/g


SRVCTL COMMANDS

#srvctl stop instance -d <database_name> -n <node_name>
#srvctl stop vip -n <node_name> -f
$srvctl config nodeapps -a
$ifconfig -a
#$GRID_HOME/bin/srvctl start scan
#$GRID_HOME/bin/srvctl start scan_listener
#$GRID_HOME/bin/srvctl stop scan_listener
#$GRID_HOME/bin/srvctl stop scan
#$GRID_HOME/bin/srvctl status scan_listener
#$GRID_HOME/bin/srvctl status scan
#$GRID_HOME/bin/srvctl config scan
$nslookup <scan-name>




Directory Path                                                     Contents
-------------------------------------------------------------------------------------------------------------------------
GRID_HOME/log/<host>/alert<host>.log                          -- Clusterware alert log
GRID_HOME/log/<host>/diskmon                                 -- Disk Monitor Daemon
GRID_HOME/log/<host>/client                                 -- OCRDUMP, OCRCHECK, OCRCONFIG, CRSCTL
GRID_HOME/log/<host>/ctssd                                     -- Cluster Time Synchronization Service
GRID_HOME/log/<host>/gipcd                                     -- Grid Interprocess Communication Daemon
GRID_HOME/log/<host>/ohasd                                     -- Oracle High Availability Services Daemon
GRID_HOME/log/<host>/crsd                                     -- Cluster Ready Services Daemon
GRID_HOME/log/<host>/gpnpd                                     -- Grid Plug and Play Daemon
GRID_HOME/log/<host>/mdnsd                                     -- Mulitcast Domain Name Service Daemon
GRID_HOME/log/<host>/evmd                                     -- Event Manager Daemon
GRID_HOME/log/<host>/racg/racgmain                             -- RAC RACG
GRID_HOME/log/<host>/racg/racgeut                             -- RAC RACG
GRID_HOME/log/<host>/racg/racgevtf                             -- RAC RACG
GRID_HOME/log/<host>/racg                                     -- RAC RACG (only used if pre-11.1 database is installed)
GRID_HOME/log/<host>/cssd                                     -- Cluster Synchronization Service Daemon
GRID_HOME/log/<host>/srvm                                     -- Server Manager
GRID_HOME/log/<host>/agent/ohasd/oraagent_oracle11             -- HA Service Daemon Agent
GRID_HOME/log/<host>/agent/ohasd/oracssdagent_root             -- HA Service Daemon CSS Agent
GRID_HOME/log/<host>/agent/ohasd/oracssdmonitor_root         -- HA Service Daemon ocssdMonitor Agent
GRID_HOME/log/<host>/agent/ohasd/orarootagent_root             -- HA Service Daemon Oracle Root Agent
GRID_HOME/log/<host>/agent/crsd/oraagent_oracle11             -- CRS Daemon Oracle Agent
GRID_HOME/log/<host> agent/crsd/orarootagent_root             -- CRS Daemon Oracle Root Agent
GRID_HOME/log/<host> agent/crsd/ora_oc4j_type_oracle11g     -- CRS Daemon Oracle OC4J Agent
GRID_HOME/log/<host>/gnsd                                     -- Grid Naming Service Daemon