OraVR

DNS Configuration For SCAN

The DNS it stands for Domain Name Server (DNS) is used to translate/resolve hostnames into IP addresses, and the IP address into Hostnames. DNS zones are used to translate IP addresses into hostnames. It is also used to deliver additional types of information to DNS clients. An increasing number of services depend on DNS, which is why configuring DNS is an important task for Linux administrators and its palys most important role in oracle RAC cluster with help of SCAN service .

BIND is most widely used DNS (Domain Name Server) software. Its name originates as an acronym of Berkeley Internet Name Domain. BIND is also called by it service name i.e. named (or Name Daemon). BIND latest version 9 is available now and distributed under Mozilla Public License (MPL). BIND is developed and maintained by Internet Systems Consortium (ISC).

Almost every Internet connection starts with a DNS lookup. Hostname to IP resolution is necessary before sending an email or browsing a website and BIND is the preferred DNS server for Unix/Linux operating systems.

Record types in DNS

A : Record: The Address record holds The IP Address of the hostname.It resolves IP into Hostname.

PTR Record : The Pointer Record resolves hostname into IP

CNAME : cname allows multiple names for each TCP/IP Address

MX RECORD : it stands for mail exchange record, It’s responsible for Mail Hostnames.

HINFO : this record holds Hosts TCP/IP Address

NS RECORD : The Name Server record simply specifies the other name servers for the domain.

SOA RECORD : It stands Start of authority. This record identifies the zone and some parameters like the source host, serial number, refresh time, retry time, A expire time. Time To Live (TTL)

Resource Record Types and it’s used.

A (IPv4 address) : Maps a hostname to an IPv4 address

AAAA (IPv6 address) : Maps a hostname to an IPv6 address.

CNAME (canonical name) : An alias for one name to another name that should have an A or AAAA record.

NS (name server) : Maps a domain name to a DNS name server that is authoritative for the DNS zone.

PTR (pointer) : Maps an IP address (v4 or v6) to a hostname.

MX (mail exchange) : Indicates which MTA mail servers are used within a DNS domain.

SOA (start of authority : Contains generic information about how a DNS zone works. It contains information about who is responsible for the administration of the domain.

TXT (text) : Maps a name to human readable text. This type of resource record is for instance used by protocols like Send Policy Framework, which in email is used to verify the name of the domain an email message was received from.

SRV (service) : Indicates which host to contact for specific services such as LDAP and Kerberos.

DNS have two Zone files:

Zones contain all domain information. Basically, the zone refers to the branch of the DNS tree for which a specific name server is responsible.

1.forward lookup Zone– it contain IP address to hostname information

2.Reverse lookup Zone: it contains hostname to IP address information.

1. Static IP address :

This is must and recommended to have a static IP configured on the DNS NIC.

2. /etc/hosts

To set hostname resolution, DNS is typically used. Configuring DNS not only for hostname resolution it will help us control to communicate with DNS Clients. Apart from DNS, you can configure host name resolution in the /etc/hosts file. Setting up an /etc/hosts file is easy; just make sure that it contains at least two columns. The first column has the IP address of the specific host, and the second column specifies the hostname Ex: localhost.localdoamin localhost.

Example :

[root@dns78 ~]# ifconfig

ens161: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.168.45.2 netmask 255.255.255.0 broadcast 192.168.46.255

[root@dns78 network-scripts]# hostnamectl

Static hostname: dns78.oravr.in

Icon name: computer-vm

Chassis: vm

Machine ID: 84e2a73b3d60416a87a548ee73dfb7f3

Boot ID: 41192f4980424572a965d536725aec3f

Virtualization: vmware

Operating System: Oracle Linux Server 7.8

CPE OS Name: cpe:/o:oracle:linux:7:8:server

Kernel: Linux 4.14.35-1902.304.6.3.el7uek.x86_64

Architecture: x86-64

[root@dns78 network-scripts]# hostname -i

192.168.45.2

[root@dns78 network-scripts]# cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

192.168.45.2 dns78.oravr.in dns78

3. To install DNS packages : we will use the yum command, as yum takes care to install the dependency packages by itself if any. If you want to Configure Local Yum Repository on RHEL

#yum install -y bind*

4. Enabling DNS IP to accept the DNS requests

A. Named.conf

Once the DNS bind packages are installed, next step is to enable the DNS configuration to enable named service to accept the request on DNS Server IP. In our lab setup, it is 192.168.45.2 To perform these changes manually edit the /etc/named.conf file.

This file we define directory (where zonefiles are located ) and zone name

[root@dns78 ~]# cat /etc/named.conf

options

{

directory "/var/named";

listen-on port 53 { any; };

};

zone "oravr.in"

{

type master;

file "oravr.in.fwd.zone";

};

zone "localhost"

{

type master;

file "localhost.fwd.zone";

};

zone "45.168.192.in-addr.arpa"

{

type master;

file "192.168.45.rev.zone";

};

zone "0.0.127.in-addr.arpa"

{

type master;

file "localhost.rev.zone";

};

[root@dns78 ~]#

B. Defines Zones

check the named.conf ..

Directory : /var/named

Port : 53

Allow : Any

Type Zone Files

Forword oravr.in oravr.in.fwd.zone

Forword localhost localhost.fwd.zone

Reverse 45.168.192.in-addr.arpa 192.168.45.rev.zone

Reverse 0.0.127.in-addr.arpa localhost.rev.zone

Lets create the files :

Domain Forword lookup zone (www dont miss )

[root@dns78 named]# cat oravr.in.fwd.zone

$TTL 1D

@ IN SOA dns78.oravr.in. root.localhost (

20200808 ; serial

8H ; refresh

4H ; retry

1W ; expiry

1D) ; minimum

@ IN NS dns78.oravr.in.

localhost IN A 127.0.0.1

dns78 IN A 192.168.45.2

rac-gns IN A 192.168.45.9

www IN A 192.168.45.2

west01 IN A 192.168.45.101

west02 IN A 192.168.45.102

west03 IN A 192.168.45.103

west01-vip IN A 192.168.45.104

west02-vip IN A 192.168.45.105

west03-vip IN A 192.168.45.106

westscan IN A 192.168.45.107

IN A 192.168.45.108

IN A 192.168.45.109

westgg-vip IN A 192.168.45.110

westoem IN A 192.168.45.90

$ORIGIN pune.oravr.in

@ IN NS rac-gns.oravr.in.

IN NS dns78.oravr.in.

rac-gns IN A 192.168.45.9; glue record

Domain reverse lookup zone (www dont miss )

[root@dns78 named]# cat 192.168.45.rev.zone

$TTL 1D

@ IN SOA dns78.oravr.in. root.localhost (

20200808 ; serial

8H ; refresh

4H ; retry

1W ; expiry

1D) ; minimum

@ IN NS dns78.oravr.in.

@ IN PTR oravr.in.

2 IN PTR dns78.oravr.in.

9 IN PTR rac-gns.oravr.in.

101 IN PTR west01.oravr.in.

102 IN PTR west02.oravr.in.

103 IN PTR west03.oravr.in.

104 IN PTR west01-vip.oravr.in.

105 IN PTR west02-vip.oravr.in.

106 IN PTR west03-vip.oravr.in.

107 IN PTR westscan.oravr.in.

108 IN PTR westscan.oravr.in.

109 IN PTR westscan.oravr.in.

110 IN PTR westgg-vip.oravr.in.

90 IN PTR westoem.oravr.in.

www IN PTR 192.168.45.2

localhost Foreword lookup zone

[root@dns78 named]# cat localhost.fwd.zone

$TTL 1D

@ IN SOA dns78.oravr.in. root.localhost (

20200808 ; serial

8H ; refresh

4H ; retry

1W ; expiry

1D) ; minimum

IN NS @

IN A 127.0.0.1

Localhost reverse lookup zone

[root@dns78 named]# cat localhost.rev.zone

$TTL 1D

@ IN SOA dns78.oravr.in. root.localhost (

20200808 ; serial

8H ; refresh

4H ; retry

1W ; expiry

1D) ; minimum

IN NS localhost.

1 IN PTR localhost.

C. Check zone files :

[root@dns78 named]# ls -alrts *.zone

4 -rw-r--r--. 1 named root 514 Aug 8 19:12 localhost.fwd.zone

4 -rw-r--r--. 1 named root 531 Aug 8 19:12 localhost.rev.zone

4 -rw-r--r--. 1 named root 1653 Aug 9 15:38 oravr.in.fwd.zone

4 -rw-r--r--. 1 named root 1528 Aug 9 15:38 192.168.45.rev.zone

D. Permission

[root@dns78 named]# chgrp named -R /var/named

[root@dns78 named]# chown -v root:named /etc/named.conf

[root@dns78 named]# named-checkzone localhost localhost.fwd.zone

zone localhost/IN: loaded serial 20200808

[root@dns78 named]# named-checkzone oravr.in oravr.in.fwd.zone

zone oravr.in/IN: loaded serial 20200808

[root@dns78 named]# named-checkzone localhost localhost.fwd.zone

zone localhost/IN: loaded serial 20200808

[root@dns78 named]# named-checkzone 45.168.192.in-addr.arpa 192.168.45.rev.zone

zone 45.168.192.in-addr.arpa/IN: loaded serial 20200808

[root@dns78 named]# named-checkzone 0.0.127.in-addr.arpa localhost.rev.zone

zone 0.0.127.in-addr.arpa/IN: loaded serial 20200808

[root@dns78 named]#

5. Firewall

[root@dns78 named]# firewall-cmd --permanent --add-port=53/udp

[root@dns78 named]# firewall-cmd --permanent --add-service=dns

success

[root@dns78 named]# firewall-cmd --reload

success

6. Resolve.conf

( Use below file to client servers )

[root@dns78 named]# cat /etc/resolv.conf

# Generated by NetworkManager

search oravr.in

nameserver 192.168.45.2

7 . Enable and restart named services :

[root@dns78 named]#systemctl enable --now named.service

Created symlink from /etc/systemd/system/multi-user.target.wants/named.service to /usr/lib/systemd/system/named.service.

Restart the named service and validate the loaded zones

[root@dns78 named]# service named restart

Redirecting to /bin/systemctl restart named.service

[root@dns78 named]# service named status

Redirecting to /bin/systemctl status named.service

● named.service - Berkeley Internet Name Domain (DNS)

Loaded: loaded (/usr/lib/systemd/system/named.service; enabled; vendor preset: disabled)

Active: active (running) since Mon 2020-08-17 18:23:50 IST; 2s ago

Process: 28877 ExecStop=/bin/sh -c /usr/sbin/rndc stop > /dev/null 2>&1 || /bin/kill -TERM $MAINPID (code=exited, status=0/SUCCESS)

Process: 4887 ExecReload=/bin/sh -c /usr/sbin/rndc reload > /dev/null 2>&1 || /bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)

Process: 28894 ExecStart=/usr/sbin/named -u named -c ${NAMEDCONF} $OPTIONS (code=exited, status=0/SUCCESS)

Process: 28891 ExecStartPre=/bin/bash -c if [ ! "$DISABLE_ZONE_CHECKING" == "yes" ]; then /usr/sbin/named-checkconf -z "$NAMEDCONF"; else echo "Checking of zone files is disabled"; fi (code=exited, status=0/SUCCESS)

Main PID: 28897 (named)

Tasks: 4

CGroup: /system.slice/named.service

└─28897 /usr/sbin/named -u named -c /etc/named.conf

Aug 17 18:23:50 dns78.oravr.in named[28897]: command channel listening on ::1#953

Aug 17 18:23:50 dns78.oravr.in named[28897]: managed-keys-zone: loaded serial 0

Aug 17 18:23:50 dns78.oravr.in named[28897]: zone 0.0.127.in-addr.arpa/IN: loaded serial 20200808

Aug 17 18:23:50 dns78.oravr.in named[28897]: zone localhost/IN: loaded serial 20200808

Aug 17 18:23:50 dns78.oravr.in named[28897]: zone 45.168.192.in-addr.arpa/IN: loaded serial 20200808

Aug 17 18:23:50 dns78.oravr.in named[28897]: zone oravr.in/IN: loaded serial 20200808

Aug 17 18:23:50 dns78.oravr.in named[28897]: all zones loaded

Aug 17 18:23:50 dns78.oravr.in named[28897]: running

Aug 17 18:23:50 dns78.oravr.in named[28897]: zone localhost/IN: sending notifies (serial 20200808)

Aug 17 18:23:50 dns78.oravr.in named[28897]: zone 0.0.127.in-addr.arpa/IN: sending notifies (serial 20200808)

[root@dns78 named]#

8. check the Resolution of host and ip address

NS Lookup :

[root@dns78 named]# nslookup westscan

Server: 192.168.45.2

Address: 192.168.45.2#53

Name: westscan.oravr.in

Address: 192.168.45.109

Name: westscan.oravr.in

Address: 192.168.45.108

Name: westscan.oravr.in

Address: 192.168.45.107

[root@dns78 named]# nslookup 192.168.45.109

109.45.168.192.in-addr.arpa name = westscan.oravr.in.

[root@dns78 named]# nslookup 192.168.45.108

108.45.168.192.in-addr.arpa name = westscan.oravr.in.

[root@dns78 named]# nslookup 192.168.45.107

107.45.168.192.in-addr.arpa name = westscan.oravr.in

[root@dns78 named]#

DIG:

[root@dns78 named]# dig westscan.oravr.in

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-16.P2.el7_8.6 <<>> westscan.oravr.in

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39242

;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 1, ADDITIONAL: 2

;; OPT PSEUDOSECTION:

; EDNS: version: 0, flags:; udp: 4096

;; QUESTION SECTION:

;westscan.oravr.in. IN A

;; ANSWER SECTION:

westscan.oravr.in. 86400 IN A 192.168.45.108

westscan.oravr.in. 86400 IN A 192.168.45.109

westscan.oravr.in. 86400 IN A 192.168.45.107

;; AUTHORITY SECTION:

oravr.in. 86400 IN NS dns78.oravr.in.

;; ADDITIONAL SECTION:

dns78.oravr.in. 86400 IN A 192.168.45.2

;; Query time: 0 msec

;; SERVER: 192.168.45.2#53(192.168.45.2)

;; WHEN: Mon Aug 17 18:55:08 IST 2020

;; MSG SIZE rcvd: 134

[root@dns78 named]# dig oravr.in

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-16.P2.el7_8.6 <<>> oravr.in

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17812

;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:

; EDNS: version: 0, flags:; udp: 4096

;; QUESTION SECTION:

;oravr.in. IN A

;; AUTHORITY SECTION:

oravr.in. 86400 IN SOA dns78.oravr.in. root.localhost.oravr.in. 20200808 28800 14400 604800 86400

;; Query time: 2 msec

;; SERVER: 192.168.45.2#53(192.168.45.2)

;; WHEN: Mon Aug 17 18:55:17 IST 2020

;; MSG SIZE rcvd: 98

[root@dns78 named]#

9. once ip resolve lets validate the dns ip swap behavior in round robin method

Example : westscan.oravr.in can swap the ips in RR method more details as below

Every times its resolve and assign new ip address

[root@dns78 named]# ping westscan.oravr.in.

PING westscan.oravr.in (192.168.45.107) 56(84) bytes of data.

64 bytes from westscan.oravr.in(192.168.45.107): icmp_seq=1 ttl=64 time=0.619 ms

--- westscan.oravr.in ping statistics ---

1 packets transmitted, 1 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.619/0.619/0.619/0.000 ms

[root@dns78 named]# ping westscan.oravr.in.

PING westscan.oravr.in (192.168.45.108) 56(84) bytes of data.

64 bytes from westscan.oravr.in (192.168.45.108): icmp_seq=1 ttl=64 time=0.741 ms

--- westscan.oravr.in ping statistics ---

1 packets transmitted, 1 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.741/0.741/0.741/0.000 ms

[root@dns78 named]# ping westscan.oravr.in.

PING westscan.oravr.in (192.168.45.108) 56(84) bytes of data.

64 bytes from westscan.oravr.in (192.168.45.108): icmp_seq=1 ttl=64 time=0.200 ms

--- westscan.oravr.in ping statistics ---

1 packets transmitted, 1 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.200/0.200/0.200/0.000 ms

[root@dns78 named]# ping westscan.oravr.in.

PING westscan.oravr.in (192.168.45.108) 56(84) bytes of data.

64 bytes from westscan.oravr.in(192.168.45.108): icmp_seq=1 ttl=64 time=0.499 ms

^V64 bytes from westscan.oravr.in (192.168.45.108): icmp_seq=3 ttl=64 time=0.331 ms

64 bytes from westscan.oravr.in(192.168.45.108): icmp_seq=4 ttl=64 time=0.205 ms

--- westscan.oravr.in ping statistics ---

4 packets transmitted, 4 received, 0% packet loss, time 3062ms

rtt min/avg/max/mdev = 0.205/0.342/0.499/0.106 ms

[root@dns78 named]# ping westscan.oravr.in.

PING westscan.oravr.in (192.168.45.109) 56(84) bytes of data.

64 bytes from westscan.oravr.in (192.168.45.109): icmp_seq=1 ttl=64 time=0.639 ms

--- westscan.oravr.in ping statistics ---

1 packets transmitted, 1 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.639/0.639/0.639/0.000 ms

[root@dns78 named]#

Oracle RAC and MTU

on Rac Environment UDP protocol is used for cache fusion. (BTW, UDP and RDS protocols are supported in UNIX platform --Exadata ; whereas Windows uses TCP protocol).

UDP : is an higher level protocol stack, and it is implemented over IP Protocol ( UDP/IP). Cache Fusion uses UDP protocol to send packets over the wire (Exadata uses RDS protocol though).

UDP protocol, fragmentation, and assembly

The Maximum Transmission Unit (MTU) size is the maximum packet size in bytes that a network interface can send over a network device. This size includes the protocol information, such as the TCP and IP headers. This size does not contain the Ethernet header. This study is concerned with the MTU settings that can be set in the configuration files in SLES, for example, in /etc/sysconfig/network/ifcfg-qeth-bus-ccw-0.0.0xxx for each device. The effective MTU size is also negotiated during the connection creation with the other end point, and the MTU can be reduced to the capacity of that device.

When the MTU is smaller than the packet size, the data must be divided into MTU sized packets. On the receiving side the packets must be reassembled, thus creating overhead that scales with the quantity of packages required. When the MTU is only slightly too small, there can also be wasted bandwidth. For example, when the MTU is slightly smaller than the average packet size the secondary packet can be almost empty.

An MTU size bigger than the packet size should have no impact on the network bandwidth, if all intermediate nodes on the path to the final destination support the MTU size, because only the real data are sent. But the number of packages processed in parallel is calculated with the TPC/IP windows size and the MTU size. When the window size stays constant and the MTU size is increased, this number is decreased, which might lead to a performance degradation, when the smaller MTU size would be appropriate as well.

Let us consider an example of MTU set to 3000 in a network interface. One 8K block transfer can not be performed with just one IP packet as the IP packet size (3000 bytes) is less than 8K. So, one transfer of UDP packet of 8K size is fragmented to 3 IP packets and sent over the wire. In the receiving side, those 3 packets are reassembled to create one UDP buffer of size 8K. After the assembly, that UDP buffer is delivered to an UDP port of a UNIX process. Usually, a foreground process will listen on that port to receive the UDP buffer.

Consider what happens If MTU is set to 9000 in the network interface: Then 8K buffer can be transmitted over the wire with just one IP packet. There is no need for fragmentation or reassembly with MTU=9000 as long as the block size is less than 8K. MTU=9000 is also known as jumbo frame configuration. ( But, if the database block size is greater than jumbo frame then fragmentation and reassembly is still required. For example, for 16KB size, with MTU=9000, there will three 1K IP packets and one 7K IP packet to be transmitted).

Fragmentation and reassembly is performed at OS Kernel layer level and hence it is the responsibility of Kernel and the stack below to complete the fragmentation and assembly. Oracle code simply calls the send and receive system calls, passes the buffers to populate.

UDP vs TCP

From RAC point of view, UDP is the right choice over TCP for cache fusion traffic. With TCP/IP, for every packet transfer has overhead, connection need to be setup, packet sent, and the process must wait for TCP Acknowledgement before considering the packet send as complete. In a busy RAC systems, we are talking about 2-3 milli-seconds for packet transfer and with TCP/IP, we probably may not be able to achieve that level of performance. With UDP, packet transfer is considered complete, as soon as packet is sent and error handling is done by Oracle code itself. As you know, reliable network is a key to RAC stability, if much of packets (closer to 100%) are sent without any packet drops, UDP is a good choice over TCP/IP for performance reasons.

If there are reassembly failures, then it is a function of unreliable network or kernel or something else, but nothing to do with the choice of UDP protocol itself. Of course, RDS is better than UDP as the error handling is offloaded to the fabric, but usually require, infiniband fabric for a proper RDS setup. For that matter, VPN connections use UDP protocol too.

What happens if an IP packet is lost, assuming MTU=3000 bytes?

From the wireshark files with mtu3000, you will see that each of the packet have a Fragment offset. That fragment offset and IP identification is used to reassemble the IP packets to create 8K UDP buffer. Consider that there are 3 puzzle pieces, each puzzle piece with markings, and Kernel uses those markings( offset and IP ID) to reassemble the packets. Let’s consider the case, one of 3 packet never arrived, then the kernel threads will keep those 2 IP packets in memory for 64 seconds( Linux kernel parameter ipfrag_time controls that time) before declaring reassembly failure. Without receiving the missing IP packet, kernel can not reassemble the UDP buffer, and so, reassembly failure is declared.

Oracle foreground process will wait for 30 seconds (it used to be 300 seconds or so in older version of RAC) and if the packet is not arrived within that timeout period, FG process will declare a ("gc cr block lost" and/or "gc current block lost") and re-request the block. Of course, kernel memory allocated for IP fragmentation and assembly is constrained by Kernel parameter ipfrag_high_thres and ipfrag_low_thres and lower values for these kernel parameters can lead to reassembly failures too (and that’s why it is important to follow all best practices from RAC installation guides).

[root@west02 ~]# netstat -s |grep assem

453433 reassemblies required

43380 packets reassembled ok

3180797 packet reassembles failed --- if GC loss occurred and cluster wait increase

Oracle Clusterware & Oracle RAC instances rely on heartbeats for node memberships. If network Heartbeats are consistently dropped, Instance/Node eviction may occur. The Symptoms below are therefore relevant for Node/Instance evictions.

1. "gc cr block lost" / "gc current block lost" in top 5 or significant wait event

2. SQL traces report multiple gc cr requests / gc current request /

3. gc cr multiblock requests with long and uniform elapsed times

4. Poor application performance / throughput

5. Packet send/receive errors as displayed in ifconfig or vendor supplied utility

6. Netstat reports errors/retransmits/reassembly failures

SQL SCRIPT :

SELECT A.VALUE "GC BLOCKS LOST 1",
B.VALUE "GC BLOCKS CORRUPT 1",
C.VALUE "GC BLOCKS LOST 2",
D.VALUE "GC BLOCKS CORRUPT 2"
FROM GV$SYSSTAT A,
GV$SYSSTAT B,
GV$SYSSTAT C,
GV$SYSSTAT D
WHERE A.INST_ID = 1
AND A.NAME = 'gc blocks lost'
AND B.INST_ID = 1
AND B.NAME = 'gc blocks corrupt'
AND C.INST_ID = 2
AND C.NAME = 'gc blocks lost'
AND D.INST_ID = 2
AND D.NAME = 'gc blocks corrupt';

The parameters in /proc/sys/net/ipv4 control various aspects of the network, including a parameter that controlls the reassembly buffer size.

ipfrag_high_threshold specifies that maximum amount of memory used to reassemble IP fragments. When the memory used by fragments reaches ipfrag_high_threshold, old entries are removed until the memory used declines to ipfrag_low_threshold.

If the output of netstat shows increasing amounts of IP fragment reassembles failing, we recommend to increase ipfrag_high_threshold. The impact can be significant. In some use cases, increasing this buffer space improved throughput from 32MB/sec to 80MB/sec.

To temporarily change the value of ipfrag_high_threshold, use this command as root:

[root@west01 ~]# cat /proc/sys/net/ipv4/ipfrag_high_thresh

4194304

[root@west01 ~]# cat /proc/sys/net/ipv4/ipfrag_low_thresh

3145728

Change Parameter :

[root@west01 ~]#echo "Value" > /proc/sys/net/ipv4/ipfrag_high_threshold

[root@west02 ~]# sysctl -a|grep ipfrag

net.ipv4.ipfrag_high_thresh = 4194304

net.ipv4.ipfrag_low_thresh = 3145728

net.ipv4.ipfrag_max_dist = 64

net.ipv4.ipfrag_secret_interval = 0

net.ipv4.ipfrag_time = 30

BTW, there are few other reasons for 'gc lost packets' too. High CPU usage also can lead to 'gc lost packets' failures too, as the process may not have enough cpu time to drain the buffers, network buffers allocated for that process becomes full, and so, kernel will drop incoming packets.

It is probably better to explain these concepts visually. So, I created a video. When you watch this video, notice that there is HD button on the top of the video. Play this in HD mode so that you will have better learning experience.

Oracle Enterprise Session Border Controller (E-SBC) : connects disparate Internet Protocol (IP) communications networks while mitigating security threats, curing interoperability problems, and ensuring reliable communications. It protects and controls real-time voice, video, and Unified Communications (UC) as they traverse IP network borders.

Errors :

Process termination requested for pid 6440 [source = rdbms], [info = 2] [request issued by pid: 52630, uid: 211819489]

Errors in file /opt/oracle/app/orcl/diag/rdbms/regtxndb/regtxndb1/trace/regtxndb1_m000_52909.trc (incident=121694) (PDBNAME=CDB$ROOT):

ORA-00603: ORACLE server session terminated by fatal error

ORA-27504: IPC error creating OSD context

ORA-27300: OS system dependent operation:sendmsg failed with status: 105

ORA-27301: OS failure message: No buffer space available

ORA-27302: failure occurred at: sskgxpsnd2

RAC Installation on 19c

Install 19c 3 node RAC
1. Install Esxi Host
3. Configure RHEL 7.8
2. DNS Server
3. Install Storage Openfiler
4. Install Redhat 7.8
5. Configure the nodes
6. Grid Setup
7. RDBMS setup
8. DBCA
9. Patch the database
10. validations

1. Network Interfaces :

. Private Network

. Public Netwok

. Optional for access internet (depend on requirement)

--/etc/hosts : Scan ip must be resolve in DNS

#----------------------------------woravr-----------------------------#

#Public

192.168.45.101 woravr01.vishalbh.com woravr01

192.168.45.102 woravr02.vishalbh.com woravr02

192.168.45.103 woravr03.vishalbh.com woravr03

#Private

192.168.46.101 woravr01-priv.vishalbh.com woravr01-priv

192.168.46.102 woravr02-priv.vishalbh.com woravr02-priv

192.168.46.103 woravr03-priv.vishalbh.com woravr03-priv

#vip

192.168.45.104 woravr01-vip.vishalbh.com woravr01-vip

192.168.45.105 woravr02-vip.vishalbh.com woravr02-vip

192.168.45.106 woravr03-vip.vishalbh.com woravr03-vip

#scan

192.168.45.107 woravrscan.vishalbh.com woravrscan

192.168.45.108 woravrscan.vishalbh.com woravrscan

192.168.45.109 woravrscan.vishalbh.com woravrscan

#----------------------------------woravr-----------------------------#

--/etc/resolve.conf

# Generated by NetworkManager

search vishalbh.com #--- DNS server

nameserver 192.168.45.2 #--- your domain name server

More DNS configuration details :

[root@woravr02 ~]# nslookup woravrscan.vishalbh.com

Server: 192.168.45.2

Address: 192.168.45.2#53

Name: woravrscan.vishalbh.com

Address: 192.168.45.107

Name: woravrscan.vishalbh.com

Address: 192.168.45.108

Name: woravrscan.vishalbh.com

Address: 192.168.45.109

2. Install required RPM's '

yum update -y

yum install xorg-x11-server-Xorg xorg-x11-xauth xorg-x11-apps -y

yum install -y oracle-database-preinstall-19c.x86_64

yum install oracleasm-support

yum install bind* -y

sysctl -p

3. Stop the firewall

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

systemctl stop firewalld

systemctl disable firewalld

systemctl status firewalld

4. Start the NTP sever or time server

systemctl enable chronyd.service

systemctl restart chronyd.service

chronyc -a 'burst 4/4'

chronyc -a makestep

5. User managment

groupadd -g 54341 oinstall

groupadd -g 54342 dba

groupadd -g 54343 oper

groupadd -g 54344 backupdba

groupadd -g 54345 dgdba

groupadd -g 54346 kmdba

groupadd -g 54347 asmdba

groupadd -g 54348 asmoper

groupadd -g 54349 asmadmin

groupadd -g 54350 racdba

usermod -g oinstall -G dba,oper,asmdba,asmadmin,asmoper,backupdba,dgdba,kmdba,racdba oracle

useradd -m -u 54342 -g oinstall -G dba,asmadmin,asmdba,asmoper -d /home/grid -s /bin/bash grid

echo "grid" | passwd --stdin grid

useradd -m -u 54341 -g oinstall -G dba,oper,backupdba,dgdba,kmdba,asmdba,asmadmin,racdba -d /home/oracle -s /bin/bash oracle

echo "oracle" | passwd --stdin oracle

6. Directory Stucture

mkdir -p /u01/app/oracle/19c/grid

mkdir -p /u01/app/oracle/oracle

mkdir -p /u01/app/oracle/19c/db

chmod -R 775 /u01

chown -R oracle:oinstall /u02/app/oracle

chown -R grid:oinstall /u02/app/oracle/19c/grid

7. Shared Disks for Oracleasm disks

1. Udev method (mostly Own by system team)

2. Oracleasm (multipath)

# rpm -qa | grep oracleasm

oracleasm-support-2.1.11-2.el7.x86_64

make Disk partion using fdisk comands (output like below):

[root@woravr02 ~]# fdisk -l | egrep 'Disk /dev/sd|dev'

Disk /dev/sdb: 209.7 GB, 209715200000 bytes, 409600000 sectors

/dev/sdb1 2048 409599999 204798976 83 Linux

Disk /dev/sdc: 157.3 GB, 157303177216 bytes, 307232768 sectors

/dev/sdc1 2048 307232767 153615360 83 Linux

Disk /dev/sdd: 104.9 GB, 104857600000 bytes, 204800000 sectors

/dev/sdd1 2048 204799999 102398976 83 Linux

Disk /dev/sde: 419.4 GB, 419430400000 bytes, 819200000 sectors

/dev/sde1 2048 819199999 409598976 83 Linux

[root@woravr02 ~]#

#oracleasm createdisk FRA /dev/sdb1

Writing disk header: done

Instantiating disk: done

#oracleasm createdisk ACFS /dev/sdc1

Writing disk header: done

Instantiating disk: done

#oracleasm createdisk OCR /dev/sdd1

Writing disk header: done

Instantiating disk: done

#oracleasm createdisk DATA /dev/sde1

Writing disk header: done

Instantiating disk: done

#/usr/sbin/oracleasm scandisks #--FROM other nodes

Reloading disk partitions: done

Cleaning any stale ASM disks...

Scanning system for ASM disks...

Instantiating disk "OCR"

Instantiating disk "DATA"

Instantiating disk "ACFS"

Instantiating disk "FRA"

#oracleasm listdisks

ACFS

DATA

FRA

OCR

8. unzip the Software

unzip LINUX.X64_193000_grid_home_2.zip -d /u01/app/oracle/19c/grid

unzip LINUX.X64_193000_db_home.zip -d /u01/app/oracle/19c/grid

9. User Equivalence

./sshUserSetup.sh -user grid -hosts "woravr01 woravr02 woravr03" -noPromptPassphrase -confirm -advanced

10. runcluvfy

./runcluvfy.sh stage -pre crsinst -n woravr01,woravr02,woravr03 -orainv oinstall -osdba asmdba -verbose

11. ./gridSetup.sh

More Details

12 . RunFixup if required .

13. orainstRoot

/u01/app/oraInventory/orainstRoot.sh ( each node )

14. root.sh

/u01/app/oracle/19c/grid/root.sh (each node take care exicution sequence)

15 . Check the Cluster status after Each root.sh script

./crsctl check cluster -all

**************************************************************

woravr02:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

woravr02:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

woravr03:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

[root@woravr02 grid_1]# crsstat

Cluster woravrdc

-------------------------------------------------------------------------------------------

cvu | cvu | Online | - | - |

vip | scan1 | - | - | Online |

vip | scan2 | Online | - | - |

vip | scan3 | - | Online | - |

vip | woravr01 | Online | - | - |

vip | woravr02 | - | Online | - |

vip | woravr03 | - | - | Online |

-------------------------------------------------------------------------------------------

x : Resource is disabled

: Has been restarted less than 24 hours ago

----------------------------------------------------------------------------------------------------------

LISTENER_SCAN1 | TCP:1521 | - | - | Online | SCAN |

LISTENER_SCAN2 | TCP:1521 | Online | - | - | SCAN |

LISTENER_SCAN3 | TCP:1521 | - | Online | - | SCAN |

-----------------------------------------------------------------------------------------------------------

: Has been restarted less than 24 hours ago

16. Invoke ASMCA to create diskgroup

/u01/app/oracle/19c/grid/bin/./asmca

17. Intsall RDBMS Software

./runInstaller.sh

18. root.sh

/u01/app/oracle/19c/db/root.sh

19. Run DBCA

Create the database

20. Create service for database : optional

srvctl modify service -db woravr -service woravrsvc -i woravr01,woravr02 -t woravr03 -tafpolicy BASIC -failback YES

21. crsstat

Cluster woravrdc

-------------------------------------------------------------------------------------------

cvu | cvu | Online | - | - |

vip | scan1 | - | - | Online |

vip | scan2 | Online | - | - |

vip | scan3 | - | Online | - |

vip | woravr01 | Online | - | - |

vip | woravr02 | - | Online | - |

vip | woravr03 | - | - | Online |

-------------------------------------------------------------------------------------------

x : Resource is disabled

: Has been restarted less than 24 hours ago

----------------------------------------------------------------------------------------------------------

LISTENER_SCAN1 | TCP:1521 | - | - | Online | SCAN |

LISTENER_SCAN2 | TCP:1521 | Online | - | - | SCAN |

LISTENER_SCAN3 | TCP:1521 | - | Online | - | SCAN |

-----------------------------------------------------------------------------------------------------------

: Has been restarted less than 24 hours ago

-----------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------

: Has been restarted less than 24 hours ago

--------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------

ORACLE_HOME references listed in the Version column

1 : /u01/app/oracle/19c/db_1 oracle oinstall

: Has been restarted less than 24 hours ago

22. Add Entry on /etc/oratab

vi /etc/oratab

23. Create Env Profiles .

[oracle@woravr02 ~]$ cat .bash_profile

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

# User specific environment and startup programs

PATH=$PATH:$HOME/.local/bin:$HOME/bin

export PATH

export TMP=/tmp

export TMPDIR=$TMP

export ORACLE_BASE=/u01/app/oracle_base

export GRID_HOME=/u01/app/oracle/19c/grid_1

export DB_HOME=/u01/app/oracle/19c/db_1

export ORACLE_HOME=$DB_HOME

export ORACLE_SID=woravr2

export ORACLE_TERM=xterm

export BASE_PATH=/usr/sbin:$PATH

export PATH=$ORACLE_HOME/bin:$GRID_HOME/bin:$BASE_PATH

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib

export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

alias grid='. /home/oracle/grid.env'

alias db='. /home/oracle/db.env'

alias l='ls -altrsh'

#-------------------------------------------------------

[oracle@woravr02 ~]$ cat db.env

export ORACLE_SID=woravr2

export ORACLE_HOME=$DB_HOME

export PATH=$ORACLE_HOME/bin:$BASE_PATH

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib

export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

[oracle@woravr02 ~]$ cat grid.env

export ORACLE_SID=+ASM2

export ORACLE_HOME=$GRID_HOME

export PATH=$ORACLE_HOME/bin:$BASE_PATH

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib

export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

[oracle@woravr02 ~]$

24 . Patch the database

below command from root user :

1. Export Grid Env

$ORACLE_HOME/OPatch/./opatchauto apply /tmp/31326369/31305339 -oh $ORACLE_HOME

2. Export the RDBMS Env

$ORACLE_HOME/OPatch/./opatchauto apply /tmp/31326369/31305339 -oh $ORACLE_HOME

repeat same steps for all nodes

open all pluggable databases and exicute the below command after patch

$ORACLE_HOME/OPatch/./datapatch -verbose

[oracle@woravr02 ~]$ db

[oracle@woravr02 ~]$ $ORACLE_HOME/OPatch/./opatch lsinventory | grep applied

Patch 31305087 : applied on Sun Aug 09 20:07:12 IST 2020

Patch 31281355 : applied on Sun Aug 09 20:02:59 IST 2020

[oracle@woravr02 ~]$ grid

[oracle@woravr02 ~]$ $ORACLE_HOME/OPatch/./opatch lsinventory | grep applied

Patch 31335188 : applied on Sun Aug 09 19:47:39 IST 2020

Patch 31305087 : applied on Sun Aug 09 19:46:48 IST 2020

Patch 31304218 : applied on Sun Aug 09 19:45:41 IST 2020

Patch 31281355 : applied on Sun Aug 09 19:42:14 IST 2020

[oracle@woravr02 ~]$