RAC-- Listeners in Oracle 11GR2 RAC, IPC,oraagent --Exadata

I decided write this post regarding the configuration and working principle of database listeners in a Oracle 11gR2 RAC environment..

First of all; in 11gR2 RAC environment, we have scan listeners.

As it can be clearly seen from the picture above, scan listeners are the first point of contact in a 11gR2 RAC environment. Clients are connect to the scan listeners in the first place. There are multiple scan listeners listening multiple scan ip address in the cluster. All the database services are registered with the Scan listeners.
Client send its connection request to a scan listener.
Note that: Client only knows the scan name. DNS redirects the connection request to one of the scan listeners in the cluster. Once the scan listener takes the client's connection request, it sends back one of the local listener's (least crowded node) address to the client.. The local listener information belongs to the local listener which services for the service name that the client requested.. Then the client and local listener communicate between eachother and the connection becomes established.
To understand better, you can take a look to my previous post regarding to the scan listeners http://ermanarslan.blogspot.com.tr/2013/05/database-about-scan-11gr2.html

The following picture represents the interprocess communication between Listeners and Oracle Instance in a RAC environment.

So, in a 2-node RAC environment (like an Exadata Quarter Machine) we have 3 scan listeners and 2 local listener. Scan listeners register the database services via IPC/TCP through pmon and Local listeners registers both the database and ASM instances via IPC/TCP through pmon.

With the 11gR2 environment, both remote and local listeners are started from the Grid Home and when we look to the listener processes, we see they use the same binaries.
ps -ef |grep inherit
 /u01/app/11.2.0.2/grid/bin/tnslsnr LISTENER -inherit
 /u01/app/11.2.0.2/grid/bin/tnslsnr LISTENER_SCAN3 -inherit
 /u01/app/11.2.0.2/grid/bin/tnslsnr LISTENER_SCAN2 -inherit

Furthermore; we have a "oraagent" process to manage these listeners..  Actually, oraagent process starts a bunch of grid components such as  Clustered ASM instance, ons, eONS, scan listener and local listener.

ps -ef |grep oraagent
oracle    8634     1  0  2013 ?        01:54:19 /u01/app/11.2.0.2/grid/bin/oraagent.bin
oracle   19967     1  0 Feb13 ?        00:26:07 /u01/app/11.2.0.2/grid/bin/oraagent.bin

As you see above, we have 2 oraagent.bin processes.
Now, I put my Linux admin hat, and look to the environments of them;
both of them were started from GRID_HOME , the binary used was /u01/app/11.2.0.2/grid/bin/oraagent.bin.. The environments are a little diffent , the file descriptors of these 2 processes are a little different, too. So these should make them different, as it s known that these two processes are used for starting different components of the cluster environments..
For example, while the first oraagent.bin starts the components such as Asm and listeners,  the second oraagent.bin starts the databases and their associated services in the cluster..

Okay lets come back to your topic..
So, when we look at the listener.ora file in Grid Home, we see a different kind of listener specification, as there are no ports or sids or even a hostname declared there. We also see the comments like "line added by Agent"

LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))) # line added by Agent
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))) # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent
ADMIN_RESTRICTIONS_ADMIN=ON

This Agent that adds these lines is the oraagent as expected.. These lines proves that those listeners are started by oraagent.
The lines with ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER*=ON are also important.
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER parameter when set to ON, makes oraagent to dynamically activate the listening point for the running instances..
Actually when we start listener, listener starts and listens on IPC. Oraagent uses this IPC listening point to connect to the listener and start the TCP connection point .. Following is an example for this;

lsnrctl start LISTENER_SCAN3

.....

lsnrctl status LISTENER_SCAN3

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))

STATUS of the LISTENER

------------------------

Alias                     LISTENER_SCAN3

Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production

Start Date                19-FEB-2014 15:17:56

Uptime                    0 days 0 hr. 0 min. 29 sec

Trace Level               off

Security                  ON: Local OS Authentication

SNMP                      OFF

Listener Parameter File   /u01/app/11.2.0.2/grid/network/admin/listener.ora

Listener Log File         /u01/app/oracle/diag/tnslsnr/osrvdb01/listener_scan3/alert/log.xml

Listening Endpoints Summary...

  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN3)))

The listener supports no services

The command completed successfully

As you see when we start Listener_scan3, the only listening protocol that the listener is listening is IPC.

On the other hand; Just after we start the listen, when we look to the oraagent log file, we see that oraagent uses this IPC listening point to activate the TCP listening point of the listener.

2014-02-19 15:31:21.288: [ora.LISTENER_SCAN3.lsnr][1358747968] {1:19319:12863} [start] Vip Address used for the Scan Listener is 192.168.0.86

2014-02-19 15:31:21.288: [ora.LISTENER_SCAN3.lsnr][1358747968] {1:19319:12863} [start] Listener obtained IP 192.168.0.86 from the VIP Resource

2014-02-19 15:31:21.392: [ora.LISTENER_SCAN3.lsnr][1358747968] {1:19319:12863} [start] LsnrAgent::regEndpoint Listener endpoint is (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))

2014-02-19 15:31:21.392: [ USRTHRD][1358747968] {1:19319:12863} Thread:RegEndpointThread:LISTENER_SCAN3 start {

2014-02-19 15:31:21.392: [ USRTHRD][1358747968] {1:19319:12863} Thread:RegEndpointThread:LISTENER_SCAN3 start }

2014-02-19 15:31:21.392: [ USRTHRD][1593657664] {1:19319:12863} Thread:RegEndpointThread:LISTENER_SCAN3 Registering Endpoint nsgfei_EndpointInit() (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.86)(PORT=1521))

2014-02-19 15:31:21.393: [ USRTHRD][1593657664] {1:19319:12863} Thread:RegEndpointThread:LISTENER_SCAN3 LsnrAgent, registered endpoint (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.86)(PORT=1521))

2014-02-19 15:31:22.393: [ora.LISTENER_SCAN3.lsnr][1358747968] {1:19319:12863} [start] LsnrAgent::start }

2014-02-19 15:31:22.393: [ora.LISTENER_SCAN3.lsnr][1358747968] {1:19319:12863} [start] (:CLSN00107:) clsn_agent::start }

2014-02-19 15:31:22.393: [    AGFW][1358747968] {1:19319:12863} Command: start for resource: ora.LISTENER_SCAN3.lsnr 1 1 completed with status: SUCCESS

So far so good; but how can oraagent obtain the needed IP and port information to start this listener?
To answer the question; lets make a demo,
When we use srvctl to add a new listener. we update the cluster inventory actually.

For demonstration,

This is my current listener.ora in grid home;

LISTENER_PROD=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_PROD)))) # line added by Agent
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))) # line added by Agent
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))) # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_PROD=ON # line added by Agent
Lets add a listener using srvctl;
I add listener named listener_erman to start from grid home and listen on port 1555;

srvctl add listener -l listener_erman -o /u01/app/11.2.0.2/grid -p 1555

okay listener_erman is added. So lets look to listener.ora again;

LISTENER_PROD=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_PROD)))) # line added by Agent
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))) # line added by Agent
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))) # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_PROD=ON # line added by Agent

So by looking above we can say that nothing is changed.. There is not even an IPC listening point description for listener_erman there..

Now I start listener_erman; (note that I dont have any description about this listener in listener.ora)

 srvctl start listener -l listener_erman

srvctl status listener -l listener_erman

Listener LISTENER_ERMAN is enabled

Listener LISTENER_ERMAN is running on node(s): osrvdb01,osrvdb02

ps -ef |grep LISTENER_ERMAN

oracle   30687     1  0 09:18 ?        00:00:00 /u01/app/11.2.0.2/grid/bin/tnslsnr LISTENER_ERMAN -inheri

[oracle@osrvdb01 ~]$ lsnrctl status LISTENER_ERMAN

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 21-FEB-2014 09:46:02

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_ERMAN)))

STATUS of the LISTENER

------------------------

Alias                     LISTENER_ERMAN

Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production

Start Date                21-FEB-2014 09:18:47

Uptime                    0 days 0 hr. 27 min. 15 sec

Trace Level               off

Security                  ON: Local OS Authentication

SNMP                      OFF

Listener Parameter File   /u01/app/11.2.0.2/grid/network/admin/listener.ora

Listener Log File         /u01/app/11.2.0.2/grid/log/diag/tnslsnr/osrvdb01/listener_erman/alert/log.xml

Listening Endpoints Summary...

  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_ERMAN)))

  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=some ip)(PORT=1555)))

  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=some vip )(PORT=1555)))

We see that listener_erman is started , and it is listening on port 1555.

So lets look to the listener.ora again;

LISTENER_ERMAN=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_ERMAN)))) # line added by Agent
LISTENER_PROD=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_PROD)))) # line added by Agent
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))) # line added by Agent
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))) # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_PROD=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_ERMAN=ON # line added by Agent

As you see, when we start the listener, oraagent adds the needed listener configuration lines into the listener.ora file in $GRID_HOME... Oraagent also adds a line for the new listener in to the endpoints_listener.ora file.  "Endpoints_listener.ora file is there for backward compatibility with pre-11.2 databases.

endpoints_listener.ora file contents;

LISTENER_ERMAN_DB01=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=db01-vip)(PORT=1555))(ADDRESS=(PROTOCOL=TCP)(HOST=10.10.1.110)(PORT=1555)(IP=FIRST)))) # line added by Agent

Now I delete the lines and restart the listener ...

Looking to the listener.ora file again ;

LISTENER_ERMAN=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_ERMAN)))) # line added by Agent
LISTENER_PROD=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_PROD)))) # line added by Agent
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))) # line added by Agent
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))) # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_PROD=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_ERMAN=ON # line added by Agent

Again, oraagent adds the needed listener configuration lines into the listener.ora file in $GRID_HOME..

So in the listener start process , oraagent looks to the listener.ora, if it cant find a configuration lines for that listener, it goes the cluster repository and looks there to find the needed configuration for that listener.. If it finds the listener configuration in repository, it updates the listener.ora , and make the listener be able to start.. 

Or not :? keep reading..:) 

Following is a proof for the above pharagraph.

Here, 

I first delete the listener_erman

Then add a fake line for configuration of the listener_erman to the listener.ora.

Lastly add the listener_erman back again and try to start the listener_erman using srvctl..

See what happens;

Stopping and removing listener_erman;

srvctl stop listener -l listener_erman

srvctl remove listener -l listener_erman

Adding a fake line for listener_erman to the listener.ora;  ( note that: I dont even change the lines written in  endpoints_listener.ora )

LISTENER_ERMAN=(This is FAKE)
LISTENER_PROD=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_PROD)))) # line added by Agent
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))) # line added by Agent
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))) # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_PROD=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_ERMAN=ON # line added by Agent

Adding listener_erman back again ;

 srvctl add listener -l listener_erman -o /u01/app/11.2.0.2/grid -p 1555

srvctl status listener -l listener_erman

Listener LISTENER_ERMAN is enabled

Listener LISTENER_ERMAN is not running

Starting listener_erman again;

srvctl start listener -l listener_erman

PRCR-1079 : Failed to start resource ora.LISTENER_ERMAN.lsnr

CRS-5016: Process "/u01/app/11.2.0.2/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0.2/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0.2/grid/log/osrvdb01/agent/crsd/oraagent_oracle/oraagent_oracle.log"

CRS-5016: Process "/u01/app/11.2.0.2/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0.2/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0.2/grid/log/osrvdb01/agent/crsd/oraagent_oracle/oraagent_oracle.log"

CRS-2674: Start of 'ora.LISTENER_ERMAN.lsnr' on 'osrvdb01' failed

:) lets look to the oraagent_oracle.log for details;

2014-02-21 10:01:29.284: [ora.LISTENER_ERMAN.lsnr][1660774720] {1:19319:17079} [start] TNS-01150: The address of the specified listener name is incorrect

So, as you see , oraagent tries to start LISTENER_ERMAN.. It looks to the listener.ora file and if it finds an entry starts with the listener_erman.. Later, it accepts that entry as true and doesn ot go to the repository at all.. That's why it can not start the listener, as we add a fake line for listener_erman. 

This also means, oraagent does not update the listener.ora file if it sees a line in listener.ora file which starts with the listener name it is requested to start..

Lets clear this mess; lets remove that listener and take everyting back again..

srvctl remove listener -l listener_erman

And try to start the listener_erman back again ( as listener.ora the fake line are still present in listener.ora)

srvctl start listener -l listener_erman

PRCR-1001 : Resource ora.LISTENER_ERMAN.lsnr does not exist

You see there is no TNS error this time..

So the truth is ->When we issue srvctl start listener command; Oracle looks to the repository and see there is no resource named listener_erman, this means It does not read listener.ora in the first place.. 

To prove that , I start the listener using lsnrctl and get the expected TNS error.

lsnrctl start LISTENER_ERMAN

TNS-01150: The address of the specified listener name is incorrect..

In summary, when we add a listener using  srvctl; listener defition is added to the cluster repository . When we use srvctl to start the listener, oraagent looks to the repository to see the listener defition for that listener, if there is a correct listener definition in repository-> it updates the listener.ora and starts the listener.

if there isnt any listener definition in repository -> it just cannot start the listener.

if  there is a correct listener definition in repository and(&&) if there is a false listener definition -> it cannot start the listener but this time produces tns error or listener specific error, but at least it tries..

Alternatively, you can always write a manuel declaration in to the GRID_HOME/listener.ora file.

For listener_erman , I can write a manuel declaration and start the listener using lsnrctl command as follows;

LISTENER_ERMAN=(DESCRIPTION_LIST=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=db01-vip.erman.net)(PORT=1555))))))

lsnrctl status LISTENER_ERMAN

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=db01-vip.karbosan.net)(PORT=1555)))

STATUS of the LISTENER

------------------------

Alias                     LISTENER_ERMAN

Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production

Start Date                21-FEB-2014 13:40:44

Uptime                    0 days 0 hr. 0 min. 11 sec

Trace Level               off

Security                  ON: Local OS Authentication

SNMP                      OFF

Listener Parameter File   /u01/app/11.2.0.2/grid/network/admin/listener.ora

Listener Log File         /u01/app/oracle/diag/tnslsnr/osrvdb01/listener_erman/alert/log.xml

Listening Endpoints Summary...

  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.82)(PORT=1555)))

As seen above, listener start successfully, but this I cant use srvctl to manage this listener as expected.. oraagent also does not control this listener any more..

srvctl status listener -l listener_erman

PRCR-1001 : Resource ora.LISTENER_ERMAN.lsnr does not exist

Also the listener listens only from a single ip address.. 

Then, I change my setting in listener.ora to be the following; 

LISTENER_ERMAN=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_ERMAN))))

Then I use lsnrctl to start the listener..
This time I see that listener_erman only listens for IPC..
To make listener_erman to listen multiple hostnames with same port numbers,I add following line to the endpoints_listener.ora file

LISTENER_ERMAN=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=db01-vip.karbosan.net)(PORT=1555))(ADDRESS=(PROTOCOL=TCP)(HOST=db01.karbosan.net)(PORT=1555))))

So I connect to the listener using lsnrctl , which connects to the listener_erman using open IPC connection point and set enable_global_dynamic_endpoint.

[oracle@osrvdb01 ~]$ lsnrctl

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 21-FEB-2014 14:20:31

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Welcome to LSNRCTL, type "help" for information.

LSNRCTL> set current_listener listener_erman

Current Listener is listener_erman

            LSNRCTL> set enable_global_dynamic_endpoint on;

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_ERMAN)))

listener_erman parameter "enable_global_dynamic_endpoint" set to ON

The command completed successfully

LSNRCTL> services

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_ERMAN)))

The listener supports no services

The command completed successfully

Still no chance.... 

Bytheway , dont bother the above lsnrctl commands . I wrote them just for fun and found them cool. Actually they are not related with this endpoint thing..:)

Okay , here IP=FIRST comes to play.

.

The (IP=FIRST) statement will make the listener create a listening endpoint on the IP address to which the given HOST resolves. By default, without (IP=FIRST), the listener will listen on all network interfaces (e.g. INADDR_ANY) .

So to able to listen to the same port using different hostnames on a machine, listener configuration should contain IP=FIRST statement..

I add following listener description to listener.ora.

LISTENER_ERMAN=(DESCRIPTION_LIST=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=db01-vip.erman.net)(PORT=1555)(IP=FIRST))))(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=db01.erman.net)(PORT=1555)(IP=FIRST)))))

This time, listener can start successfully listening for two different hostnames with the same port numbers..

[oracle@db01 admin]$ lsnrctl start LISTENER_ERMAN

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 21-FEB-2014 15:22:42

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Starting /u01/app/11.2.0.2/grid/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 11.2.0.3.0 - Production

System parameter file is /u01/app/11.2.0.2/grid/network/admin/listener.ora

Log messages written to /u01/app/oracle/diag/tnslsnr/db01/listener_erman/alert/log.xml

Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.82)(PORT=1555)))

Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.10.100)(PORT=1555)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=db01-vip.karbosan.net)(PORT=1555)(IP=FIRST)))

STATUS of the LISTENER

------------------------

Alias                     LISTENER_ERMAN

Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production

Start Date                21-FEB-2014 15:22:42

Uptime                    0 days 0 hr. 0 min. 0 sec

Trace Level               off

Security                  ON: Local OS Authentication

SNMP                      OFF

Listener Parameter File   /u01/app/11.2.0.2/grid/network/admin/listener.ora

Listener Log File         /u01/app/oracle/diag/tnslsnr/db01/listener_erman/alert/log.xml

Listening Endpoints Summary...

  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.82)(PORT=1555)))

  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.10.100)(PORT=1555)))

As you see this is how it s done manually .. 

Do we need to make our listener to listen from the Administration network? No :) I wrote the things above just to show you some advanced listener operations..  

In conclusion,

In a 11gR2 Rac environment, srvctl should be used to manage the listeners.. Adding, removing, starting and stopping the listeners should be made by using srvctl commands. By using srvctl management is easier.. Oraagent takes the control, checkes our resources with this. 

Using $GRID_HOME for listener home is recommended, by using $GRID_HOME for all our listeners, we have a centralized location and environment for all of our listener processes. This also simplifies the management activities.. 

On the other hand, if you want to use manual technics for listener declaration and management in a RAC envrionment, it is possible ,as well. 

The examples you have read above explain the listener endpoints and manual declarations, clearly..

I hope you will find this helpful.