Oracle Enterprise Manager: Agent commands

Here is the list of Oracle Enterprise Manager (OEM) Agent basic commands -

1. Start EM agent - emctl start agent

2. Get EM agent status - emctl status agent

3. Stop EM agent - emctl stop agent

4. Find the timeout value of the EM agent - emctl status agent [TIMEOUT]

5. Upload of metrics from agent to the EM server - emctl upload agent

6. Display the status of the connection pools - emctl status agent dbconnections

7. Print the version of the EM agent - emctl getversion

Weblogic: Datasource error - Maximum Capacity of Pool

Error Details -

Error An error occurred during activation of changes, please see the log for details.
Error prepareUpdate failed for JDBC Module DSName: Cannot set Maximum Capacity of Pool DSName to (10), this value is lower than the configured Minimum Capacity of the pool (20).
Error Cannot set Maximum Capacity of Pool DSName to (10), this value is lower than the configured Minimum Capacity of the pool (20).

Occurs when -

While trying to activate the datasource connection pool changes (update of Minimum capacity and Maximum capacity), the above error is displayed.

Solution -

1. First make the changes to the Minimum capacity of the connection pool
2. Save and activate the changes
3. Change the Maximum capacity of the connection pool
4. Save and activate the changes
5. Restart the managed server for the changes to take effect

BEA-090403: Authentication for user denied

Error message from weblogic - 

<DATE> <Critical> <Security> <BEA-090403> <Authentication for user  denied>

<DATE> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication for user  denied

weblogic.security.SecurityInitializationException: Authentication for user  denied

        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:965)

        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1050)

        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)

        at weblogic.security.SecurityService.start(SecurityService.java:141)

        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

        Truncated. see log file for complete stacktrace

Caused By: javax.security.auth.login.FailedLoginException: [Security:090304]Authentication Failed: User  javax.security.auth.login.LoginException: [Security:090301]Password Not Supplied

        at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:261)

        at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)

        at java.security.AccessController.doPrivileged(Native Method)

        at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        Truncated. see log file for complete stacktrace

> 

<DATE> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>

Solution - 

1. validate the boot.properties files if the username and password have been set

2. If not, update the boot.properties files with the username, password of the weblogic server and restart the managed server. Username/password can be provided in cleartext which will be encrypted by weblogic server

3. If the boot.properties file does not exist, please create a new file.

OSB: Debug logs and troubleshooting

OSB troubleshooting and debugging could be more easier after enabling the below settings

Debug file settings -

alsbdebug.xml can be found in the domain home path. 

Set module debug to true in alsbdebug.xml -

<java:alsb-module-debug>true</java:alsb-module-debug>

Restart the managed server for the change to take effect

BEA-141196 The managed server could not update the configuration files

Error -

<Error> <Management> <BEA-141196> <The managed server could not update the configuration files during the registration with the deployment service. The update failed due to an exception:

weblogic.management.DeploymentException: [Deployer:149233]An unexpected error was encountered during the deployment process.

Solution –

1.       Config directory in the managed server could have got corrupted.

2.       Take a backup of the existing config directory of the server - <domain>/config

3.       Copy the config folder from Admin server to the Managed server.

4.       Restart the Managed server

BEA-001129: Received exception while creating connection for pool

While starting the Weblogic Admin / Managed servers, we get the Connection pool error 

Error Message - 

<Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "conpoolname": ORA-01034: ORACLE not available

ORA-27101: shared memory realm does not exist

<Error> <JDBC> <BEA-001112> <Test "SELECT 1 FROM DUAL" set up for pool "Conpoolname" failed with exception: "oracle.jdbc.xa.OracleXAException".>

<Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "Conpoolname": ORA-01034: ORACLE not available

ORA-27101: shared memory realm does not exist

Solution - 

The issue mostly is that the Datasource is not able to connect to the Oracle Database.

1. Login to the Admin server, navigate to Services -> Datasources

2. Go to Monitoring tab -> and do a Test Datasource.

3. If the issue still persists, go to the putty session, navigate to the <domain path>, and select 'config', 'jdbc' folders. check the jdbc.xml if the datasource values are correctly entered.

 4. After validation, restart the admin/managed server

Weblogic: Log files in Weblogic server

Log files in Weblogic server

Linux Tips: backup fstab

When copying or moving a fstab file, it is displays a permission denied error.

So use the following mechanism to backup or copy the fstab file by providing the destination file name.

Syntax - cp etc/fstab fstab

Linux Tips: backup crontab

How to save contents of a crontab into a text file as backup copy

Execute the command -

crontab -l > crontab.txt

OEM12c: Oracle Enterprise Manager - Creating a blackout

This post talks about creating a Blackout in Oracle Enterprise Manager.

We setup Enterprise Manager to monitor, notify any issues with the Database, Middleware and application stack. Enterprise manager sends alerts if it finds any issues with the systems or any thresholds being exceeded their limits.

If you need to perform a controlled activity like a restart or patch apply, then you may need to bring down the server or database. While you bring down, the EM will start bombarding with Alerts and this might cause unnecessary emails to be sent, and the important alerts might get missed.

Enterprise Manager provides an option to create a Blackout specifying the time and the targets during which no alert will be sent to the recipient

Steps to create a Blackout

1. Go to  Enterprise -> Monitoring -> Blackouts
    Click “Create” or if there is a previous one created that you want to use as template click “Create Like”
2. Provide 'Blackout Name'  (Eg. Restart OSB server). Comments and Reason are not required fields.
     Add targets (targets could be Database, Middleware, Cluster, Host servers, BI, Security suite, and other Oracle Products). Click Next button
3. Validate the Member Targets which are dependant on the 'Target' selected in the previous page.
4. Provide the schedule for the Blackout.
Blackout can be started immediately and provided with a duration.
Blackout can also be scheduled to start and end at certain times.
5. If the Blackout needs to be repeated weekly/daily, the Repeating drop down has to be selected with the appropriate option

BEA-000365 - Weblogic Admin / Managed server startup issues

Admin / Managed server startup Issues –

a. LDAP Files lok

<Error> <EmbeddedLDAP> <BEA-171519> <Could not obtain an exclusive lock to the embedded LDAP data files directory: /<domainhomepath>/<domainname>/servers/AdminServer/data/ldap/ldapfiles because another WebLogic Server is already using this directory. Ensure that the first WebLogic Server is completely shutdown and restart the server.>

<Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>

<Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>

<Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

Suggestion  –

1.       Navigate to <domainhomepath>/<domainname>/servers/Adminserver/data/ldap/Idapfiles

2.       Rename the file “EmbeddedLDAP.lok” to “EmbeddedLDAP_bkp.bkp_lok”

3.       Restart the Admin server

b. Admin Server lok

weblogic.management.ManagementException: Unable to obtain lock on /<domainhomepath>/<domainname>/servers/AdminServer/tmp/AdminServer.lok. Server may already be running at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:206)

 <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>

 <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>

 <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

Suggestion  –

1.       Navigate to <domainhomepath>/<domainname>/servers/Adminserver/tmp

2.       Rename the file “AdminServer.lok” to “AdminServer_bkp.bkp_lok”

3.       Restart the Admin server

c. WLS_ADMINSERVER DAT

<Emergency> <Store> <BEA-280060> <The persistent store "_WLS_AdminServer" encountered a fatal  error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_AdminServer" cannot open file _WLS_ADMINSERVER000000.DAT.

The persistent file store "_WLS_AdminServer" cannot open file _WLS_ADMINSERVER000000.DAT.

weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_AdminServer" cannot open file _WLS_ADMINSERVER000000.DAT.

Suggestion  –

1.       Navigate to <domainhomepath>/<domainname>/servers/Adminserver/data/store/default

2.       Rename the file “WLS_ADMINSERVER000000.DAT” to “WLS_ADMINSERVER000000_bkp.bDAT”

3.       Restart the Admin server

d. WLS_ADMINSERVER DAT

weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.

<Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>

Suggestion  –

1.       Navigate to <domainhomepath>/<domainname>/servers/Adminserver/data/store/default

2.       Rename the file “WLS_DIAGNOSTICS.DAT” to “WLS_DIAGNOSTICS_bkp.bDAT”

3.       Restart the Admin server

Weblogic: Log Rotation in weblogic server

Log Rotation can be setup for Admin and Managed servers in a Oracle Weblogic server.

Below log files can be setup for log rotation

a. Server log
b. Access log
c. Domain log

Domain log rotation -

1. Login to the Admin console of the weblogic server
2. In the Domain structure, click on the Domain name.
3. Navigate to the 'Logging' tab and edit the Logging properties. Restart the Admin and Managed servers

Server and Access Log rotation -
1. Login to the Admin console of the weblogic server
2. Select 'Environments', click on 'Servers' and select the Admin or Managed server for which the logs needs to be rotated
3. Navigate to the 'Logging' tab, select General for Server log, and HTTP for access log and edit the Logging properties. Restart the Admin and Managed servers as appropriate.

File Rotation Steps -

1. Provide the 'Rotation Type' - By Time or By Date
2. If you want to setup 'RotationType' as 'By Time' then fill the following fields
             a. Begin Rotation Time - Eg. 23:45 (provide in 24 hrs)
             b. Rotation Interval - 24 (every 24 hrs or every 4 hrs)
             c. Limit number of retained files - 5 (if you need the old files to be removed from the system) - you can check this option, if you do not want to delete the old files
             d. Log file Rotation directory - Provide the absolute path of the rotation directory
3. If you want to setup 'Rotation Type' as 'By Date' then fill the following fields
              a. Rotation file size - 5000 (in kilobytes)
              b. Limit number of retained files - 5 (if you need the old files to be removed from the system) - you can check this option, if you do not want to delete the old files
             c. Log file Rotation directory - Provide the absolute path of the rotation directory

Linux Tips: Frequently used Unix commands

Some Unix commands and their usage which are being frequently used are listed in this post

df -h

Lists the free space data for the mount points in a human readable format.

Eg. 50G is 50 Gigabytes
      2T is 2 Terabytes

df -k

Lists the free space data in Kilo bytes format

To understand more about the df command, use df --help.