Enterprise Manager OMS high cpu数据收集

Q1. Was there any resource intensive activities were scheduled at that time ?
Q2. What is the frequency of issue i.e. CPU /mem spike ? Was this spike rauns for very long hours (more than 10 min or so) ?

Q3. What the patch details atside ?
$$ORACLE_HOME/OPatch/opatch lsinventory -detail

Q4. From the given screens/report ,,not sure which PID (process ID) is consuming more resource ? if your admin captured , details about the PIDs etc , please share with us .

Please share the following logs for the timestamp when issue was seen:

a) gc_inst/em/EMGC_OMS1/sysman/log/emoms.log, emoms_pbs.trc, emoms_pbs.log ,.emctl.msg file
b) gc_inst/userprojects/Domain/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out.*
c) Have you captured any system dump or java thread dump during the CPU or memory spike ? if please share the same
d) Get me SYstem log which has captured the TOP resource (CPU & memory) Consumer PID details

Best Regards
Basu

What to collect when high CPU is seen by a java pid ?
Further if you notice any CPU spike or high CPU BY 12c agent /OMS , gather these details

1) OS command , PID details
$topas

2) If Java PID shown in top CPU consumer list , get thread dump using

$<OMS_HOME>/bin/emctl dump oms
$kill -3 <java pid> > /tmp/threadpid.out — generated threaddump ofpid

 

【12c Cloud Control】EM Agent可能消耗大量内存

em process consuming large memory suspected 中建议在AIX平台上监控多个目标时设置LDR_CNTRL和 AIX_THREADSCOPE 这2个参数。

 

 

实际在12c的官方文档Oracle® Enterprise Manager Cloud Control Administrator’s Guide 12c Release 2 (12.1.0.2)中也提到了http://docs.oracle.com/cd/E24628_01/doc.121/e24473/emctl.htm#CBHCDGCD

 

 

On IBM AIX environment with a large memory configuration where the Management Agent is monitoring a large number of targets, the Agent may not start. To prevent this issue, prior to starting the Management Agent, add the following parameters to the common environment file:

 

 

LDR_CNTRL="MAXDATA=0x80000000"@NOKRTL

AIX_THREADSCOPE=S

 
The LDR_CNTRL variable sets the data segment size and disables loading of run time libraries in kernel space. The AIX_THREADSCOPE parameter changes AIX Threadscope context from the default Processwide ‘P’ to Systemwide ‘S’. This causes less mutex contention.

 

可见这2个环境变量是在12c上推荐设置的。

 

推荐 follow Doc 1332522.1中给agent设置内存使用上限的方法, 且该方法不会影响RDBMS数据库的正常使用。


1 & 2)
emwd.pl & emdprocstats.pl are part of scheduled collection. They will be executed as per the schedule mentioned in your agent.

 

3) Yes. It is recommended to make the changes as per this note. In many occassions, this settings have solved the issue.
They are kernel settings which is suitable and recommended for Agent in AIX. So, it ususally don’t affect other applications or system.”

 

 

 

Grid Control OMS Agent代理工作原理图

我们在使用Grid Control集中化管理OS、Oracle数据库时要求在host上安装Agent代理程序,以便Agent定期收集OS、Oracle信息传输给Oracle Grid Control Management Server(OMS),并执行OMS下达的一系列指令。

大多数人对于Agent的了解仅限于如何安装和启动agent,下图展示了OMS Agent的架构:

 

 

Agent主要由2个组件(component)部分组成,分别是Collector 收集器和 Metric Engine 度量引擎。

 

Collector收集器是agent的重要子系统。它负责收集并上传metric data度量数据到OMS(oms最终将这些数据存入数据库中)。Collector 利用collection file中的信息判定针对哪些target目标需要收集metric data以及多久收集一次。 为了获取数据,Collector将查询投递给Metric Engine,而Metric Engine负责实际的metric data的收集。 Metric Engine  通过Fetchlets 、Metadata原信息文件(Metadata files defined in OH/sysman/admin/metadata)和 已发现的target 信息文件(Targets defined in OH/sysman/emd/targets.xml)来获得每一个目标的metrics监控信息。 同时 metadata原信息文件也提供了实际如何去计算metrics度量的算法。

 

基于以上这些信息,Metric Engine 将使用恰当的fetchlets从监控目标获取数据, 这里的 Fetchlets指的是指定数据的访问方式, 例如访问数据库性能数据会采用SQL Fetchlets,而访问OS数据则使用OS Fetchlets。

 

一旦Collector 收集到metric data,它会将这些度量数据和已定义的阀值做对比,检查是否发送警告(alert waring), 同时将这些度量信息保存到本地文件系统上($OH/sysman/emd/upload目录)。 这些文件最后通过http 或 https 协议 传送到OMS服务器的指定URL上,该URL被$OH/sysman/config/emd.properties 配置文件中的REPOSITORY_URL指定,如以下例子:

 

 

[root@nas ~]# cat /w01/wls/agent/core/12.1.0.1.0/stage/sysman/config/emd.properties
#
#   emd Root directory(read-only location). Metrics should not create files
#   under this directory
#
#
emdRoot=/w01/wls/agent/core/12.1.0.1.0

#
#   agent Root directory(writeable).s
#   Use this property to base any temporary file creation.
#
#
agentStateDir=%EMSTATE%

#  perl executable directory  
#
perlBin=/w01/wls/agent/core/12.1.0.1.0/perl/bin

#
# script directory
#
scriptsDir=/w01/wls/agent/core/12.1.0.1.0/sysman/admin/scripts

#
# stage directory for provisioning
#
emStageDir=/tmp

#
#  EMD main servlet URL
#
EMD_URL=http://nas:%EM_SERVLET_PORT%/emd/main/

#
#  OMS Upload URL
#
#  if there is no receiving OMS or if you wish to disable the UploadManager
#  please set this value to empty or comment out below line
#
REPOSITORY_URL=https://:4900/empbs/upload/

#
#The following properties are advanced read-only properties
#

#
# The location of the file that contains the root certificate.
#
emdRootCertLoc=/w01/wls/agent/core/12.1.0.1.0/sysman/config/b64LocalCertificate.txt
internetCertLoc=/w01/wls/agent/core/12.1.0.1.0/sysman/config/b64InternetCertificate.txt

#
# The download URL for the EMD Oracle Wallet and its local file location.
#
# Note: Ensure that this URL references a valid port number at which the
# console is available on http
#
emdWalletSrcUrl=https://:4900/em/wallets/emd
emdWalletDest=/w01/wls/agent/core/12.1.0.1.0/sysman/config/server

# JAVA HOME required for agent operations
#
JAVA_HOME=/w01/wls/agent/core/12.1.0.1.0/jdk

#
# This string is used by the agent to determine which algorithm to use for encrypted data
# The string value will be same as the release version
#
agentVersion=12.1.0.1.0

#
# To enable the metric browser, uncomment the following line
# This is a reloadable parameter
#
#_enableMetricBrowser=true

#
# These are the optional Java flags for the agent
#
agentJavaDefines=-Xmx128m

#
#   The agent base directory.
#
agentBaseDir=/w01/wls/agent

#
############################################################################
########################### Modifiable Properties ##########################
############################################################################
#

#
#### Tracing related properties
#

#
# emagent perl tracing levels
# supported levels: DEBUG, INFO, WARN, ERROR
# default level is WARN
#
#
EMAGENT_PERL_TRACE_LEVEL=INFO

# logging properties
Logger.log4j.appender.Rolling=org.apache.log4j.RollingFileAppender
Logger.log4j.appender.Rolling.File=%EMSTATE%/sysman/log/gcagent.log
Logger.log4j.appender.Rolling.Append=true
Logger.log4j.appender.Rolling.MaxFileSize=5000000
Logger.log4j.appender.Rolling.MaxBackupIndex=10
Logger.log4j.appender.Rolling.layout=oracle.sysman.gcagent.util.logging.GCPattern
# FOR NOW add a nother log for errors
Logger.log4j.appender.Errors=org.apache.log4j.RollingFileAppender
Logger.log4j.appender.Errors.File=%EMSTATE%/sysman/log/gcagent_errors.log
Logger.log4j.appender.Errors.Append=true
Logger.log4j.appender.Errors.Threshold=ERROR
Logger.log4j.appender.Errors.layout=oracle.sysman.gcagent.util.logging.GCPattern
Logger.log4j.appender.Errors.MaxFileSize=50000000
Logger.log4j.appender.Errors.MaxBackupIndex=3
# Add a test appender for individual tests
Logger.log4j.appender.Test=org.apache.log4j.FileAppender
Logger.log4j.appender.Test.File=/dev/null
Logger.log4j.appender.Test.Append=true
Logger.log4j.appender.Test.Threshold=DEBUG
Logger.log4j.appender.Test.layout=oracle.sysman.gcagent.util.logging.GCPattern

#
# If you increase the maximum file size for the Mdu and Errors logs, you
# should consider setting _maxFileSizeToCopy to a value that is higher then the
# new number (please note that this will potnetially increase the size of your
# incidents)
#

#
# Set root category priority to INFO and its only appender to Rolling.
Logger.log4j.rootCategory=INFO, Rolling, Errors, Test

#
# Enable HTTPListener (jetty) at INFO level.
# TODO: remove this when true trace is supported
Logger.log4j.category.oracle.sysman.gcagent.comm.agent.http.HTTPListener=INFO

Logger.log4j.appender.stdout=org.apache.log4j.ConsoleAppender
Logger.log4j.appender.stdout.layout=oracle.sysman.gcagent.util.logging.GCPattern

# Set the class loaders to level INFO
Logger.log4j.category.oracle.sysman.gcagent.metadata.impl.ChainedClassLoader=INFO
Logger.log4j.category.oracle.sysman.gcagent.metadata.impl.ReverseDelegationClassLoader=INFO
Logger.log4j.category.oracle.sysman.gcagent.metadata.impl.PluginLibraryClassLoader=INFO
Logger.log4j.category.oracle.sysman.gcagent.metadata.impl.PluginClassLoader=INFO

# Add an appender for MetaData Updates
Logger.log4j.appender.Mdu=org.apache.log4j.RollingFileAppender
Logger.log4j.appender.Mdu.File=%EMSTATE%/sysman/log/gcagent_mdu.log
Logger.log4j.appender.Mdu.Append=true
Logger.log4j.appender.Mdu.Threshold=INFO
Logger.log4j.appender.Mdu.layout=org.apache.log4j.PatternLayout
Logger.log4j.appender.Mdu.layout.ConversionPattern=%d [%t] - %m%n
Logger.log4j.appender.Mdu.MaxFileSize=50000000
Logger.log4j.appender.Mdu.MaxBackupIndex=3

Logger.log4j.category.oracle.sysman.gcagent.dispatch.MetadataUpdater=INFO, Mdu
Logger.log4j.additivity.oracle.sysman.gcagent.dispatch.MetadataUpdater=false

# Turn off QA log by default
Logger.log4j.category.QA=FATAL, QA

#Logger._enableTrace=true

#
#### Scalability related properties
#

#List of ora errors which can be ignored and need not be uploaded to repos
IgnoreDownOraErrors=12541,01033,01034,12505,03134,12170,12500,01219,1089,12560,12514,12528,12545

################################
#
# Put all additional properties here
#
################################

# uncomment for ease of debugging
#MaxThreads=1

# Set the server's graceful shutdown delay.
GracefulShutdownDelay=3

# Dump the dispatcher when overloaded
_dumpDispatcherWhenOverloaded=true

# Whether the EMD should listen on all NICs on the current host (the default)
# or just the NIC associated with the hostname in EMD_URL
AgentListenOnAllNICs=true

# Dump each request
#_dumpEveryDispatcherRequest=true

# Dynamic properties timeout for specific target types
dynamicPropsComputeTimeout_rac_database=180
dynamicPropsComputeTimeout_cluster=180
dynamicPropsComputeTimeout_has=180
dynamicPropsComputeTimeout_oracle_database=180
dynamicPropsComputeTimeout_oc4jjvm=180
dynamicPropsComputeTimeout_microsoft_sqlserver_database=180
dynamicPropsComputeTimeout_host=180
dynamicPropsComputeTimeout_osm_instance=180

_disableLoadDPFromCacheNormal=true

#Enable jobsystem streams tracing
_enableJobSystemStreamsTracing=true

# Allow beacon aplication to have 500 megabytes of space. Primarily for ATS collections.
# 500 * 1024 * 1024 = 524288000
applicationMetadataQuota_BEACON=524288000

#Enable auto tuning out of the box
enableAutoTuning=true

 

由Collector最终收集到的这些信息文件仅在满足以下任意条件时实际传送给OMS:

1) 有一条alert告警信息需要发送
2) Collector收集到的信息文件的大小超过一个预定值(默认为20MB 20480KB), 该预定限制值由$OH/sysman/config/emd.properties中UploadFileSize参数指定。
3) 从上一次数据加载算起时间超过30分钟(默认),该预订限制值由$OH/sysman/config/emd.properties中UploadInterval 参数指定。

 

注意与Agent的处理方式不同,由Agent发送给OMS的Alert severities告警信息,OMS会直接将其存入到EM Repository数据库中,而不是以临时文件的形式暂存。

 

Agent除了Metric Engine和Collector 2个主要模块外, 还有其他子系统负责完成不同的工作:

 

  • Target Manager
    • Target Manager holds monitored targets
    • Target data in $EM/sysman/emd/targets.xml
    • lists managed targets, each with name, type, and other properties
    • Credential properties are encrypted
    • Targets can be marked broken
      • Required properties not provided
      • Dynamic properties take too long to compute
    • Discovery of new target instances possible by running perl scripts that list unmonitored instances.
  • Metric Engine
    • Driven by XML target metadata
    • one file per target-type, found in $OH/sysman/admin/metadata/*.xml
    • defines metrics; each may have multiple columns
    • for each metric, defines how data is collected:
      • QueryDescriptor : by fetchlet
      • PushDescriptor: by recvlet
      • ExecutionDescriptor: aggregation from other metrics
    • Supports multiple target versions with ValidIf
    • Defines properties for target type
      • Instance properties: specified in targets.xml
      • Dynamic properties: computed by metric engine
    • Metric Engine holds target-type metadata
      • given a target and a metric name, calls fetchlet manager and/or metric cache and returns a metric result
    • Metric Cache caches last-collected data for use in computing expressions
    • Aggregate metric support allows metrics to be computed via views, joins and group bys over other metrics
      • GetView: select columns or rows from a MetricResult
      • GroupBy: compute aggregation information (SUM, COUNT, MIN, MAX)
      • Union: add rows returned by multiple MetricResults
      • JoinTables: combine multiple metrics’ columns
  • Fetchlet Manager
    • A fetchlet is a data-access mechanism available to compute metric data
      • OS fetchlets : launch an OS process and interpret output
        • OS Fetchlet
        • OSLine Fetchlet
        • OSLineToken
        • UDM : User Defined Metric
      • SQL fetchlet : run a SQL or PL/SQL statement
      • URL fetchlets
        • HTTP data
        • URLTiming Fetchlet
      • and more…
  • Collection Manager
    • Holds all collections, both default and per-target
    • CollectionItem is the basic unit of scheduled collection
    • multiple metrics collected from the same target at the same interval can be collected in the same thread (MetricColl)
    • Once data is collected for a CollectionItem, any Conditions are evaluated
      • three states: Clear, Warning, Critical or Unknown
      • last evaluated Condition states are stored in $EM/sysman/emd/state/*
    • Collection XML files
      • default collections defined for all targets of a type in $OH/sysman/admin/default_collection/*.xml
      • additional collections for a particular target in $EM/sysman/emd/collection/*.xml
      • specifies, by metric, schedule for collection and thresholds to be applied to columns
  • Blackout Manager
    • Manage blackout information stored in $EM/sysman/emd/blackouts.xml
    • Scheduled collections consult Blackout Manager; if target is currently blacked-out, collection does not proceed
    • Targets may be affected by multiple blackouts; if any blackout is effective on a target, the target is blacked-out
    • Node blackouts affect all targets monitored by the agent
    • Blackouts file :
      • blackouts in $EM/sysman/emd/blackouts.xml
      • each blackout can be applied to one or more targets; if target is node, blackout applies to all targets
      • blackout can be immediate or scheduled; if scheduled, can be one-time or repeated
  • Scheduler
    • Schedules activities in order of next run time
      • multiple schedule formats:
        • Once: happens only once
        • Interval: happens every n minutes/hours/days
        • Week: happens on certain day of week
        • Month: happens on certain day of month
      • can specify begin time/end time
    • Spawns threads to do work whose time has arrived
    • Used by Collector and Blackout Manager
    • Health Monitor checks that the scheduler is doing its work
    • emctl status agent scheduler
      • Dumps out all the scheduled elements
  • Upload Manager
    • As data is collected by other agent components, serializes writing of  intermediary .dat files (stored in $AS/sysman/emd/upload)
    • .dat files merged into .xml files on five priority channels
    • XML files sent to OMS as HTTP requests
    • maintains statistics on pending xml files; will disable collections based on number of files, aggregate size of files, and percentage free disk space on upload filesystem
    • Upload interval dynamic, based on properties and previous upload status
  • Ping Manager
    • Periodically, sends HTTP heartbeat request to OMS and verifies response
    • OMS response dictates interval before next ping
    • Exchange timezone information
    • A successful ping from the agent to the OMS is required before any uploads will occur

如何启用EM-Grid/Cloud Control的HTTP而非HTTPS协议登陆

默认EM Grid/Cloud Control的HTTPS协议登陆端口是7799,而HTTP协议的端口是7788,但是请注意默认不会启用HTTP协议登陆EM。准确地说Oracle更prefer 安全的HTTPS协议。

如果你使用HTTP登陆EM,如http://hostname:7788/em时可能遇到403错误(HTTP 403 Forbidden),这是由于默认不启用HTTP协议登陆EM。

该问题可以通过emctl secure unlock -console解锁console,并重启OMS来解决:

Enterprise Manager Grid Control – Version: 11.1.0.1 and later [Release: 11.1 and later ]
Information in this document applies to any platform.
Symptoms

Trying to access the 11g Grid Console using the URL: http://omsmachine.domain:7788/em resulted in the error:

The website declined to show this webpage

Using the https URL: https://omsmachine.domain:7799/em works fine.
Unlocked the Console access from Secure mode using:

cd <OMS_HOME>/bin
emctl secure unlock -console

The command completes successfully and the ’emctl status oms -details’ command returns the output:

HTTP Console Port : 7788
HTTPS Console Port : 7799
HTTP Upload Port : 4889
HTTPS Upload Port : 4900
OMS is not configured with SLB or virtual hostname
Agent Upload is unlocked.
OMS Console is unlocked.

But, trying to accessing the 11g Grid Console using the HTTP Console port still fails.
Internet explorer returns:

‘HTTP 403 Forbidden’ message

and Mozilla Firefox returns:

Forbidden

You are not authorised to access this resource on the server.
Changes

Access to the 11g Grid Console in Secure mode has been unlocked.
Cause

The OMS has to be restarted after unlocking the Console access in secure mode. Without this, the Console will not be accessible using the http protocol.
Solution

– After executing:

cd <OMS_HOME>/bin
emctl secure unlock -console

– Stop and re-start the OMS:

cd <OMS_HOME>/bin
emctl stop oms
emctl start oms

– After the above, the 11g Grid Console will be accessible using the http protocol and port.

 

Script:List Grid Control Jobs

以下脚本可以用于列出Grid Control中的定式作业:

SET verify OFF
SET linesize 255
SET pagesize 128
SET trimout ON
SET trimspool ON
SPOOL jobdump.log
ALTER SESSION SET nls_date_format='MON-DD-YYYY hh:mi:ss pm';
 
COLUMN status format a15
 
COLUMN job_name FORMAT a64
COLUMN job_type FORMAT a32
COLUMN job_owner FORMAT a32
COLUMN job_status format 99
COLUMN target_type format a64
 
COLUMN frequency_code format a20
COLUMN  interval format 99999999
 
VARIABLE JOBID VARCHAR2(64);
 
PROMPT *********************** JOB INFO ********************************
 
REM Get the job id
SET serveroutput on
BEGIN
    SELECT job_id INTO :JOBID
    FROM   MGMT_JOB
    WHERE  job_name='&&jobName'
    AND    job_owner='&&jobOwner'
    AND    nested=0;
 
EXCEPTION
    WHEN NO_DATA_FOUND THEN
    BEGIN
        DBMS_OUTPUT.put_line('JOB NOT FOUND, TRYING NAME ONLY');
        SELECT job_id INTO :JOBID
        FROM   MGMT_JOB
        WHERE  job_name='&&jobName'
        AND    nested=0
        AND    ROWNUM=1;
    EXCEPTION
        WHEN NO_DATA_FOUND THEN
            DBMS_OUTPUT.put_line('JOB NOT FOUND');
    END;
END;
/
 
SELECT  job_name, job_owner, job_type, system_job, job_status, target_type
FROM    MGMT_JOB
WHERE   job_id=HEXTORAW(:JOBID);
 
PROMPT *********************** JOB SCHEDULE ****************************
SELECT  DECODE(frequency_code,
               1, 'Once',
               2, 'Interval',
               3, 'Daily',
               4, 'Day of Week',
               5, 'Day of Month',
               6, 'Day of Year', frequency_code) "FREQUENCY_CODE",
        start_time, end_time, execution_hours, execution_minutes,
        interval, months, days, timezone_info, timezone_target_index,
        timezone_offset, timezone_region
FROM    MGMT_JOB_SCHEDULE s, MGMT_JOB j
WHERE   s.schedule_id=j.schedule_id
AND     j.job_id=HEXTORAW(:JOBID);
 
PROMPT ********************** PARAMETERS ********************************
SELECT  parameter_name,
        decode(parameter_type,
               0, 'Scalar',
               1, 'Vector',
               2, 'Large', parameter_type) "PARAMETER_TYPE",
        scalar_value, vector_value
FROM    MGMT_JOB_PARAMETER
WHERE   job_id=HEXTORAW(:JOBID)
AND     execution_id=HEXTORAW('0000000000000000')
ORDER BY parameter_name;
 
PROMPT ********************** TARGETS ********************************
SELECT  target_name, target_type
FROM    MGMT_JOB_TARGET jt, MGMT_TARGETS t
WHERE   job_id=HEXTORAW(:JOBID)
AND     execution_id=HEXTORAW('0000000000000000')
AND     jt.target_guid=t.target_guid
ORDER BY target_type, target_name;
 
PROMPT ********************** FLAT TARGETS ********************************
SELECT  target_name, target_type
FROM    MGMT_JOB_FLAT_TARGETS jft, MGMT_TARGETS t
WHERE   job_id=HEXTORAW(:JOBID)
AND     jft.target_guid=t.target_guid
ORDER BY target_type, target_name;
 
 
PROMPT ************************ EXECUTIONS *******************************
SELECT  execution_id,
        DECODE(status,
               1, 'SCHEDULED',
               2, 'RUNNING',
               3, 'FAILED INIT',
               4, 'FAILED',
               5, 'SUCCEEDED',
               6, 'SUSPENDED',
               7, 'AGENT DOWN',
               8, 'STOPPED',
               9, 'SUSPENDED/LOCK',
               10, 'SUSPENDED/EVENT',
               11, 'SUSPENDED/BLACKOUT',
               12, 'STOP PENDING',
               13, 'SUSPEND PENDING',
               14, 'INACTIVE',
               15, 'QUEUED',
               16, 'FAILED/RETRIED',
               17, 'WAITING',
               18, 'SKIPPED', status) "STATUS",
        scheduled_time, start_time, end_time
FROM    MGMT_JOB_EXEC_SUMMARY e
WHERE   job_id=HEXTORAW(:JOBID)
ORDER BY scheduled_time;
 
 
UNDEFINE jobName
UNDEFINE jobOwner
UNDEFINE JOBID


SPOOL OFF

Script:GridControl Repository Health Check

以下脚本可以用于检查Grid Control(OMS) Repository的健康程度:

set linesize 130
set pagesize 50000
Set feedback off
set heading off
SPOOL RepositoryHealth.log

PROMPT ********* Report to capture the Health of the EM Grid Control Repository  ******

Prompt
Prompt ***   EM Vital Statistics   ***
Prompt --------------------------------

--- Total number of targets monitored by EM
select 'Total Targets=', TO_CHAR(count(*)) from mgmt_targets;

--- Number of targets that are not listed with an 'UP' availability status.
select 'Targets Not Up=',to_char(count(*)) from mgmt_current_availability where current_status != 1;

--- Loader Thread count
select  'Loader Threads=', TO_CHAR(count(distinct key_value))
from mgmt_metrics_1hour h, mgmt_metrics m, mgmt_targets t where
t.target_type = 'oracle_emrep' and
m.target_type = t.target_type and
m.metric_name = 'Management_Loader_Status' and
m.metric_column = 'load_processing' and
m.metric_guid = h.metric_guid and
h.target_guid = t.target_guid and
rollup_timestamp > sysdate-7 and
t.type_meta_ver = m.type_meta_ver and
(t.category_prop_1 = m.category_prop_1 OR m.category_prop_1 = ' ') and
(t.category_prop_2 = m.category_prop_2 OR m.category_prop_2 = ' ') and
(t.category_prop_3 = m.category_prop_3 OR m.category_prop_3 = ' ') and
(t.category_prop_4 = m.category_prop_4 OR m.category_prop_4 = ' ') and
(t.category_prop_5 = m.category_prop_5 OR m.category_prop_5 = ' ');

--- Rows per second per loader thread.
select 'Avg Loader Rows/Second/Thread=', TO_CHAR(round(avg(value_average),2))
from mgmt_metrics_1hour h, mgmt_metrics m, mgmt_targets t where
t.target_type = 'oracle_emrep' and
m.target_type = t.target_type and
m.metric_name = 'Management_Loader_Status' and
m.metric_column = 'load_processing' and
m.metric_guid = h.metric_guid and
h.target_guid = t.target_guid and
rollup_timestamp > sysdate-7 and
t.type_meta_ver = m.type_meta_ver and
(t.category_prop_1 = m.category_prop_1 OR m.category_prop_1 = ' ') and
(t.category_prop_2 = m.category_prop_2 OR m.category_prop_2 = ' ') and
(t.category_prop_3 = m.category_prop_3 OR m.category_prop_3 = ' ') and
(t.category_prop_4 = m.category_prop_4 OR m.category_prop_4 = ' ') and
(t.category_prop_5 = m.category_prop_5 OR m.category_prop_5 = ' ')
group by metric_name, metric_column;
 
--- Loader rows per hour
select 'Avg Loader Rows/Hour/Thread=', TO_CHAR(round(avg(value_average),2))
from mgmt_metrics_1hour h, mgmt_metrics m, mgmt_targets t where
t.target_type = 'oracle_emrep' and
m.target_type = t.target_type and
m.metric_name = 'Management_Loader_Status' and
m.metric_column = 'loader_processing_hour' and
m.metric_guid = h.metric_guid and
h.target_guid = t.target_guid and
rollup_timestamp > sysdate-7 and
t.type_meta_ver = m.type_meta_ver and
(t.category_prop_1 = m.category_prop_1 OR m.category_prop_1 = ' ') and
(t.category_prop_2 = m.category_prop_2 OR m.category_prop_2 = ' ') and
(t.category_prop_3 = m.category_prop_3 OR m.category_prop_3 = ' ') and
(t.category_prop_4 = m.category_prop_4 OR m.category_prop_4 = ' ') and
(t.category_prop_5 = m.category_prop_5 OR m.category_prop_5 = ' ')
group by metric_name, metric_column;
 
--- Loader thread run time % of hour
select 'Avg Loader Pct Hour Run/Thread=', TO_CHAR(round((avg(value_average)/3600)*100,2))
from mgmt_metrics_1hour h, mgmt_metrics m, mgmt_targets t where
t.target_type = 'oracle_emrep' and
m.target_type = t.target_type and
m.metric_name = 'Management_Loader_Status' and
m.metric_column = 'load_run' and
m.metric_guid = h.metric_guid and
h.target_guid = t.target_guid and
rollup_timestamp > sysdate-7 and
t.type_meta_ver = m.type_meta_ver and
(t.category_prop_1 = m.category_prop_1 OR m.category_prop_1 = ' ') and
(t.category_prop_2 = m.category_prop_2 OR m.category_prop_2 = ' ') and
(t.category_prop_3 = m.category_prop_3 OR m.category_prop_3 = ' ') and
(t.category_prop_4 = m.category_prop_4 OR m.category_prop_4 = ' ') and
(t.category_prop_5 = m.category_prop_5 OR m.category_prop_5 = ' ')
group by metric_name, metric_column;
 
--- Rollup Rows per hour
select 'Avg Rollup Rows/Hour=', TO_CHAR(round(avg(value_average),2))
from mgmt_metrics_1hour h, mgmt_metrics m, mgmt_targets t where
t.target_type = 'oracle_emrep' and
m.target_type = t.target_type and
m.metric_name = 'DBMS_Job_Status' and
m.metric_column = 'jobthroughput' and
h.key_value = 'Rollup' and
m.metric_guid = h.metric_guid and
h.target_guid = t.target_guid and
rollup_timestamp > sysdate-7 and
t.type_meta_ver = m.type_meta_ver and
(t.category_prop_1 = m.category_prop_1 OR m.category_prop_1 = ' ') and
(t.category_prop_2 = m.category_prop_2 OR m.category_prop_2 = ' ') and
(t.category_prop_3 = m.category_prop_3 OR m.category_prop_3 = ' ') and
(t.category_prop_4 = m.category_prop_4 OR m.category_prop_4 = ' ') and
(t.category_prop_5 = m.category_prop_5 OR m.category_prop_5 = ' ')
group by metric_name, metric_column;

--- Rollup % of hour run
select 'Avg Rollup Pct Hour Run=', TO_CHAR(round(avg(value_average),2))
from mgmt_metrics_1hour h, mgmt_metrics m, mgmt_targets t where
t.target_type = 'oracle_emrep' and
m.target_type = t.target_type and
m.metric_name = 'DBMS_Job_Status' and
m.metric_column = 'jobprocessing' and
h.key_value = 'Rollup' and
m.metric_guid = h.metric_guid and
h.target_guid = t.target_guid and
rollup_timestamp > sysdate-7 and
t.type_meta_ver = m.type_meta_ver and
(t.category_prop_1 = m.category_prop_1 OR m.category_prop_1 = ' ') and
(t.category_prop_2 = m.category_prop_2 OR m.category_prop_2 = ' ') and
(t.category_prop_3 = m.category_prop_3 OR m.category_prop_3 = ' ') and
(t.category_prop_4 = m.category_prop_4 OR m.category_prop_4 = ' ') and
(t.category_prop_5 = m.category_prop_5 OR m.category_prop_5 = ' ')
group by metric_name, metric_column;

--- Number of job dispatchers (hint: equal to number of Management Servers)
select  'Job Dispatchers=', TO_CHAR(count(distinct key_value))
from mgmt_metrics_1hour h, mgmt_metrics m, mgmt_targets t where
t.target_type = 'oracle_emrep' and
m.target_type = t.target_type and
m.metric_name = 'Job_Dispatcher_Performance' and
m.metric_column = 'throughput' and
m.metric_guid = h.metric_guid and
h.target_guid = t.target_guid and
rollup_timestamp > sysdate-7 and
t.type_meta_ver = m.type_meta_ver and
(t.category_prop_1 = m.category_prop_1 OR m.category_prop_1 = ' ') and
(t.category_prop_2 = m.category_prop_2 OR m.category_prop_2 = ' ') and
(t.category_prop_3 = m.category_prop_3 OR m.category_prop_3 = ' ') and
(t.category_prop_4 = m.category_prop_4 OR m.category_prop_4 = ' ') and
(t.category_prop_5 = m.category_prop_5 OR m.category_prop_5 = ' ');

 
--- Job steps processed per second
select 'Avg Job Steps/Second=', TO_CHAR(round(avg(value_average),2))
from mgmt_metrics_1hour h, mgmt_metrics m, mgmt_targets t where
t.target_type = 'oracle_emrep' and
m.target_type = t.target_type and
m.metric_name = 'Job_Dispatcher_Performance' and
m.metric_column = 'throughput' and
m.metric_guid = h.metric_guid and
h.target_guid = t.target_guid and
rollup_timestamp > sysdate-7 and
t.type_meta_ver = m.type_meta_ver and
(t.category_prop_1 = m.category_prop_1 OR m.category_prop_1 = ' ') and
(t.category_prop_2 = m.category_prop_2 OR m.category_prop_2 = ' ') and
(t.category_prop_3 = m.category_prop_3 OR m.category_prop_3 = ' ') and
(t.category_prop_4 = m.category_prop_4 OR m.category_prop_4 = ' ') and
(t.category_prop_5 = m.category_prop_5 OR m.category_prop_5 = ' ')
group by metric_name, metric_column;

 
---  Notifications Per Second
select 'Avg Notifications/Second=', TO_CHAR(round(avg(value_average),2))
from mgmt_metrics_1hour h, mgmt_metrics m, mgmt_targets t where
t.target_type = 'oracle_emrep' and
m.target_type = t.target_type and
m.metric_name = 'Notification_Performance' and
m.metric_column = 'notificationthroughput' and
m.metric_guid = h.metric_guid and
h.target_guid = t.target_guid and
rollup_timestamp > sysdate-7 and
t.type_meta_ver = m.type_meta_ver and
(t.category_prop_1 = m.category_prop_1 OR m.category_prop_1 = ' ') and
(t.category_prop_2 = m.category_prop_2 OR m.category_prop_2 = ' ') and
(t.category_prop_3 = m.category_prop_3 OR m.category_prop_3 = ' ') and
(t.category_prop_4 = m.category_prop_4 OR m.category_prop_4 = ' ') and
(t.category_prop_5 = m.category_prop_5 OR m.category_prop_5 = ' ')
group by metric_name, metric_column;

--- Notification % of hour run
select 'Avg Notification Pct Hour Run=', TO_CHAR(round(avg(value_average),2))
from mgmt_metrics_1hour h, mgmt_metrics m, mgmt_targets t where
t.target_type = 'oracle_emrep' and
m.target_type = t.target_type and
m.metric_name = 'Notification_Performance' and
m.metric_column = 'notificationprocessing' and
m.metric_guid = h.metric_guid and
h.target_guid = t.target_guid and
rollup_timestamp > sysdate-7 and
t.type_meta_ver = m.type_meta_ver and
(t.category_prop_1 = m.category_prop_1 OR m.category_prop_1 = ' ') and
(t.category_prop_2 = m.category_prop_2 OR m.category_prop_2 = ' ') and
(t.category_prop_3 = m.category_prop_3 OR m.category_prop_3 = ' ') and
(t.category_prop_4 = m.category_prop_4 OR m.category_prop_4 = ' ') and
(t.category_prop_5 = m.category_prop_5 OR m.category_prop_5 = ' ')
group by metric_name, metric_column;

--- Severities per hour
select 'Avg Severities Per Hour=', TO_CHAR(round(avg(count(*)),2)) as sev_count from mgmt_severity
where collection_timestamp > sysdate - 7
group by trunc(collection_timestamp, 'HH');

--- OMS Host CPU Util
select 'Avg Management Server Host CPU=', TO_CHAR(round(avg(value_average),2)), t.target_name
from mgmt_metrics_1hour h, mgmt_targets t where
h.target_guid in (select target_guid from mgmt_targets t where
target_type = 'host' and
target_name in (select substr(host_url,1,instr(host_url, '_Management_Service',-1,1)-1)
                                 from mgmt_oms_parameters)) and
h.metric_guid = (select m.metric_guid from mgmt_metrics m where
                          m.target_type = 'host' and
                          m.metric_name = 'Load' and
                          m.metric_column = 'cpuUtil'and
                          t.type_meta_ver = m.type_meta_ver and
                         (t.category_prop_1 = m.category_prop_1 OR m.category_prop_1 = ' ') and
                         (t.category_prop_2 = m.category_prop_2 OR m.category_prop_2 = ' ') and
                         (t.category_prop_3 = m.category_prop_3 OR m.category_prop_3 = ' ') and
                         (t.category_prop_4 = m.category_prop_4 OR m.category_prop_4 = ' ') and
                         (t.category_prop_5 = m.category_prop_5 OR m.category_prop_5 = ' ')) and
h.target_guid = t.target_guid and
rollup_timestamp > sysdate-7
group by t.target_name;

--- Create a small table to get DB Guids. GV$INSTANCE performance is inconsistant across db versions.
create table mgmt_temp_vsign_db_guids as select h.target_guid from mgmt_targets h where
                   h.target_type = 'host' and
                   h.target_name in (select s.target_name from mgmt_targets s, gv$instance g where
                                                      s.target_type = 'host' and
                                                      s.target_name like g.host_name || '%');

--- Run CBO stat analysis on temp table for performance.
exec dbms_stats.gather_table_stats('SYSMAN','MGMT_TEMP_VSIGN_DB_GUIDS',NULL, DBMS_STATS.AUTO_SAMPLE_SIZE, FALSE,'FOR ALL COLUMNS SIZE AUTO',NULL,'GLOBAL',TRUE,NULL,NULL,NULL);

--- EM Repository CPU
select 'Avg Management Repository Host CPU=', TO_CHAR(round(avg(value_average),2)), t.target_name
from mgmt_metrics_1hour h, mgmt_targets t where
h.target_guid in (select target_guid from mgmt_temp_vsign_db_guids) and
h.metric_guid = (select m.metric_guid from mgmt_metrics m where
                           m.target_type = 'host' and
                           m.metric_name = 'Load' and
                           m.metric_column = 'cpuUtil'and
                           t.type_meta_ver = m.type_meta_ver and
                          (t.category_prop_1 = m.category_prop_1 OR m.category_prop_1 = ' ') and
                          (t.category_prop_2 = m.category_prop_2 OR m.category_prop_2 = ' ') and
                          (t.category_prop_3 = m.category_prop_3 OR m.category_prop_3 = ' ') and
                          (t.category_prop_4 = m.category_prop_4 OR m.category_prop_4 = ' ') and
                          (t.category_prop_5 = m.category_prop_5 OR m.category_prop_5 = ' ')) and
h.target_guid = t.target_guid and
rollup_timestamp > sysdate-7
group by t.target_name;

--- Repository Used Space
select 'Repository Used Space (GB)=',  TO_CHAR(round(max(value_average)/1000,2))
from mgmt_metrics_1hour h, mgmt_metrics m, mgmt_targets t where
t.target_type = 'oracle_emrep' and
m.target_type = t.target_type and
m.metric_name = 'Configuration' and
m.metric_column = 'usedRepSpace' and
m.metric_guid = h.metric_guid and
h.target_guid = t.target_guid and
rollup_timestamp > sysdate-1 and
t.type_meta_ver = m.type_meta_ver and
(t.category_prop_1 = m.category_prop_1 OR m.category_prop_1 = ' ') and
(t.category_prop_2 = m.category_prop_2 OR m.category_prop_2 = ' ') and
(t.category_prop_3 = m.category_prop_3 OR m.category_prop_3 = ' ') and
(t.category_prop_4 = m.category_prop_4 OR m.category_prop_4 = ' ') and
(t.category_prop_5 = m.category_prop_5 OR m.category_prop_5 = ' ');


--- Drop the temp table
drop table mgmt_temp_vsign_db_guids;
 
set heading on

PROMPT ------------------------------------------------------------------------------------------
PROMPT
PROMPT **** The number of management Errors that have occurred per Module in the past 24 hours
 SELECT module_name Module, COUNT(*)
        FROM   mgmt_system_error_log
        WHERE  occur_date> SYSDATE-1
        GROUP BY module_name
        ORDER BY module_name;

PROMPT **** The Following Errors have occurred in the past two days
alter session set nls_date_format='MM/DD/YYYY HH24:MI';
column Module format a15
column Agent format a20
column error_msg format a60
select occur_date, module_name Module, emd_url Agent, error_msg
  from mgmt_system_error_log
 where occur_date > sysdate - 2
 order by occur_date;


 
PROMPT ------------------------------------------------------------------------------------------

PROMPT **** Check the status of all DBMS_JOBS and look for broken jobs or next_run data that is already in the past.
column Job format a8
column interval format a25
column what format a45

        SELECT to_char(job) Job, TO_CHAR(next_date,'DD-MON-YYYY HH24:MI:SS') next_run,
           interval, broken, what
        FROM   user_jobs
        ORDER BY next_date DESC;

PROMPT  -----------------------------------------------------------------------------------------

PROMPT **** Show the performance statistics of all jobs over the last 24 hours:
column Job_name format a50
     SELECT job_name, COUNT(*) n_recs,
            MIN(duration) min_duration, MAX(duration) max_duration, ROUND(AVG(duration),2) avg_duration
     FROM   mgmt_system_performance_log
     WHERE  is_total = 'Y'
       AND  time> SYSDATE-1
     GROUP BY job_name
     ORDER BY job_name;

 
PROMPT ------------------------------------------------------------------------------------------

PROMPT **** Overview of all Agent XML activity of the last day:
SELECT module, cnt "Number", TO_CHAR(FLOOR(mins/60),'999')||':'||TO_CHAR(MOD(mins,60),'09')||':'||TO_CHAR(MOD(secs,60),'09') "Time Spent"
FROM   (SELECT module, COUNT(*) cnt, ROUND(SUM(duration)/1000) secs, FLOOR(SUM(duration)/60000) mins
        FROM   mgmt_system_performance_log
        WHERE  job_name = 'LOADER'
          AND  time > SYSDATE-1
        GROUP BY module)
ORDER BY cnt DESC;

spool off

Enterprise Manager show database status is unavailable

客户的一套11.2.0.1 RAC数据库部署了 Enterprise Manager 11g以方便管理,但系统运行一段时间后EM就变得无法正常使用了,页面显示”the database status is currently unavailable…….”,但实际上数据库和监听都处于正常可用的状态。

sysman_password

作为乙方来说对Enterprise Manager并不太熟悉,虽然EM是一个十分强大的图形化监控工具,但是我们很少有机会在生产环境中用到它,这一点可能和甲方的DBA不同。

针对该EM不可用的问题首先想到了尝试重启dbconsole,以验证该问题是否可重现。

emctl stop  dbconsole
emctl start dbconsole

重启dbconsole后问题依旧,查MOS发现引发该EM不可用问题的可能性有多种,通过$ORACLE_HOME/_/sysman/log/emoms.log的EM相关日志可以进一步定位该问题:

tail -500 emoms.log

 at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:871)
        at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:453)
        at com.evermind.server.http.HttpRequestHandler.serveOneRequest(HttpRequestHandler.java:221)
        at com.evermind.server.http.HttpRequestHandler.run(HttpRequestHandler.java:122)
        at com.evermind.server.http.HttpRequestHandler.run(HttpRequestHandler.java:111)
        at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
        at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234)
        at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29)
        at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879)
        at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
        at java.lang.Thread.run(Thread.java:595)
2011-08-02 14:39:46,131 [EMUI_14_39_46_/console/aboutApplication] ERROR em.console doGet.363 - java.lang.IllegalStateException: Response has already been committed,
be sure not to write to the OutputStream or to trigger a commit due to any other action before calling this method.
java.lang.IllegalStateException: Response has already been committed, be sure not to write to the OutputStream or to trigger a commit due to any other action before calling this method.
        at com.evermind.server.http.EvermindHttpServletResponse.sendRedirect(EvermindHttpServletResponse.java:1371)
        at oracle.sysman.emSDK.svlt.EMServlet.myDoGet(EMServlet.java:812)
        at oracle.sysman.emSDK.svlt.EMServlet.doGet(EMServlet.java:343)
        at oracle.sysman.eml.app.Console.doGet(Console.java:157)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
        at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64)
        at oracle.sysman.eml.app.EMRepLoginFilter.doFilter(EMRepLoginFilter.java:107)
        at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:15)
        at oracle.sysman.db.adm.inst.HandleRepDownFilter.doFilter(HandleRepDownFilter.java:146)
        at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
        at oracle.sysman.eml.app.BrowserVersionFilter.doFilter(BrowserVersionFilter.java:122)
        at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
        at oracle.sysman.emSDK.svlt.EMRedirectFilter.doFilter(EMRedirectFilter.java:102)
        at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
        at oracle.sysman.eml.app.ContextInitFilter.doFilter(ContextInitFilter.java:327)
        at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:621)
        at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:370)
        at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:871)
        at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:453)
        at com.evermind.server.http.HttpRequestHandler.serveOneRequest(HttpRequestHandler.java:221)
        at com.evermind.server.http.HttpRequestHandler.run(HttpRequestHandler.java:122)
        at com.evermind.server.http.HttpRequestHandler.run(HttpRequestHandler.java:111)
        at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
        at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234)
        at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29)
        at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879)
        at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
        at java.lang.Thread.run(Thread.java:595)
2011-08-02 14:39:46,150 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:39:46,157 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean01.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:39:46,182 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean04.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:39:51,352 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:39:51,394 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean03.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:39:51,509 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:39:51,537 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean03.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:39:56,757 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:03,574 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:03,603 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean02.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:40:03,729 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:03,800 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean02.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:40:09,328 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:09,361 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean03.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:40:11,946 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:16,285 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:16,307 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:16,321 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean01.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:40:16,368 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean04.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:40:16,437 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:16,472 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean01.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:40:16,507 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:16,536 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean04.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:40:17,187 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:17,189 [EMUI_14_40_17_/console/aboutApplication] ERROR svlt.PageHandler handleRequest.639 - java.lang.IllegalStateException: Response has already been committed
java.lang.IllegalStateException: Response has already been committed
        at com.evermind.server.http.EvermindHttpServletResponse.resetBuffer(EvermindHttpServletResponse.java:1892)
        at com.evermind.server.http.ServletRequestDispatcher.unprivileged_forward(ServletRequestDispatcher.java:249)
        at com.evermind.server.http.ServletRequestDispatcher.access$100(ServletRequestDispatcher.java:51)
        at com.evermind.server.http.ServletRequestDispatcher$2.oc4jRun(ServletRequestDispatcher.java:193)
        at oracle.oc4j.security.OC4JSecurity.doPrivileged(OC4JSecurity.java:284)
        at com.evermind.server.http.ServletRequestDispatcher.forward(ServletRequestDispatcher.java:198)
        at oracle.sysman.emSDK.svlt.PageHandler.render(PageHandler.java:1113)
        at oracle.sysman.emSDK.svlt.PageHandler.myRender(PageHandler.java:1173)
        at oracle.sysman.emSDK.svlt.PageHandler.handleRequest(PageHandler.java:597)
        at oracle.sysman.emSDK.svlt.EMServlet.myDoGet(EMServlet.java:787)
        at oracle.sysman.emSDK.svlt.EMServlet.doGet(EMServlet.java:343)
        at oracle.sysman.eml.app.Console.doGet(Console.java:157)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
        at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64)
        at oracle.sysman.eml.app.EMRepLoginFilter.doFilter(EMRepLoginFilter.java:107)
        at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:15)
        at oracle.sysman.db.adm.inst.HandleRepDownFilter.doFilter(HandleRepDownFilter.java:146)
        at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
        at oracle.sysman.eml.app.BrowserVersionFilter.doFilter(BrowserVersionFilter.java:122)
        at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
        at oracle.sysman.emSDK.svlt.EMRedirectFilter.doFilter(EMRedirectFilter.java:102)
        at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
        at oracle.sysman.eml.app.ContextInitFilter.doFilter(ContextInitFilter.java:327)
        at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:621)
        at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:370)
        at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:871)
        at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:453)
        at com.evermind.server.http.HttpRequestHandler.serveOneRequest(HttpRequestHandler.java:221)
        at com.evermind.server.http.HttpRequestHandler.run(HttpRequestHandler.java:122)
        at com.evermind.server.http.HttpRequestHandler.run(HttpRequestHandler.java:111)
        at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
        at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234)
        at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29)
        at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879)
        at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
        at java.lang.Thread.run(Thread.java:595)
2011-08-02 14:40:17,189 [EMUI_14_40_17_/console/aboutApplication] ERROR em.console doGet.363 - java.lang.IllegalStateException: Response has already been committed, be sure not to write to the OutputStream or to trigger a commit due to any other action before calling this method.
java.lang.IllegalStateException: Response has already been committed, be sure not to write to the OutputStream or to trigger a commit due to any other action before calling this method.
        at com.evermind.server.http.EvermindHttpServletResponse.sendRedirect(EvermindHttpServletResponse.java:1371)
        at oracle.sysman.emSDK.svlt.EMServlet.myDoGet(EMServlet.java:812)
        at oracle.sysman.emSDK.svlt.EMServlet.doGet(EMServlet.java:343)
        at oracle.sysman.eml.app.Console.doGet(Console.java:157)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
        at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64)
        at oracle.sysman.eml.app.EMRepLoginFilter.doFilter(EMRepLoginFilter.java:107)
        at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:15)
        at oracle.sysman.db.adm.inst.HandleRepDownFilter.doFilter(HandleRepDownFilter.java:146)
        at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
        at oracle.sysman.eml.app.BrowserVersionFilter.doFilter(BrowserVersionFilter.java:122)
        at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
        at oracle.sysman.emSDK.svlt.EMRedirectFilter.doFilter(EMRedirectFilter.java:102)
        at com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:17)
        at oracle.sysman.eml.app.ContextInitFilter.doFilter(ContextInitFilter.java:327)
        at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:621)
        at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:370)
        at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:871)
        at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:453)
        at com.evermind.server.http.HttpRequestHandler.serveOneRequest(HttpRequestHandler.java:221)
        at com.evermind.server.http.HttpRequestHandler.run(HttpRequestHandler.java:122)
        at com.evermind.server.http.HttpRequestHandler.run(HttpRequestHandler.java:111)
        at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
        at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234)
        at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29)
        at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879)
        at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
        at java.lang.Thread.run(Thread.java:595)
2011-08-02 14:40:21,661 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:21,691 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean03.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:40:21,822 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:21,886 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean03.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:40:27,141 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:33,941 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:33,975 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean02.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:40:34,092 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:34,124 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean02.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:40:42,088 [PAFStartupThread] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:42,332 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:42,584 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:42,611 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean01.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:40:46,594 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:46,622 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean01.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:40:46,653 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:46,683 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean04.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:40:46,738 [HTTPThreadGroup-6] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:46,766 [HTTPThreadGroup-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean01.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:40:46,793 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

2011-08-02 14:40:46,822 [HTTPThreadGroup-4] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://maclean04.askmaclean.com:3938/emd/main)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-28000: the account is locked
)
2011-08-02 14:40:48,244 [HTTPThreadGroup-4] ERROR conn.ConnectionService verifyRepositoryEx.891 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked

从以上em日志中可以发现最显著的问题为ORA-28000,显然是EM相关的数据库用户被lock住了。10g中与EM相关的数据库用户包括SYSMAN和DBSNMP,分别查询该2个用户的状态:

SQL> select account_status,lock_date,username from dba_users where username in ('SYSMAN','DBSNMP');

ACCOUNT_STATUS                   LOCK_DATE       USERNAME
-------------------------------- --------------- ------------------------------
OPEN                                             DBSNMP
LOCKED(TIMED)                    02-JUL-11       SYSMAN

发现SYSMAN用户在7月份被锁定了,而且是LOCKED(TIMED),说明是因为登陆密码不正确多次登陆后被锁的。LOCKED(TIMED)的相关信息以下Note:

DBA_USERS.ACCOUNT_STATUS shows LOCKED(TIMED) instead of LOCKED even if PASSWORD_LOCK_TIME set to unlimited [ID 284344.1]
  	Modified 18-FEB-2011     Type PROBLEM     Status PUBLISHED

In this Document
  Symptoms
  Cause
  Solution
  References

Applies to:
Oracle Server - Enterprise Edition - Version: 9.2.0.1 to 11.2.0.1 - Release: 9.2 to 11.2
Information in this document applies to any platform.
Checked for relevance on 21-Sep-2010
Symptoms
DBA_USERS.ACCOUNT_STATUS shows LOCKED(TIMED) instead of LOCKED for a user with a profile with a PASSWORD_LOCK_TIME
set to unlimited and number of failed logins > FAILED_LOGIN_ATTEMPTS

For eg.

create profile my_profile limit failed_login_attempts 3 password_lock_time unlimited ;

SQL> create user my_user identified by my_user default tablespace users temporary tablespace temp profile my_profile ;

User created.

SQL> grant connect to my_user ;

(after 3 failed login attempts)

USERNAME PROFILE ACCOUNT_STATUS
------------------------------ ------------------------------ --------------
MY_USER MY_PROFILE LOCKED(TIMED)
Cause
Up until Oracle RDBMS version 11.1.0.7, after a number of failed logins > FAILED_LOGIN_ATTEMPTS, the account
status of the user will always be LOCKED(TIMED).
One would only see LOCKED if the account was explicitly locked during create or alter user/role
(ie after using CREATE USER or ALTER USER statements)

eg.

alter user my_lock account lock ;

USERNAME PROFILE ACCOUNT_STATUS
-------------------------------
MY_USER MY_PROFILE LOCKED

Solution
Expected behaviour is
1. Oracle release is  FAILED_LOGIN_ATTEMPTS

2. Oracle release is >= 11.2 and PASSWORD_LOCK_TIME = unlimited:
DBA_USERS.ACCOUNT_STATUS = LOCKED whenever the number of failed login attempts is > FAILED_LOGIN_ATTEMPTS

3. Oracle release is >= 11.2 and PASSWORD_LOCK_TIME =
DBA_USERS.ACCOUNT_STATUS = LOCKED(TIMED) whenever the number of failed login attempts is > FAILED_LOGIN_ATTEMPTS

于是我尝试将SYSMAN用户解锁,并重启EM并再次尝试登陆:

SQL> alter user sysman account unlock;
User altered.

重启EM后重新登陆,发现问题仍未解决,再次检查账户信息

SQL> select account_status,lock_date,username from dba_users where username in ('SYSMAN','DBSNMP');

ACCOUNT_STATUS                   LOCK_DATE       USERNAME
-------------------------------- --------------- ------------------------------
OPEN                                             DBSNMP
LOCKED(TIMED)                    02-AUG-11       SYSMAN

发现SYSMAN用户又被LOCK住了,显然是有人在数据库内修改了SYSMAN的密码后,没有重新配置EM中的SYSMAN密码,导致每次EM启动时都会尝试用错误的密码登陆数据库,最终造成SYSMAN用户被LOCKED(TIMED)住,并出现”ORA-28000: the account is locked”错误。

找出根本原因后问题就显得十分简单了,我们只需要关闭EM,并同步数据库和EM中的SYSMAN密码,再重启EM就可以了:

1.
首先关闭EM
export ORACLE_SID=
export ORACLE_UNQNAME=
cd $ORACLE_HOME/bin
./emctl stop dbconsole

2.
修改DB中的SYSMAN密码并解锁用户,并验证登陆正常
alter user sysman identified by  ;

alter user sysman account unlock;

conn sysman/password

3.
更新EM中的SYSMAN密码
export ORACLE_SID=
export ORACLE_UNQNAME=
cd $ORACLE_HOME/bin
./emctl setpasswd dbconsole

Oracle Enterprise Manager 11g Database Control Release 11.2.0.1.0
Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.
https://maclean.askmaclean.com:1158/em/console/aboutApplication
Please enter new repository password:
Repository password successfully updated.

4.
启动EM
export ORACLE_SID=
export ORACLE_UNQNAME=
cd $ORACLE_HOME/bin
./emctl start dbconsole

10.2.0.1 db console启动失败问题一例

帮网友诊断10.2.0.1数据库安装后配置EM的问题一例,主要是在EM启动过程中遇到了错误,日志如下:

2009-7-7 20:11:39 oracle.sysman.emcp.EMConfig perform
严重: 启动 Database Control 时出错
有关详细资料, 请参阅 oracle\product\10.2.0\db_1\cfgtoollogs\dbca\orcl\emConfig.log 中的日志文件。
2009-7-7 20:11:39 oracle.sysman.emcp.EMConfig perform
配置: Stack Trace:
oracle.sysman.emcp.exception.EMConfigException: 启动 Database Control 时出错
	at oracle.sysman.emcp.EMDBPostConfig.performConfiguration(EMDBPostConfig.java:569)
	at oracle.sysman.emcp.EMDBPostConfig.invoke(EMDBPostConfig.java:181)
	at oracle.sysman.emcp.EMDBPostConfig.invoke(EMDBPostConfig.java:150)
	at oracle.sysman.emcp.EMConfig.perform(EMConfig.java:155)
	at oracle.sysman.assistants.util.em.EMConfiguration.run(EMConfiguration.java:430)
	at java.lang.Thread.run(Thread.java:534)

出现错误的java stack call是EMDBPostConfig说明em db repository 已经完成配置在尝试启动EM,但此时出现了问题。

实际造成该问题的有多种可能性,这里列举最常见的几种:

1. 杀毒软件或者防火墙造成的,这里建议在安装windows期间彻底关闭杀毒类软件和Windows自带的防火墙
2. EM使用的端口已经被其他软件所占用,建议使用netstat命令诊断,或另择网络端口
3. Windows主机名以”U”字母开头

针对第三种可能性,可以在MOS上找到Note <Problem: Startup: Db Control 10.2.0.1 Fails To Start when the hostname begins with the letter “u”>:

Problem: Startup: Db Control 10.2.0.1 Fails To Start when the hostname begins with the letter "u" 

Applies to:
Enterprise Manager for RDBMS - Version: 10.2.0.1 and later   [Release: 10.2 and later ]

Symptoms

The dbconsole 10.2.0.1 is created with emca.The repository creation completes successfully and the
dbconsole is created and configured without any error.
The last phase of emca is to start the dbconsole.
The standalone agent is started successfully but the dbconsole fails to start.

The file $ORACLE_HOME/cfgtoollogs/emca/sid/emca*.log shows:
===================================================
05/11/2005 23:07:49 oracle.sysman.emcp.util.PlatformInterface serviceCommand
CONFIG: Waiting for service 'OracleDBConsoleAMITAL' to fully start
05/11/2005 23:07:59 oracle.sysman.emcp.EMConfig perform
SEVERE: Error starting Database Control
Refer to the log file at
d:\oracle\product\10.2.0\db_1\cfgtoollogs\emca\amital\emca_2005-11-05_10-55-42-PM.log for more
details.
05/11/2005 23:07:59 oracle.sysman.emcp.EMConfig perform
CONFIG: Stack Trace:
oracle.sysman.emcp.exception.EMConfigException: Error starting Database Control
  at oracle.sysman.emcp.EMDBPostConfig.performConfiguration(EMDBPostConfig.java:569)
  at oracle.sysman.emcp.EMDBPostConfig.invoke(EMDBPostConfig.java:181)
Cause
This issue can be reproduced on all windows platforms which name begins with the letter u.
This issue has been logged in the following bug:
Bug:4714774 DBCONSOLE DOES NOT WORK HAVING A HOSTNAME STARTING WITH "U"
This bug will be fixed in the patchset 10.2.0.2
Solution

Download and apply the RDBMS patchset 10.2.0.2 once/if available or

To enable the dbconsole to start successfully, follow the steps listed below:

1. Save the file $ORACLE_HOME\host_SID\sysman\config\emd.properties to emd.properties.orig
2. Update the file $ORACLE_HOME\host_SID\sysman\config\emd.properties, replacing \ with / in the
   following line:
   For example change:
   omsRecvDir=d:\oracle\product\10.2.0\db_1\ukp001_db0\sysman\recv
   to
   omsRecvDir=d:/oracle/product/10.2.0/db_1/ukp001_db0l/sysman/recv
3. Stop and restart the DB Control 

Hdr: 4714774 10.2.0.1 EMCP 10.2.0.1 PRODID-1370 PORTID-912
Abstract: DBCONSOLE DOES NOT WORK HAVING A HOSTNAME STARTING WITH "U"

Problem Description
-------------------
this seems to be releated to bug 3610052.
Having a hostname like u1234, install rdbms10.2.
start dbconsole will take 4-5 min. Even if the service is shown as started,
dbconsole java processes stops and restarts.

Environment Information
-----------------------

Test Case Step-by-Step Instructions
-----------------------------------
1. rename the hostanme to uXYZ....
2. install 10.2 rdbms into its own oraclehome
3. strt listener, create a database using dbca.
4. access the dbconsole page shows: "page cannot be displayed"

Check the dbconsole java process in taksmanager , you will easily see, that
it is coninously restarting. dbconsole service is shown as "started".

Test Case Location
------------------
not needed easy to reproduce

Diagnostic Analysis
-------------------
run the java command from file emd.nohup file, this will show something like:

java...Exception...
Malformed \uxxxx encoding.

and a java stack trace.

I assume that this problem will always happen, having a "\u" in the path for
the dbconsole

Performance
-----------

NLS Information
---------------

Patches
-------

Log Files Location
------------------

Reproducibility
---------------
everytime, only on windows platforms:
w2k w2003, windowsXP

URL
---
not needed

Did you test with the latest version?
-------------------------------------
yes

Available Workarounds
---------------------
Change the hostname from uXYZ to UXYZ,
do
set ORACLE_SID=
emca -config dbcontrol db -reposrecreate 

Related Bugs
------------
3610052

Hdr: 6313490 10.2.0.1.0 CONSOLE 10.2.0.1.0 PRODID-1370 PORTID-215 4714774
Abstract: 10.2.0.1DBCONSOLE IN A DB CREATED FROM IAS INSTALL FAILS TO START

Problem Description
-------------------
The dbconsole 10.2.0.1.0 creating is failing while starting the dbconsole
dbconsole in a db created from iAS install.

as per the logs:-

SEVERE: Error starting Database Control
    Refer to the log file at

D:\oracle\product\10.2.0\db_1\cfgtoollogs\emca\mmds\emca_2007-06-19_09-47-70-A
M.log for more details.
    Jun 19, 2007 9:58:17 AM oracle.sysman.emcp.EMConfig perform
    CONFIG: Stack Trace:
    oracle.sysman.emcp.exception.EMConfigException: Error starting Database
Control at
oracle.sysman.emcp.EMDBPostConfig.performConfiguration(EMDBPostConfig.java:569
) at oracle.sysman.emcp.EMDBPostConfig.invoke

Environment Information
-----------------------
dbconsole in a db created from iAS install.
db console 10.2.0.1.0 on Microsoft Windows Server 2003
The 10g application server version  is 10.1.2.0.2. to include January 2007
database and application server CPUs.

Applied the  database CPU patch and after that reconfigured the Dbconsole .
Now it doesn't get configured .Db console generating core dump

Test Case Step-by-Step Instructions
-----------------------------------

Test Case Location
------------------

Diagnostic Analysis
-------------------
Stopped the application server service and then started the dbcosnole .But
still getting the error

D:\oracle\product\10.2.0\db_1\BIN>emctl status dbconsole
    Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
    Oracle Enterprise Manager 10g is not running.
    ------------------------------------------------------------------
    Logs are generated in directory 

Asked him to move the file from Oracle_Home>/_/emctl.pid file
to another location . Now the dbconsole is started:-

>emctl start dbconsole
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Starting Oracle Enterprise Manager 10g Database Control ...The
OracleDBConsoleMM
DS service is starting...................The OracleDBConsoleMMDS service was
started successfully.

D:\oracle\product\10.2.0\db_1\BIN>emctl upload
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet..

Asked him to Edit the file
ORACLE_HOME>\oc4j\j2ee\home\applications\dms\WEB-INF\web.xml

Still the dbconsole is not getting started .

可能是因为Windows上主机名以”U”字母开头后导致相关路径发生了转义。

可以通过在cmd中输入hostname来确认Windows上的主机名。

给出了2种解决方案:

1.升级到10.2.0.2以上版本,目前推荐的是10.2.0.5 Patch set; 版本10.2.0.1到10.2.0.3上的bug太多了..

2.修改$ORACLE_HOME\host_SID\sysman\config\emd.properties中的内容,将omsRecvDir变更改成使用斜杠而非反斜杠的路径形式,并重启EM.

沪公网安备 31010802001379号

TEL/電話+86 13764045638
Email service@parnassusdata.com
QQ 47079569