Patching issue in Grid Control and the FIX

Oracle Support recently advised us to apply a number of patches to our Grid Control 

installation, as per a Metalink Note 427577.1 (OMS Patches required for setting up 

Provisioning and Patching in 10.2.0.3 GC / 10.2.0.4 GC).  The main purpose was to 

improve the Provisioning and Patching facilities in Grid Control.

However, we had an issue with applying one particular patch 7214697 from this list. 

This was giving an error towards the end in the opatch apply. The error occurred 

when PARdeploying prereqs.par :

C:\OracleHomes\oms10g\bin\PARDeploy -action deploy -parFile 

C:\OracleHomes\oms10g\sysman\prov\paf\prereqs.par

0 [main] DEBUG oracle.sysman.eml.swlib.DBEntity - SWLIB getRepositoryFormatForRevision

...

...

78 [main] DEBUG oracle.sysman.eml.swlib.DBEntity - SWLIB getDocuments

78 [main] DEBUG oracle.sysman.eml.swlib.DBEntity - SWLIB getRepositoryFormatForParameters

125 [main] DEBUG oracle.sysman.eml.swlib.DBEntity - SWLIB error=An entity with

the specified name already exists

Even though there was an error during the patch application, this patch showed up in the 

"opatch lsinventory". But then, when Grid Control was started, we were not even 

able to enter the patching screen.  As soon as we went to the "Deployments" tab 

and clicked on "Patching through Deployment Procedures", we got :

"Internal Error has occured. Check the log file for details."

We got exactly the same error when we clicked on "Deployment Procedures",

"Patch Procedures" etc.

The patch, even when rollbacked, failed with exactly the same error.

It disappeared from the inventory, but the internal error was still present

in the patching screens in Grid Control.

Finally, the fix to this issue turned out to be a very simple one:

cd %OMS_ORACLE_HOME%\patches\p7214697_10204_GENERIC\7214697\custom\scripts 

Edit post.bat and change the line: 

%ORACLE_HOME%\bin\PARDeploy -action deploy -parFile %ORACLE_HOME%\sysman\prov\paf\prereqs.par 

to 

%ORACLE_HOME%\bin\PARDeploy -action -force deploy -parFile %ORACLE_HOME%\sysman\prov\paf\prereqs.par

 

This prevents the error when applying the patch, by forcing the PARDeploy of prereqs.par. 

After the patch is successfully applied without any errors, the Grid Control patch deployment procedures work ok.

By the way this seems to be a Windows Grid Control issue. 

Merry X'Mas and Happy New Year

We are getting close to that time of year again, when all the supermarkets play X'Mas tunes and everyone is thinking of holidays and gifts.

Well, let us wish all our readers a very Merry X'Mas and a Very Happy New Year,
a peaceful and prosperous time with your families, and wishing peace for the whole world
in the New Year. May it be so as we wish.

Treat EM as Production

One word of advice is to always treat your EM install as production and not superfluous to production. By this I mean have production standards, such as scheduled backups, server security, and so on. Also have a test and development site for EM. This is specially important for testing patches - dont apply patches straight on production, apply them to your test site first.
If everything works as expected then apply on the production EM site.

How to resolve a Windows locking issue that prevents patching via Grid Control

We ran into a peculiar issue when patching an 11g Windows database using Grid Control, and I want to describe the peculiar way we used to resolve this issue in Windows.

As a background, Grid Control enables the DBA to patch an Oracle Database using the Provisioning Pack. Oracle supplies deployment procedures that can be executed out of the box, or copied and modified, to patch a single database, or a RAC database, and so on. The deployment procedure, when it reaches the step of actually patching the database, ultimately uses opatch at the database server level. 

Now, the problem was, that when trying to patch an 11g database on Windows using Grid control,  opatch was failing since the "oci.dll" file in the 11g db home was locked. 

We first used a third party utility ProcessExplorer.zip to find the file, and this found it was locked by windows process svchost (Service Host). (There are multiple svchost processes running. To see which copy of svchost is running what service, use "tasklist /svc" in a command window.)

As per a hint found in Google search, we set the windows service "Distributed Link Tracking  Client" to manual and stopped this service. This made no difference and the file stayed locked.  As per another hint found in Google search, we renamed oci file to oci.bak then killed the scvhost process using process explorer. The scvhost process restarts automatically but cant find the oci file. But this effected the working of Grid Control (because the listner cannot really work without oci.dll being used) so do not use this renaming technique.

We then found that when the listener process is stopped, and scvhost is killed, scvhost restarts without the lock. So the issue obviously happens because during the patch process, when the listener is stopped, the lock is not released by scvhost. This was the main cause of the issue.

We then edited a custom deployment procedure and enabled the extra step to stop all Windows services in that home. Normally this step is not enabled. But this had no effect on the error.

The solution that ultimately worked was that we used unlocker (another third-party utility) to unlock oci.dll just after the databases and listeners were stopped by the deployment procedure. Make sure you exit process manager before doing the unlocking, because process manager itself locks other dlls. So the steps are:

1. Start the patching process.
   
2. Dont start the third-party process manager at all. 
   
3. In windows explorer go to the 11g Oracle Home/bin and find oci.dll. 
   
4. Wait till the grid control patching reaches the step of stopping databases.
   
5. Right click on oci.dll and unlock the lock of scvhost on this file, using third-party unlocker. 
   
6. Patching then continues successfully since the file is not locked any more.
   

Managing SAP R/3 with Enterprise Manager

Managing SAP R/3 with Enterprise Manager - would you have thought it possible? Doubtful to say the least. 

I checked with one of the top Oracle experts here in Singapore, Alivin, who surprised me by saying SAP R/3 can definitely be managed using a plug-in to Enterprise Manager known as NimBUS SAP Plug-In. This is a third-party plugin. 

This is described at http://www.nimsoft.com/oem/ if you are using SAP and want to use Enterprise Manager to

manage and monitor SAP. 

Grid Control Sessions in DBA Round Tables by Oracle OTN

Fellow DBAs,

The Oracle Developer Program team (which is the same as the Oracle Technical Network ie. OTN) is holding an Oracle DBA Round Table in Kuala Lumpur on 25th November and in Singapore on 27 November. Oracle DBA Round Table is a half-day session and the purpose of the round table is to create a sense of community, share experiences and provide DBAs with the opportunity to network with Oracle experts, Oracle ACE/ACE Directors and other DBAs.

As the only Oracle ACE Director in this region, I will be conducting a session at both events on "Patching Databases with Grid Control". The scheduled agenda is as follows:

Kuala Lumpur, 25 November
Time	Activity	Speaker
10.00a.m.	Introduction	Blair Layton
10.15a.m.	Flashback Data Archive	Rajender Singh, Oracle ACE
11.15a.m.	Backup and Recovery with Oracle Database
12.15p.m.	Lunch
1.00p.m.	Patching Oracle Databases via Grid Control	Porus Homi Havewala, Oracle ACE Director, S&I Systems Pte Ltd
2.00p.m.	Customer Presentation
3.00p.m.	Break & General Discussion	Blair Layton
4.00p.m.	Closing Remark	Blair Layton
Singapore, 27 November
Time	Activity	Speaker
10.00a.m.	Introduction	Blair Layton
10.15a.m.	Creating an Ideal Database Infrastructure	Ravishankar Buddha
11.15a.m.	Backup and Recovery with Oracle Database
12.15p.m.	Lunch
1.00p.m.	Patching Oracle Databases via Grid Control	Porus Homi Havewala, Oracle ACE Director, S&I Systems Pte Ltd
2.00p.m.	Customer Presentation - Olam 11g Reference
3.00p.m.	Break & General Discussion	Blair Layton
4.00p.m.	Closing Remark	Blair Layton

The events will be held in the Oracle offices:

Oracle KL address:
Langkawi Room
Level 38 Menara Citibank, 165 Jalan Ampang, 50450 Kuala Lumpur, Malaysia

Oracle Singapore address:
Hong Kong Room
6 Temasek Boulevard #18-01 Suntec Tower 4 Singapore 038986

Please let me know if any of you will be attending the round tables in either Kuala Lumpur or Singapore, and I will let Oracle know. You can write to oracleacedirector@si-asia.com , thanks.

Use Grid Control to Secure Backup your database to the Internet Cloud!

This is more great thought and technical leadership from Oracle. In a recent press release, read the part about secure cloud-based backup ie, backing up your production database onto an internet backup cloud (from Amazon) and you will know what I mean. Absolutely innovative and mind-boggling, this is the Oracle Secure Backup Cloud Module. You can use Grid Control to backup to the cloud using RMAN and recover your database too from the cloud.

The full press release is on :

http://www.oracle.com/us/corporate/press/017521_EN?msgid=7004671

This mentions:

Oracle is also introducing a secure Cloud-based backup solution. Oracle Secure Backup Cloud Module, based on Oracle’s premier tape backup management software, Oracle Secure Backup, enables customers to use the Amazon Simple Storage Service (Amazon S3) as their database backup destination. Cloud-based backups offer reliability and virtually unlimited capacity, available on-demand and with no up-front capital expenditure. The Oracle Secure Backup Cloud Module also enables encrypted data backups to help ensure complete privacy in the Cloud environment. It’s fully integrated with Oracle Recovery Manager and Oracle Enterprise Manager, providing users with familiar interfaces for Cloud-based backups.

Changing Error handling in Patch deployment procedures in Grid Control

During Patch application via Grid Control (Provisioning Pack), we can auto bring up the database up again if the patch application fails for any reason (like files in use). This was the main concern expressed recently by a group of DBAs, that if the patch application fails, the database was not being brought up. The way we fix this is as follows.

Oracle supplies deployment proceedures out of the box, such as “Patch Oracle Database”. We simply make a copy of this procedure and make our changes.

In the Grid Control Console, we go to the “Deployments” tab then select “Patching through Deployment procedures”. Now select the “Patch Oracle database”  procedure and click on “Create Like”. This is our copy we can work on.

In our copy, scroll down to the “Apply Patches” step and in its “Error Handling Mode” which is the last column, choose “contnue on error” rather than the default “Inherit (Stop on error)”. This will allow the procedure to continue even if the patch application fails, and bring up the database.

This proves that Oracle has supplied a lot of capabilities in the Provisioning pack of Grid Control.

Grid Control Workshops successful

Oracle DBAs / Database Managers / Senior Management, 

The latest news is that the series of workshops we had in Singapore have been highly 

successful with a huge number of attendants in the second and third in the series.

The fourth week is this week.

A number of Senior Dbas in my LinkedIn group of 1060+ Dbas have looked at the contents 

and said it looks like a "great" workshop series with "excellent" content, the series 

focuses on practical aspects of easing a DBA's life via Enterprise Manager Grid Control, 

such as setting up RMAN backups, applying database patches, and improving performance, 

all with the automation and ease of Grid Control.  

We can also show a combined version of the entire workshop at client premises.

Here is a recap of what we offered:

If you are based in Singapore, please feel free to attend the Oracle GRID CONTROL workshop we will be running at our Office @ 2.30 to 5pm Every Thursday starting 25th Sept. These workshops are totally free. Snacks and drinks will be provided.

S&I Oracle Core Technology Workshop Series

25th Sept : 2 hour session on Grid Control architecture and day to day Database Management using Grid Control

· Explain Grid Control architecture and scalability for multiple target monitioring and management
· Explain Grid Control multiple-level security via target groups, administrators, roles, privileges
· Overview of Grid Control Console and access to target information such as database versions, server configurations, server performance, alerts and patch notifications
· Overview of Database Home Page
· Day to day Database Administration using Grid Control
· Execution of server scripts on multiple hosts or sql scripts on multiple databases

2nd Oct: 2 hour session on Database Performance monitoring and Performance improvement using Grid Control

· Overview of Database Performance page
· Drilldown to specific performance issues and top events
· Generate stress load on the database using Swinggbench and demonstrate ADDM capabilities
· Demonstrate Advisor Central capabilites eg. Sql Tuning Advisor, Sql tuning sets

9th Oct: 2 hour session on Database RMAN Backups using Grid Control

· Overview of Database Maintenance page
· Configure RMAN settings for a database eg. controlfile autobackup, retention policy
· Setup and schedule full RMAN backups for a database
· Setup and schedule incremental RMAN backups for a database
· Setup and schedule image copy refreshes for a database

16th Oct: 2 hour session on Database Patch application using Grid Control

· Overview of Patch Deployments using Grid Control
· Critical patch advisory for databases on Grid Control
· Connect to Metalink and find a patch, download to staging area on oms server,
· After testing, deploy patch on multiple databases in an automated “lights-out” operation

Our office address is curently:

S&I Systems Ptd Ltd
298 Tiong Bahru Road
#12-01/06 Central Plaza
Singapore 168730
http://www.si-asia.com/

Please register (free) by emailing your name, title, company and contact information to
OracleAceDirector@si-asia.com

Look forward to seeing you in the workshop.

Installing Grid Control repository on RAC with ASM

Praveen Chandran asked: I am facing a typical problem in my Grid setup. I am installing Grid on my rac database, which is under asm.I selected the option to use "Existing Database". After providing the correct hostname, port and service, i have given the datafile path for the managemnt respository tablespace like '+DATA/mydata/mgmt1.dbf' , its giving me an error , "Do not have sufficient privilegae for creating tablespace on the specfied location" But i am able to create a tablespace from sqlplus, only OUI is telling about a problem. I could not find any problem in log files with logging level to trace and debug enabled. strace is alos not giving any input. BTw i am using Oracle Enterprise Linux 4.5 for my test and using external harddisk as a shared storage Please let me know , if you find any similar kind of issues

My reply: Praveen, I have never installed the repository onto an existing RAC database which is on top of it using ASM. You have introduced too many complexities into the picture. Can I suggest you keep it simple. Regarding the management repository tablespace where you have an issue, first simply create it as a non-ASM tablespace. Once that succeeds, later on, move it to ASM. That may be the trick required.

Agent issue - target not seen

If the agent installation has succeeded but database target not seen, 

issue still not solved. kindly advice please.
Regards,

sawant

Grid Control manages the Lot

Ray Payne,  Principal Architect, Oracle Infrastructure at The Johns Hopkins Applied Physics Laboratory, mentioned in an answer to a LinkedIn question: 

We are finding that Oracle Enterprise Manager Grid Control and various management packs and plug-ins are pretty comprehensive. We are using for Database, Operating System, Middleware, Applications (EBusiness Suite and Peoplesoft), as well as our EMC Storage and F5 BigIP load balancers. Comprehensive, and all in one tool supported by one vendor. 

I fully support and second what Ray has said. Grid Control indeed manages the lot.

agent upload after OMS hostname change

hi,
after change of OMS hostname like hsd01 to fcs01 ,target agent could not able to upload all
xml file and show error as upload error because hostname is stored in various file like emd.properties etc.
to avoid wastage of effort its better to reinstalled agent and see all XML file uploaded smoothly.

target not seen

hi,
agent has been installed successfully on AIX but database(target) could not automatically detect. when i tried to add manually, it gives error as hostname_svc file not available.
How to get the right result?

best regards
rvsawant

Great Future for Grid Control

Oracle Enterprise Manager is perfectly positioned to become the #1 in Systems and Applications Management, as per IDC and Forrester Reports.  This was told to us by Oracle in one of their workshops in Singapore in the past few weeks. Sounds like there is a Great Future ahead for Grid Control.

Bare Metal Provisioning

Just saw a blog by Johan Louwers in which he speaks of moving to Oracle

Enterprise Linux by kick-starting from an image of Linux on the network:

http://johanlouwers.blogspot.com/2008/08/kickstart-oracle-unbreakable-linux.html

Good decision to move to Oracle Enterprise Linux. I wonder though, did he consider using the Provisioning Pack of Oracle Enterprise Manager Grid Control, and the "bare metal provisoning" feature which can create Linux servers out of bare metal based on a Gold image. That would have saved him from re-inventing manually what Oracle has already done with Grid Control.

Free Oracle GRID CONTROL workshops in Singapore

Oracle DBAs / Database Managers / Senior Management,

If you are based in Singapore, please feel free to attend the Oracle GRID CONTROL workshop we will be running at our Office @ 2.30 to 5pm Every Thursday starting 25th Sept. These workshops are totally free. Snacks and drinks will be provided.

S&I Oracle Core Technology Workshop Series

25th Sept : 2 hour session on Grid Control architecture and day to day Database Management using Grid Control

· Explain Grid Control architecture and scalability for multiple target monitioring and management
· Explain Grid Control multiple-level security via target groups, administrators, roles, privileges
· Overview of Grid Control Console and access to target information such as database versions, server configurations, server performance, alerts and patch notifications
· Overview of Database Home Page
· Day to day Database Administration using Grid Control
· Execution of server scripts on multiple hosts or sql scripts on multiple databases

2nd Oct: 2 hour session on Database Performance monitoring and Performance improvement using Grid Control

· Overview of Database Performance page
· Drilldown to specific performance issues and top events
· Generate stress load on the database using Swinggbench and demonstrate ADDM capabilities
· Demonstrate Advisor Central capabilites eg. Sql Tuning Advisor, Sql tuning sets

9th Oct: 2 hour session on Database RMAN Backups using Grid Control

· Overview of Database Maintenance page
· Configure RMAN settings for a database eg. controlfile autobackup, retention policy
· Setup and schedule full RMAN backups for a database
· Setup and schedule incremental RMAN backups for a database
· Setup and schedule image copy refreshes for a database

16th Oct: 2 hour session on Database Patch application using Grid Control

· Overview of Patch Deployments using Grid Control
· Critical patch advisory for databases on Grid Control
· Connect to Metalink and find a patch, download to staging area on oms server,
· After testing, deploy patch on multiple databases in an automated “lights-out” operation

Our office address is curently:

S&I Systems Ptd Ltd
298 Tiong Bahru Road
#12-01/06 Central Plaza
Singapore 168730
http://www.si-asia.com/

Please register (free) by emailing your name, title, company and contact information to
OracleAceDirector@si-asia.com

Look forward to seeing you in the workshop.

Question on Sybase plugin for Grid Control

Rajesh asked this question:

===========================================
Hi, I'm trying to configure Sybase ASE plug-in for OEM Grid control. Have downloaded the plug-in and imported into OEM Grid (in setup ---> Management Plug-in's section). The question is about the below section of the pre-requisites which says the apart from OEM Grid Control, we also need to have the agent for windows.
*****************************
-Oracle Management agent for Windows 10.2.0.1 or higherYou can install the agent on the same computer as Sybase Adaptive Server (referred to as local agent monitoring), or you can install the agent on a different computer from Sybase Adaptive Server (referred to as remote agent monitoring).
*****************************
Does it mean do we need to install the agent software on the Sybase host (our Sybase ASE runs on Linux) and if so, do we need to install Agent software for Linux? Do we need to install that as Sybase user or do we need to create the Oracle account on the server? Documentation is not clear on this area. Can someone who installed and configured the plug-in for ASE explain this? Thanks, Rajesh
===========================================

Rajesh, you need to install the Oracle EM10g Agent software on your target Sybase linux server. Install it as the oracle user in its own home. The agent is required for all targets, treat that as a general rule. All the Best.

Patch Deployment via Grid Control

A large DBA team of a big corporate, used to doing everything manually for 100s of databases, were very impressed this week when I showed them how EM10g Grid Control can assist them in patch deployment.

This requires the Provisioning Pack, which is a licensable pack for Grid Control.

The patch deployment procedure in Grid Control is quite mature - it allows the DBA to search for patches for any version of any Oracle software from Metalink, download the patch to a staging area on the Grid Control management server, and then apply the patch using a supplied deployment procedure.

The patch can be scheduled to be applied at any time using the Grid Control scheduler, it automatically starts a blackout for the database (so no alerts are generated), shuts it down, applies the patch, and then starts the database again.

The DBAs felt this would save them a lot of manual steps for patches that have to be applied to 100s of databases. For starters, think how much time it would take to ftp / scp the patch to 100s of servers. In the case of Grid Control, the patch is downloaded once and stored once, on the central EM Management server in the patch staging area.

8i Databases managed with EM10g Grid Control

With respect to the question a friend using 8i on VMS asked me:

“Can 8i Databases be managed with EM10g Grid Control?”

The answer is definitely yes. Oracle have published a white paper to that respect.

http://www.oracle.com/technology/products/oem/pdf/em_gc_8i-9i_4.pdf
Managing Oracle 8i and Oracle 9i databases with Oracle Enterprise Manaqer
10g Grid Control

The only caveat is that the OS must be certified for the 10g Grid Control agent.

As per Metalink Note: 412431.1 Oracle Enterprise Manager 10g Grid Control Certification Checker, Open-VMS or VMS is NOT supported. Therefore the main issue is not the 8i version but the OS. Only the following are supported:

Linux x86
Linux x64 (AMD64/EM64T)
Windows x86
Solaris SPARC (64-BIT)
HP-UX Itanium
HP-UX PA-RISC (64-BIT)
AIX 5L (64-BIT)

In any case, Oracle does not support the 8i database anymore except if you have an extended support licence. It may be time for you to move from OpenVMS to Oracle Enterprise Linux (free distribution, lower support cost) from Oracle and have one point of support for both the Oracle OS and the Oracle database. Also upgrade the database at the same time to 11g.

Grid Control or 3rd Party products

To a friend, I had this to say:

I would prefer Oracle products like Grid Control dealing with the Oracle
database rather than 3rd party products, especially when tweaking is involved.
No one would know Oracle better than Oracle.

In fact, even reading by 3rd party products may be unsupported. As an
example, take the case of 3rd party products that read Oracle redo logs to
replicate data and dont go through the Logminer interface. Have a loook at
Oracle Metalink Note:97080.1 which says that this is not a supported interface
and the data in the destination database cannot be assured in the case of
unsupported proprietarty interfaces to the log files since Oracle can and does
change the data in the log files when fixing bugs. In such a case, 3rd party
products are not aware of such internal changes in their proprietary interfaces.

Likewise, even if a performance Guru were to write a tool to interact with the
Oracle database, I would still prefer Oracle Grid Control. Why?

Because Oracle know what they are doing. Because they are the ones who have created the database and all the tuning features in 9i, 10g and 11g. Because of the world-class Enterprise level support for the tool, available around the world and around the clock (how many companies can equal Oracle Worldwide support's strengths?). Because of the knowledge that Grid Control is already better (SLAs, Dashboards, Quality Control. etc) than previous releases and will be enhanced more and more by Oracle in future releases.

And because of the knowledge that Oracle will be there for us much more than any other 3rd party company or perfornance gurus.

Just my view, but it is echoed by a lot of other DBAs.

Why Sysadmins are happy with Grid Control

A friend said Sysadmins are happy with Grid Control when they see that Grid Control can capture and store a history of CPU and Memory utilization for each host server. I replied:

Thats a good one mate. The sysadmins would definitely be happy with that
history. Also all that is stored in the Central Oracle EM Repository instead of
files all over the place on different host servers.

Its just a matter of knowing what is possible. I have shown a lot of managers/techos what can be done with Grid Control with their Oracle databases, and all of them have been pleasantly surprised if not outright amazed. 99% expressed an interest to use Grid Control to simply make their lives easier.

Jaguars and Grid Control

A friend was talking about licences of Grid Control packs. I told him:

Mate, prices are always negotiable with Oracle. And besides Oracle openly
publishes its list prices. The packs are only a fraction of the cost of the
Enterprise Edition software in any case, Besides, you pay for what you get. You
have the Oracle database, and you need the best management tool in the world to
handle your database.

Oracle know best about their database themselves, why would you buy a tool from some other company to manage it? Get the management tool from Oracle. If you have a Jaguar (owned by Tata), and all cars have electronics and computers nowadays, would you get the management software to run the car from a totally different car manufacturer, or from Jaguar itself?

Good Discussion

There is a good discussion about Grid Control going on in the Oracle Wiki at
http://wiki.oracle.com/thread/1000866/Why+companies+are+not+using+EM+Grid+Control ,
Please have a look.

Atom Feed for Blog Posts

Ilmar Kerm said:
"Thank you for this interesting and useful blog.
But I have one technical request, please make the blog posts also available as a (Atom) feed. Currently all the feed links I found on this site point to the podcast. Thank you."

Ok Ilmar, thanks for the kind words. I have made an atom feed available for the posts. All the Best.

Presentation Impressions

Today I had a presentation to the managers and IT / Finance directors of a large insurance company, and showed them Grid Control and all its capabilites. My manager was also present and noted that the seniormost director wrote down 2 pages of notes from my presentation, and seemed very excited by the possiblities of Grid Control.

This I have seen time and again - when demonstrated properly, Grid Control never fails to impress top management with what it can do.

Grid Control Levels of Security

A large site told me that their engineering team is concerned that once you login to Grid Control, "you are able to observe and manage all databases in the company and that is a security risk".

We need to make these people aware that there are two levels of security in Grid Control. First you login to the Grid Console using either the "sysman" login (not recommended for large sites, the sysman schema is the owner of the Grid Control repository) or a Grid Control Administrator login (recommended), and then you must also logon to the database.

So it is not possible that a login at the console is able to manage all databases in the whole compnay, unless some has saved the password for all databases when logging on. The rule of course is, never save the database password for the Dba user - always force a logon.

The second point to note is, we are able to create Target Groups in Grid control. Each database, listener, and host is a target. So we can group together targets and then assign to Grid Control Administrators that we can create easily on the console.

The DBA then logins to Grid Control using the partitcular console administrator login that is assigned to him/her. Such an admin is not sysman, so he/she can only see and manage the targets in the target group that is assigned to that console administrator. So it is not possible that a console login can even see all targets in the company, unless of course the login is the sysman, or the login is a console admin that has been purposely assigned ALL targets.

This I have been explaining to many clients in the past 3-4 years as a consultant, and even before that to project teams and the security team in my past companies.

You have the security capability, and you should use it. Take the case of a database - you can easily have all schemas assigned the DBA role, and I have seen that done by many development outfits just as a shortcut. Or you can have proper role-level security set up at the database level. So, just because every schema has been set up by a developer as a DBA, does it mean there is no security in the database? The truth is, there is enough security, and we should know the way to use it.

Grid Control Live Demos in Oracle Booth

Just manned the Oracle booth yesterday at a Computerworld exhibtiion, and had a running demonstration of EM10g Grid Control 10.2.0.4 on Windows, managing multiple 10g and 11g databases. I had the Grid Control Home page open (also had 11g Database Control up on
another browser window).

There were a number of attendees who had lots of databases (some in the 1000s ) and were doing everything manually. When I showed them what they could do with Grid Control:

1. See at a glance on the home page the up or down status of targets, which versions of databases they had, how many numbers of each versions, and the same for the server OS,
2. On the server targers, see the server performance, and the configuration of each server for eg. which OS packages loaded, and compare configurations of servers to find what missing OS package or OS patch is causing an issue,
3. On the database targets, on the database home page, see at a glance the GB size and the general health of the database, then see the database performance on the performance page with the ability to drill down to Sql statements having the biggest hit on performance, and on the other database pages, the ability to perform DBA tasks on database objects, and the ability to setup and schedule RMAN backups without using unix scripts or cron, the ability to setup Dataguard DR databases, and the ability to clone entire Oracle databases and even entire Oracle Homes, and also the ability to download patches from Metalink and apply them to multiple databases,

And they were convinced Grid Control was very powerful and the best tool to work with Oracle databases.

Manually manage 1000 databases or use Grid Control

Just met a friend whose 18 member team is managing more than 1000 Oracle databases,
manually, without Grid Control. Their main issue is patch management since they patch
more than 400 Oracle homes regularly with the quaterly critical patch updates (CPUs) from
Oracle, and they do that by hand.

What is the solution? Use Grid Control in a central location, put agents on all database servers,
and with the Provisoning Pack, they will be able to download patches from Oracle Metalink to a
staging area, validate them and apply them. And the other management features of Grid Control
will save a lot of their DBA activity time.

Blogger Credential

Oracle USA informed me today that I have been approved to receive the Blogger Credential at Oracle Openworld in addition to my ACE status. As per Oracle themselves, the blogger credential is very similar to having a press pass -- it is a press-like credential.

Thanks to our Enterprise Manager Grid Control Blog.

Confirmed about Grid Control and EE

Confirmed with Oracle, Grid Control (base software) is only available free with EE and not with SE. The license for using the EM Packs such as Diagnostics, Tuning, etc. is payable to Oracle.

Benefits of Oracle Database Enterprise Edition for Grid Control

Zaffer Khan (DBA, ISCO, Kuwait) wrote:
"This blog is promising in elaboration of EM for all the DBAs around the world. Thanks Porus for such an effort."

Zaffer, thanks for the thanks.

"1 query!!! We are to migrate to 10gr2 from 9i Standard Edition License. Can I use Grid Control Feature under Oracle DB Standard Edition License?"

Regarding your query, the Oracle Price List supplement states that Enterprise Manager is included in Oracle Standard Edition, but remember Enterprise Manager can mean Database Control (for managing one database) as well as Grid Control (for multiple databases).

I feel this refers to Database control. Why? Because Grid Control has built-in partitions when it is installed, and you can't use partitioning without the Enterprise Edition (EE), so I feel Grid Control requires the Enterprise Edition and not the Standard Edition.

Also, even if you can use Grid Control with Enterprise Edition, remember to pay a licence to Oracle if you use any of the Management Packs in production/test, such as the Diagnostic, Tuning packs etc.

In all I would recommend to speak with your Oracle representative for your company on licencing, since he/she would give you the 100% correct picture, we can only guess. Please do this before using Grid Control in production/test.

There are various other benefits of moving to EE. You should convince your management of upgrading your licence to EE. I have listed a few below.

EE Includes:

<> Normal Dataguard
= Standby database for purpose of:
+ Disaster Recovery
+ High Availability
= Can be opened for reporting purposes for some window of time.
= Requires EE licence on standby server.

<> Advanced Backup and Recovery via these RMAN features:
= Incremental Backups – very important to avoid full backups each day.
= Block Level Media Recovery.
= Trial Recovery.

<> Parallel Query
= Ability to break large queries into chunks handled by multiple processes.
= Very important for large tables (more than 1GB in size).

<> Automatic Storage Managmeent (ASM)
= Manages storage as file system and volume manager.
= No need to buy 3rd party file system and volume management software.
= Automatic I/O balancing and hot spot fixing, no other volume mgr can do so automatically

<> Streams
= Oracle’s new method of replication between 2 databases.
= Can be 2-way replication (multimaster).
= Multi-master must be programmed to resolve conflicts, not out-of-box.

Added interesting blog list

Added other interesting blogs, especially the Application Managment Blog,
and the OTN tech blog, in a blog list to this blog.

If any of you have a blog with some discussion on Grid Control going on,
let me know and I will add it to the blog list.

Mentioned in OTN Techblast

Our Grid Control blog, the first of its kind in the world, has been mentioned in the Oracle OTN Techblast (July 2008). The Techblast is an immensely popular mass monthly email communication sent out by Oracle to tens of thousands of the Oracle community worldwide, pointing them to technical information and community events.

The Techblast contains this under the section "Oracle Ace Tracker":

The Grid Guy: Oracle ACE (Database) Porus Homi Havewala of S&I systems Singapore has created the first blog dedicated to Grid Control, and is inviting authors to contribute. Based on real life experience in a large corporate, the blog also features a podcast on Grid Control Architecture in large sites.

New in 11g

An interesting new parameter in 11g for the EM Packs:

In Oracle Database 11g, you need to set the initialization parameter 'control_management_pack_access' to disable
or enable Database Diagnostic and Tuning Packs.

These are the presentations for EM in Oracle Open World 2007.

Open World 2007 Enterprise Manager Presentations

Quite informative, especially the RAC deployment presentation:

"Automating Oracle RAC Deployment and Patching
on Dell PowerEdge Servers, using Oracle Enterprise Manager
Provisioning Pack"

Using Oracle Enterprise Linux for Grid Control

Last week a colleague asked me what was the best platform to install Enterprise Manager Grid Control. He wanted to install everything on 1 server since it was only going to manage about 20 targets. Then he suggested he would get a Windows server.

Well, I have personally installed Grid Control on Windows (as also on Solaris, Aix and Red Hat Linux) and it works fine on all these platforms. However, when Oracle supplies Oracle Enterprise Linux as a FREE distribution, the only cost being support (which is less than the support cost of other commerical Linux Distros), then what's the point in getting any other op system?

Just get the Oracle Free Distro of Enterprise Linux (which is equivalent to Red Hat Release 5) and install Grid Control on top of that. You get savings instantly because you are not paying the initial licence fee. And you get ONE support point for both Oracle and Linux, which is Oracle's Worldwide Support.

That got me thinking about why Oracle is giving out Enterprise Linix for free. Personally I feel Oracle is doing this for a very noble reason, to make sure Linux Distros that are certified fully with Oracle, stay very affordable. Oracle is doing this to support the Linux OS in the commerical world, and I feel the Linux community would be grateful.

Interesting links on your Grid Control Home page

How do you find the exact version of Grid Control installed at your site? A lot of clients ask that question.

The answer is quite simple. Go to the Grid Control Home page at your site. There is a link at the bottom of the home page:

About Oracle Enterprise Manager

Click on this link. It takes you to a new page where your installed version of Grid Control is shown. For example, in our case it is:

Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0

There is another interesting link on this page:

Oracle Enterprise Manager 10g Release 4 Grid Control License Information

If you go to this link, you will be taken to a page with a list of all the EM packs available, and a detailed description of each. So if you ever wanted to know what the Data Masking pack is all about, this is where to look.

EM Packs and Plug-Ins

Many people do not realize how powerful Enterprise Manager Grid Control
has become, with the ability to monitor and manage almost anything. This
is done via Enterprise Manager Packs and Plug-Ins.

These are the main licensable packs for Grid Control, ie, using these
packs needs a licence fee, most people pay per processor:

• Diagnostic Pack
• Tuning Pack
• Provisioning Pack
• Change Management Pack
• Configuration Management Pack
• Data Masking Pack

Most of the Dbas using Grid Control may have used the Diagnostic Pack and
Tuning Pack that are the most commonly used packs for performance issues.
Some of these packs can be used for the database, application server as
well as other infrastructure, for eg, the diagnostic pack.

A number of Dbas may also have used the Provisioning pack, which is used for
patch application, Oracle software cloning, RAC provisioning. The Change
management pack has also been used by some Dbas for comparing
schemas, databases, and dictionary baselines.

The Configuration Management Pack and Data Masking Pack are quite new,
we would love to hear from Dbas who have used these and what their
experience is.

Other packs available for Grid Control are:

• Diagnostic Pack for non-Oracle middleware
• Oracle Linux Management Pack
• SOA Managment Pack
• Identity Management Pack
• Service Level Management Pack
• Business Intelligence Management Pack
• Management Pack for E-Business Suite
• Management Pack for Peoplesoft Enterprise
• Management Pack for Siebel

Then there are a host of Plug-Ins for the Netapps Filer, EMC Clarion, EMC Symmetrix DMX, MS Exchange Server, BEA Weblogic, MS Sql Server, IBM Websphere and so on.

Full details are on http://www.oracle.com/technology/products/oem/datasheets.html

Grid Control Architecture Podcast

Friends, Please listen to my first podcast on my real-life experienceswith Grid Control architecture in large sites. Let me know your feedback.

I was searching for blogs on Grid Control in Google search and suddenly realized that this is probably the first and only blog fully dedicated to discussions on Grid Control, so I changed the description of the blog accordingly. Its strange how no one has made a blog on this fascinating technology before.

Santosh, you are quite right, Oracle has definitely improved the database management utilities over the years. I remember Server manager and now look at Grid Control. certainly a world of difference.

Oracle got it right when it moved Enterprise Manager to the current web-based OC4J (Oracle container for Java) and Application server based architecture, which is inherently much more scalable (more on that later). The EM management server is a cut-down version of Oracle Application server in that respect.

You said "I feel very relieved today to have EM for DBA work that can be accessed using a browser only." You are right. Some Dbas who say they are traditional Dbas frequently look down on EM, but the question is how long they can afford to do so, and is it wise to do so? Oracle database facilities are becoming more and more sophisticated. For eg. sure you can create 10g tuning sets in the command line, and run the tuning advisor, and other 10g performance things, but you have to issue an awful lot of complicated commands. Why do that when you can simply and easily use Grid Control? Find the busiest period on your database, drill down to the top statements during that period, select them and create a tuning set, then run the tuning advisor. It is that simple and cuts down on the manual time of typing all those commands.

Also Oracle strongly recommends the use of Grid Control for managing sophisticated systems like RAC, especially for performance. Sure you can have RAC and manage RAC without Grid Control, but would you drive a Ferarri with bullock cart reins? Would you run RAC with just SqlPlus? (I do apologize to the writers of SqlPlus for comparing SqlPlus to bullock cart reins, but that is how we see it, as compared to powerful tools like Grid Control). Management also should be convinced that when they go in for RAC, they should have it managed by Grid Control.

As consultants we frequently strongly recommend this to management of any company who wants to use RAC.

Your analysis of the benefits of using Grid Control over Database control is also spot on. With Grid Control you can manage multiple databases, set up RMAN backups, see all RMAN backup jobs, set up Dataguard, monitor Dataguard and so on, just as you have pointed out.

Thanks Santosh for your excellent feedback on the advantages of Grid Control.
Do keep writing to this blog.

EM Grid Controls Advantages

Hi,

I am using Oracle DB since last 7 years and have worked on versions 7.3, 8iR3, 9iR2 and 10gR2. With every new version the Tools to do DBA work got very much improved.

With no tool in 7.3 to EM Dbcontrol in 10g. When I look back and think how I do DBA activity in 7.3 I feel very relieved today to have EM for DBA work that can be
accessed using a browser only.

But again when I got introduced with 10g DB, I selected DB control over Grid Control,
thinking why to install agent software additional when I have already installed DB
control which got installed with Oracle DB Software! But my decision was wrong. I soon come to realize the power of Grid Control.

We have multiple databases for different purpose. With DB Control, I could manage
all the databases but at what cost?

I have to start DB console of every DB, plus each individual em process consumed
a lot of CPU resource, plus I also have Oracle AS, a separate em of it also. I
really have to consume a lot of memory (of server and myself ). Then I
implemented RMAN backups for all of our DBs. I did it using DB control, but our backup policy was such that I have to schedule backups as per weekdays (eg.
incremental level 0 backup on every Monday, Wednesday,Friday), this I could not
do using DB control, hence I have to write backup script and schedule it from OS
crontab. Then I have to implement Standby Databases and Dataguard for some of
the databases. For 1st DB I did it manually. Oh, my god, It was really a test of your
patience. Anyway I finished it successfully but I could not forget the stress while
doing it. Then It was the time when Oracle consultant (Mr. Porus, one of the author on this blog) suggested to use Grid control.

I implemented it with help of him. After that I did implementation of all the standby
DB creation and Dataguard activity with the help of Grid control. It was very easy
to do the things with Grid, plus the system resources ware used better. With Grid then
I scheduled RMAN backups as per our policy (schedule on weekdays) and I can
also manage AS with it.

I can summarize like this:

1] Dataguard and Scheduleing RMAN backup on weekdays (like crontab in Unix) can
be done with the help of Grid control.

2] No need to start individual DB Controls for DBs and iascontrols for ASs.
3] Can save resources of your system.
4] Many additional features for Administering the DBs and App. Servers.

Please vote in the survey

Readers and Authors, please also vote in the survey I have
put up on this blog ie.

"The best way you have used Grid Control so far is:"

I have put a number of interesting options.

Let me know ideas for other surveys as well.

Invited people to write

In the last 2 days I invited a number of my friends and technical contacts
to write to the blog, especially those who have used Grid Control in some
form or the other.

If any of the numerous Blog readers from the internet also want to become
authors of this site, do add a comment to any of the articles and mention
you want to be an author to this blog, and I will send you an invite.

Once you are an author, you can send an email about Grid Control
directly from your gmail accouunt. Just send your email to :

<<your gmail login>.enterprise-manager@blogger.com>,

It will be published immediately on the Enterprise Manager blog.

The power of EM

Here is another nice post on the power of Enterprise Manager:

"Steps to Fusion - Centralize the Management of Your Applications on
Oracle Enterprise Manager"

http://appmanagementblog.blogspot.com/2008/05/step-to-fusion-centralize-management-of.html

More Reasons to use Em10g

One technical friend wrote to me:

We are planning an OEM deployment to make use of Grid Control across the company...Our plan is to have a single OEM "view" of the company for all Oracle DBAs and will include global server load balancing using Big-IP...

The main reasons for the OEM build are :

1) Provide the DBAs with a single view of the Oracle estate along with
instance management improvements;

2) Provide a management interface for the planned RAC build.

I wrote back:

Your reasons for deploying EM10g are excellent, may I add another reason, to provide senior Management with a view of Oracle licences across the organization, and more importantly, the use or non-use of expensive allocated San storage space. This was a very important reason in a large telecom whereby management was able to save more than a million dollars each year. The telecom had asked Oracle to write a special disk storage usage report. Oracle has since incorporated the report in Release 2 of EM10g so it comes along with the product along with many other reports.

Regards,

Porus.

RMAN and Enterprise Manager

One of the best uses of Enterprise Manager Grid Control is to set up and
schedule daily maintainance tasks, such as RMAN backups. RMAN is
one of the best utilities from Oracle to back up your databases, and it
amazes me that there are some people out there still doing cold backups.
This is one of the oldest techniques of backup, when it was not possible
to do any online (hot) backups, people were forced to do offline (cold)
backups. But when Oracle created online backups (with the alter
tablespace ... backup command), when Oracle created the RMAN
backup utility, then who would still use offline backups?

In any case, coming back to the point, RMAN backups can be set up
very easily using Grid Control. In the past, we used to write unix scripts
to call RMAN, then these scripts had to be installed on each new database
server with modifications, then tested. This normally took about 4 hours
or so for each new database server. But when the same is done using
Grid Control, all we need is the Enterprise Manager Agent on each
server, then we can easily configure and set up and schedule the
RMAN backup for the databases on that server. It would take no
more than 10 minutes or so to do so, which is a big saving in time,
besides the fact that no unix scripts or cron jobs at the unix level
are involved. Grid Control generates the scripts and schedules
the scripts at the times you want, and all logs are accessible
at the Grid Control level.

More on this later. Right now I want to mention my OTN article
"Technical Note: Using Oracle Grid Control with Filer Snapshotting"
which you can read at
http://www.oracle.com/technology/pub/notes/technote-oem-filer.html ,
this explains how you can use Grid Control with Netapps
Filer snapshots to backup the database.

Regards,

Porus.

Introduction and First Post

Just to introduce myself:

I am a Senior Oracle DBA / Database Consultant with broad experience in Oracle technology since 1994 including 7+ years as a Production Dba, 7 years as a Senior Database Consultant, 2.5 years as an Oracle E-Business Technical Dba and Systems Administrator, and approx. 6 years as a Development Dba / Database Designer Modeller (yes, using Oracle Designer).

I am also an Oracle ACE (for those who are not aware, this is an award from Oracle Corporation).

I started using Oracle Enterprise Manager right from the days of Server Manager. Well, used it as far as it was possible in those days, but I was always interested in Gui management of the database. I have seen Enterprise Manager become more and more better over the years, until finally, when Oracle released Enterprise Manager 10g Grid Control and Database control in the early 2000s, it was good enough to deploy at the Enterprise level. At Telstra in Australia, we did just that, we had a central EM10g installation with three load balanced management servers, and this was being used to manage literally 100s of databases and servers (more than 700) all around Australia and managed by tens of database teams.

We were the first in the world to put Grid Control in production. I was also a Beta tester of EM10g Grid Control Release 2 when it came out. I have been an enthusiast of Grid Control ever since, because I discovered it really delivered the goods, in day to day administration, in things like RMAN backups and Dataguard setup and monitoring.

When I was employed at Oracle India, I convinced a number of banks and a large telecom in India to use Grid Control, and they started to use it very effectively.

I thought of setting up this blog so that people can discuss the use of Grid Control. Please go ahead and post,

Regards,

Porus Homi Havewala.