Db2 is running our of shared memory.
The environment: Virtual computers running under VMware ESXi hypervisor version 5.5. Two different virtual machines, each running CentOS 5, db2 express C, and Apache/PHP and vmware tools. The machines each have 8 GB of memory, 4 processors, and a couple of hundred GB of disk space available. Everything is running on the 64 bit versions. The actual hardware for VMware is a dual processor machine with two Xeon E5410 processors, each with four cpu cores and 32 GB of ram.
The difference in the two virtual machines is that one is running db2 version 9.7.1 and the other is running db2 version 10.5.5.
The error occurs while restoring a backup taken on another server to the test server in question. Worse than that, it occurs when restoring a second database after the first restore works OK.
The error message we are getting is this: On the 9.7 system: RESTORE DATABASE XYZZY USER db2inst9 using FROM '/db2home/restore1' INTO XYZZZ REPLACE EXISTING WITHOUT ROLLING FORWARD WITHOUT PROMPTING SQL1084C Shared memory segments cannot be allocated. SQLSTATE=57019
On the 10.5 system: RESTORE DATABASE XYZZY USER db2inst9 using FROM '/db2home/restore1' INTO XYZZZ REPLACE EXISTING WITHOUT ROLLING FORWARD WITHOUT PROMPTING SQL1084C The database manager failed to allocate shared memory because an`operating system kernel memory limit has been reached. SQLSTATE=57019
Everything I have read online says to increas the value of Shared Memory Size in the /etc/sysctl.conf file. However, on these systems it is already set to a value eight times larger than the actual memory size. The command "ipcs -l" give:
[root@cowtest etc]# ipcs -l
------ Shared Memory Limits -------- max number of segments = 4096 max seg size (kbytes) = 67108864 max total shared memory (kbytes) = 17179869184 min seg size (bytes) = 1
------ Semaphore Limits -------- max number of arrays = 1024 max semaphores per array = 250 max semaphores system wide = 256000 max ops per semop call = 32 semaphore max value = 32767
------ Messages: Limits -------- max queues system wide = 2048 max size of message (bytes) = 65536 default max size of queue (bytes) = 65536
[root@cowtest etc]#
the file /etc/sysctl.conf has the following in it
# Controls the maximum size of a message, in bytes kernel.msgmax = 65536
# Controls the maximum shared segment size, in bytes kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages kernel.shmall = 4294967296
These numbers just don't make sense. kernel.shmmax is set to 64 GB, 8 times the actual memory of the machine. the value of "Total Max Shared Memory" from the ipcs command is 16 TB!
I have other machine running on real hardware with only 6 GB of memory on which this works fine. What is so strange about the virtual machines?
I'm posting this to both the db2 and CentOS mailing lists. Sorry for the duplication if you are subscribed to both.
Harold
Attached below is the complete output of the backup.restore job which ran on the db2 10.5 machine. The backup was taken on a db2 9.7 machine, hence the message about database upgraded to current db2 version.
backup.restore 1.0.0 - submitted Mon Mar 30 05:45:01 EDT 2015 on cowtest waiting... 1 05:45:01 waiting... 2 05:46:01 waiting... 3 05:47:01 waiting... 4 05:48:01 waiting... 5 05:49:03 backup.restore - started Mon Mar 30 05:50:03 EDT 2015 on cowtest File to be restored is: /db2home/www/daily/ANIMALS.0.db2inst9.NODE0000.CATN0000.20150330030114.001.bz2 Testing file to be restored for integrity at Mon Mar 30 05:50:03 EDT 2015 unzipping backup file ------------ /db2home/db2inst9/archivelogs/animals ------------ total 0 ------------ /db2home/db2inst9/archivelogs/animals ------------ SQL1064N DB2STOP processing was successful. Waiting for 1 minute... SQL1063N DB2START processing was successful. ATTACH TO db2inst9
Instance Attachment Information
Instance server = DB2/LINUXX8664 10.5.5 Authorization ID = DB2INST9 Local instance alias = DB2INST9
DEACTIVATE DATABASE ANIMALS DB20000I The DEACTIVATE DATABASE command completed successfully.
RESTORE DATABASE ANIMALS USER db2inst9 using FROM '/db2home/restore1' REPLACE EXISTING WITHOUT ROLLING FORWARD WITHOUT PROMPTING SQL2555I The database was restored and then successfully upgraded to the current DB2 release where you issued the RESTORE DATABASE command.
CONNECT to animals user db2inst9 using
Database Connection Information
Database server = DB2/LINUXX8664 10.5.5 SQL authorization ID = DB2INST9 Local database alias = ANIMALS
SELECT count(bhid) from is3.dna
1 ----------- 220
1 record(s) selected.
DETACH DB20000I The DETACH command completed successfully.
DISCONNECT CURRENT DB20000I The SQL DISCONNECT command completed successfully.
ACTIVATE DATABASE ANIMALS DB20000I The ACTIVATE DATABASE command completed successfully.
TERMINATE DB20000I The TERMINATE command completed successfully.
ATTACH TO db2inst9
Instance Attachment Information
Instance server = DB2/LINUXX8664 10.5.5 Authorization ID = DB2INST9 Local instance alias = DB2INST9
RESTORE DATABASE ANIMALS USER db2inst9 using FROM '/db2home/restore1' INTO VERIFY REPLACE EXISTING WITHOUT ROLLING FORWARD WITHOUT PROMPTING SQL1084C The database manager failed to allocate shared memory because an operating system kernel memory limit has been reached. SQLSTATE=57019
connect to verify user db2inst9 using SQL1084C The database manager failed to allocate shared memory because an operating system kernel memory limit has been reached. SQLSTATE=57019
select count(bhid) from is3.dna SQL1024N A database connection does not exist. SQLSTATE=08003
DETACH DB20000I The DETACH command completed successfully.
TERMINATE DB20000I The TERMINATE command completed successfully.
system is cowtest SQL1064N DB2STOP processing was successful. SQL1063N DB2START processing was successful. DB20000I The ACTIVATE DATABASE command completed successfully. ------------ /db2home/db2inst9/archivelogs/20150330 ------------ total 0 ------------ /db2home/db2inst9/archivelogs/20150330 ------------ backup.complete file has already been removed... backup.restore - ended Mon Mar 30 07:44:52 EDT 2015 on cowtest