Db2 is running our of shared memory.
The environment:
Virtual computers running under VMware ESXi hypervisor version 5.5.
Two different virtual machines, each running CentOS 5, db2 express C,
and Apache/PHP and vmware tools. The machines each have 8 GB of memory,
4 processors, and a couple of hundred GB of disk space available.
Everything is running on the 64 bit versions. The actual hardware for VMware
is a dual processor machine with two Xeon E5410 processors, each with four
cpu cores and 32 GB of ram.
The difference in the two virtual machines is that one is running db2
version 9.7.1 and the other is running db2 version 10.5.5.
The error occurs while restoring a backup taken on another server to the test
server in question. Worse than that, it occurs when restoring a second
database after the first restore works OK.
The error message we are getting is this:
On the 9.7 system:
RESTORE DATABASE XYZZY USER db2inst9 using
FROM '/db2home/restore1' INTO XYZZZ REPLACE
EXISTING WITHOUT ROLLING FORWARD WITHOUT PROMPTING
SQL1084C Shared memory segments cannot be allocated. SQLSTATE=57019
On the 10.5 system:
RESTORE DATABASE XYZZY USER db2inst9 using
FROM '/db2home/restore1' INTO XYZZZ REPLACE
EXISTING WITHOUT ROLLING FORWARD WITHOUT PROMPTING
SQL1084C The database manager failed to allocate
shared memory because an`operating system kernel
memory limit has been reached. SQLSTATE=57019
Everything I have read online says to increas the value of Shared Memory Size
in the /etc/sysctl.conf file. However, on these systems it is already set to
a value eight times larger than the actual memory size. The command "ipcs -l"
give:
[root at cowtest etc]# ipcs -l
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 67108864
max total shared memory (kbytes) = 17179869184
min seg size (bytes) = 1
------ Semaphore Limits --------
max number of arrays = 1024
max semaphores per array = 250
max semaphores system wide = 256000
max ops per semop call = 32
semaphore max value = 32767
------ Messages: Limits --------
max queues system wide = 2048
max size of message (bytes) = 65536
default max size of queue (bytes) = 65536
[root at cowtest etc]#
the file /etc/sysctl.conf has the following in it
# Controls the maximum size of a message, in bytes
kernel.msgmax = 65536
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
These numbers just don't make sense. kernel.shmmax is set to 64 GB, 8 times
the actual memory of the machine. the value of "Total Max Shared Memory"
from the ipcs command is 16 TB!
I have other machine running on real hardware with only 6 GB of memory on
which this works fine. What is so strange about the virtual machines?
I'm posting this to both the db2 and CentOS mailing lists. Sorry for the
duplication if you are subscribed to both.
Harold
Attached below is the complete output of the backup.restore job which ran on
the db2 10.5 machine. The backup was taken on a db2 9.7 machine, hence the
message about database upgraded to current db2 version.
backup.restore 1.0.0 - submitted Mon Mar 30 05:45:01 EDT 2015 on cowtest
waiting... 1 05:45:01
waiting... 2 05:46:01
waiting... 3 05:47:01
waiting... 4 05:48:01
waiting... 5 05:49:03
backup.restore - started Mon Mar 30 05:50:03 EDT 2015 on cowtest
File to be restored is: /db2home/www/daily/ANIMALS.0.db2inst9.NODE0000.CATN0000.20150330030114.001.bz2
Testing file to be restored for integrity at Mon Mar 30 05:50:03 EDT 2015
unzipping backup file
------------ /db2home/db2inst9/archivelogs/animals ------------
total 0
------------ /db2home/db2inst9/archivelogs/animals ------------
SQL1064N DB2STOP processing was successful.
Waiting for 1 minute...
SQL1063N DB2START processing was successful.
ATTACH TO db2inst9
Instance Attachment Information
Instance server = DB2/LINUXX8664 10.5.5
Authorization ID = DB2INST9
Local instance alias = DB2INST9
DEACTIVATE DATABASE ANIMALS
DB20000I The DEACTIVATE DATABASE command completed successfully.
RESTORE DATABASE ANIMALS USER db2inst9 using
FROM '/db2home/restore1' REPLACE EXISTING WITHOUT
ROLLING FORWARD WITHOUT PROMPTING
SQL2555I The database was restored and then successfully upgraded to the
current DB2 release where you issued the RESTORE DATABASE command.
CONNECT to animals user db2inst9 using
Database Connection Information
Database server = DB2/LINUXX8664 10.5.5
SQL authorization ID = DB2INST9
Local database alias = ANIMALS
SELECT count(bhid) from is3.dna
1
-----------
220
1 record(s) selected.
DETACH
DB20000I The DETACH command completed successfully.
DISCONNECT CURRENT
DB20000I The SQL DISCONNECT command completed successfully.
ACTIVATE DATABASE ANIMALS
DB20000I The ACTIVATE DATABASE command completed successfully.
TERMINATE
DB20000I The TERMINATE command completed successfully.
ATTACH TO db2inst9
Instance Attachment Information
Instance server = DB2/LINUXX8664 10.5.5
Authorization ID = DB2INST9
Local instance alias = DB2INST9
RESTORE DATABASE ANIMALS USER db2inst9 using
FROM '/db2home/restore1' INTO VERIFY REPLACE EXISTING
WITHOUT ROLLING FORWARD WITHOUT PROMPTING
SQL1084C The database manager failed to allocate shared memory because an
operating system kernel memory limit has been reached. SQLSTATE=57019
connect to verify user db2inst9 using
SQL1084C The database manager failed to allocate shared memory because an
operating system kernel memory limit has been reached. SQLSTATE=57019
select count(bhid) from is3.dna
SQL1024N A database connection does not exist. SQLSTATE=08003
DETACH
DB20000I The DETACH command completed successfully.
TERMINATE
DB20000I The TERMINATE command completed successfully.
system is cowtest
SQL1064N DB2STOP processing was successful.
SQL1063N DB2START processing was successful.
DB20000I The ACTIVATE DATABASE command completed successfully.
------------ /db2home/db2inst9/archivelogs/20150330 ------------
total 0
------------ /db2home/db2inst9/archivelogs/20150330 ------------
backup.complete file has already been removed...
backup.restore - ended Mon Mar 30 07:44:52 EDT 2015 on cowtest