May 07

Archive and active log directories are full and server will not start

ActiveLog ArchiveLog Comments Off on Archive and active log directories are full and server will not start

Question

If a FULL database backup is not taken in time to prune the archive logs before the directory becomes full. Either the active log becomes full, or the active directory becomes full of the logs that could not be archived. The result is that the Tivoli Storage Manager server shuts down.

Cause

Tivoli Storage Manager server shuts down and archive log directory and active log directory are full. The server cannot be started again.

Answer

When the archive log directory or active log directory fills up, or when the pre-allocated active logs fill up, the Tivoli Storage Manager shuts down. In order to get the Tivoli Storage Manager server up and running, you must update the DB2 configuration to change the location of the active log directory to a new location. This will allow to active logs to free up space and then the Tivoli Storage Manager server can start. You can then clear the archive log directory by issuing database backups. Complete the following procedure:

1. Create a temporary directory large enough to place the active logs.
See dsmserv.opt for value of ACTIVELOGSIZE.

2. Open a DB2 command prompt:
Windows:
Go to Start > All Programs > IBM DB2 > DB2COPY1 > Command Line Tools > Command Window – Administrator.

AIX/Linux/Unix:
1. Open a terminal session.
2. Log in as the Tivoli Storage Manager instance user.

3. Update the DB2 configuration to specify the new location for active log directory that you just created. From the DB2 command prompt, issue the following commands:
set db2instance=SERVER1
db2start
db2 update db cfg for tsmdb1 using newlogpath <path_from_step1>
db2 connect to tsmdb1

4. The active log files will not be removed until the following ‘activate’ command is issued.
First perform a deactivation to ensure that the database is not already activated:
db2 deactivate db tsmdb1
Then run the activate command:
db2 activate db tsmdb1

Note: The active logs will be COPIED and RENAMED (not moved) to the new temporary directory that you specified in step 1.

5. You have 2 options to prune the archive logs to get the Tivoli Storage Manager sever back up and running:

Hide details for Option A: Have Tivoli Storage Manager prune the archive logsOption A: Have Tivoli Storage Manager prune the archive logs
You can have Tivoli Storage Manager prune the archive logs by issuing database backups. You will need to update the DB2 configuration to point the active log direction location back to the original location, and then issue the activate command to copy over the log files to the original location. Since these newly created active log files have space to write transactions, the Tivoli Storage Manager server can start, and you can initiate DB backups to prune the archive logs. Complete the following steps:

A.5.1. Update the DB2 parameter to point the location of the active log back to the original location.
See ACTIVELOGDIRECTORY value in dsmserv.opt.
db2 UPDATE DATABASE CONFIG FOR TSMDB1 USING NEWLOGPATH <path_to_activelogdir>

A.5.2. Issue the activate command to copy the logs to the original location:
db2 activate db tsmdb1

A.5.3. Start the Tivoli Storage Manager server in the foreground.
Open a command terminal and navigate to the server directory and issue the dsmserv command to start the correct instance. Refer to the command here:
http://publib.boulder.ibm.com/infocenter/tsminfo/v6r2/topic/com.ibm.itsm.srv.ref.doc/r_cmd_dsmserv_start.html

A.5.4. Run a database backup to prune some of the archive logs.
dsmserv backup db type=full dev=<your device class>

A.5.5. Review the archive log directory. Is the drive/volume free space still low? You will have to decide if you need to run a second database backup to avoid having the archive logs fill up again before the next scheduled database backup. This depends on the average growth of archive logs per day. Review the timestamps on the archive log.

Hide details for Option B: Have DB2 prune the archive logsOption B: Have DB2 prune the archive logs
You can have DB2 prune the archive logs by running a DB2 database backup.

B.5.1. Create a directory on a drive/volume with enough space to hold a full DB2 database backup.

B.5.2. From the DB2 command prompt, issue the following commands:
db2 deactivate db tsmdb1
db2 backup db tsmdb1 to <path_to_database_directory>

Examples:
Windows: db2 backup db tsmdb1 to C:\dbbackup
AIX/Linux: db2 backup db tsmdb1 to /tmp/dbbackup

The following message will appear once the backup is complete:
Backup successful. The timestamp for this backup image is: 20090721130818
At the end of this backup, archive logs will start pruning.
Note: depending on the size of the database and archive logs, this could take a while to complete the backup.

B.5.3 Review the archive log directory. Is the drive/volume free space still low? You will have to decide if you need to run a second DB2 database backup to avoid having the archive logs fill up again before the next scheduled Tivoli Storage Manager database backup. This depends on the average growth of archive logs per day. Review the timestamps on the archive log.
If need, issue another DB2 database backup.

B.5.4. Delete any DB2 database backups as follows:
db2stop
db2start
db2 connect to tsmdb1
db2 PRUNE HISTORY <timestamp> WITH FORCE OPTION AND DELETE

where <timestamp> is the timestamp that was displayed after the DB2 database backup completed. See step B.5.2, where the timestamp is: 20090721130818.

B.5.5. Update the DB2 parameter to point the location of the active log back to the original location.
Ssee ACTIVELOGDIRECTORY value in dsmserv.opt.
db2 UPDATE DATABASE CONFIG FOR TSMDB1 USING NEWLOGPATH <path_to_activelogdir>

B.5.6. Connect to the database to start automatic moving of active logs from the temporary location to original active log location.
db2 force application all
db2stop
db2start
db2 connect to tsmdb1

The connect command will trigger DB2 to move (not copy) the active logs to the original directory.
Note: depending on the size of the active logs, this process can take a long time.

B.5.7. Start Tivoli Storage Manager server in foreground.
Open a command terminal and navigate to the server directory and issue the dsmserv command to start the correct instance. Refer to the command here:
http://publib.boulder.ibm.com/infocenter/tsminfo/v6r2/topic/com.ibm.itsm.srv.ref.doc/r_cmd_dsmserv_start.html

B.5.8. Take a full database backup by issuing the following commands:
disable sessions
backup db type=full dev=<your device class>
enable sessions

Now that the Tivoli Storage Manager server is up and running, attention should be paid to see how often FULL Tivoli Storage Manager database backups are needed in order to prune the archive log directory.

written by Bosse