Mar 31

Tivoli Storage Manager database backup fails with ANR2984E

DB2 Comments Off on Tivoli Storage Manager database backup fails with ANR2984E

Problem(Abstract)

After an upgrade of the Tivoli Storage Manager server, database backup operations fail with error message ANR2984E.

Symptom

The following error is shown in the activity log:

ANR1360I Output volume O00046L5 opened (sequence number 1). ANR4626I Database backup will use 1 streams for processing with the number originally requested 1. ANR2984E Database backup terminated due to environment or setup issue related to DSMI_CONFIG – DB2 sqlcode -2033 sqlerrmc 406 . ANR1361I Output volume O00046L5 closed. ANR0515I Process 3 closed volume O00046L5.

 

Resolving the problem

To solve this issue, complete the following steps:

  1. Verify that the DB2_VENDOR_INI parameter is set correctly to tsmdbmgr.env. Run the following command: db2set -all
    If DB2_VENDOR_INI is not set correctly, issue the following command to update it: db2set -i DB2_VENDOR_INI=<instance_directory>\tsmdbmgr.env
  2. Stop the Tivoli Storage Manager server.
  3. Open a DOS Command window to start the Tivoli Storage Manager server in the foreground.
  4. Back up the Tivoli Storage Manager database.

written by Bosse

Mar 16

Auditing a Tivoli Storage Manager deduplicated storage pool

Info Comments Off on Auditing a Tivoli Storage Manager deduplicated storage pool

Problem(Abstract)

Under some circumstances, an audit of a deduplicated storage pool might be necessary. A utility is now available that can be used to audit the deduplication tables to ensure referential integrity, proper chunk linkage, and cleanup of previously-marked, damaged files.

Symptom

The following messages are primary indicators of a potential problem within the deduplication catalog:

ANR4895E ANR1165E ANR1529I ANR1162W

 

Cause

The most common cause of invalidated links within a deduplicated storage pool is the forcible removal of data chunks via the DELETE VOLUME XXXXXX DISCARDDATA=YES command. This command will remove data chunks from the database/media regardless of dependencies within the catalog, which will create invalidated links. As a result, you must follow the DELETE VOLUME DISCARDD=YES command with an audit against the ANR4589E message to cleanup the invalidated links. Until the audit is run, you will see ANR4589E or ANR1162W messages in the activity log that indicates that there are invalidated links.
Another potential cause of invalidated link and referential integrity errors is from DB2 table damage that can occur from a hardware failure. As a result, you can use the DB2DART utility in order to restore structural integrity. Then, you can use the dedupAuditTool perl script, attached to this document, to resolve referential integrity gaps.
The following list include APARs to describe the problem that links can be invalidated during data movement operations:
IC90390 IC96993
It is recommended that the version of the server is at 6.3.4.210 or higher to prevent the problems that are documented in the APARs from occurring in your deduplication environment.

Environment

The problem affects Tivoli Storage Manager servers with a DEVT=FILE storage pool that has deduplication enabled. The data can be deduplicated either by client-side or server-side (Identify).

 

Diagnosing the problem

Use the dedupAuditTool Perl script that is attached to this document.

 

Resolving the problem

 

Overview

As noted in the Abstract area of this document, there can be various reasons for encountering server messages that indicate damage within a deduplicated storage pool. The first step to resolving any deduplication catalog inconsistency is to run an analyzer against the storage pool to determine where the problem might be. Then you would complete the steps to resolve the error. The attached dedupAuditTool.pl script does the analyzing and cleanup to ensure the quickest resolution possible.

The dedupAuditTool.pl is a perl script, which is broken in to 2 audit phases. The first phase is called an ANALYSIS phase and its designed to accept environment and symptom information from the user. With this information, the tool interrogates the various Tivoli Storage Manager tables that are associated with the given symptom that was specified. You are prompted to either create an audit report or proceed directly to the RECOVERY phase. The recovery phase is responsible for either removing the damage, restoring from a copy source, or potentially both, depending on the salvage ability for each given object.

 

Script Details
General Script Information:

  • Perl must be installed on the system where the Tivoli Storage Manager server resides. Because the Perl script interrogates the DB2 database directly, it must be run from this system. To obtain code for the Perl installation, go to http://www.perl.org
  • The tool must be run with a user id that has access to the DB2 instance.
  • The instance user must have FULL read/write access to the directory where the script is being executed and the directory where the client is installed (i.e. /opt/tivoli/tsm/client/ba/bin64).
  • The tool requires access to both DB2 and Tivoli Storage Manager via the dsmadmc and DB2 CLIs.
  • You must use Perl 5.8.0 or higher (A Perl interpreter can be download from – www.perl.org).
  • Two different scripts are available depending on which interpreter is available

– dedupAuditTool.pl (5.10 or higher) – dedupAuditTool_perl_v580 (5.8.0 – 5.10)

  • The tool is an interactive script. It does not accept command line parameters.
  • The tool is a multi-phase script – Environment Setup, Audit Tool Setup, Audit Analysis and Recovery(Cleanup).

How Is The Script Executed?: perl dedupAuditTool.pl
Important Usage Notes:

  • If a DELETE VOLUME DISCARDDATA=YES command has been issued against a volume that is located in a deduplicated storage pool, this script should be run immediately afterwards with the ANR4895E symptom to resolve all of the invalidated links.
  • The script should only be run in a deduplicated environment for which the following requirements are all true (the script will prompt for these as well):

– A current FULL database backup is available – All data movement and expiration activity for the deduplicated storage pool has been quiesced
– The Tivoli Storage Manager server is at 6.3.4.210 higher

  • It is typically recommended to run the RECOVERY mode with BOTH so that all restorable objects are recovered from a copy source and all non-restorable objects are removed. This will result in the fastest and most straightforward cleanup possible.

Audit Details Audit Symptom Categories and Coverage Details: – ANR4895E An invalidated deduplication chunk link has been found.
– ANR1165E and ANR1162W A damaged file has been found in a deduplicated storage pool.
– ANR1529I
Damaged and expired base chunks have been found in a deduplicated storage pool.
– MISSING_EXTENT_ENTRY Category that checks for any deduplicated object which no longer cataloged.
– ORPHANED_EXTENT
Category that checks for an invalidated chunk that only exists in the deduplication catalog table.
– INVALIDATED_LINKS Category that checks for any invalidated links that might exist in the deduplicated storage pool.
– MISSING_EXTENT_ENTRY Category that checks for any deduplicated object which no longer has an extent in the deduplication
catalog.
– MISSING_AF_ENTRIES Category that checks for any deduplicated extent that no longer has a corresponding entry in the volume tracking table.
– MISSING_VOL_ENTRIES Category that checks for a deduplicated object that no longer has a valid volume associated to it.
– ZERO_LENGTH_CHUNKS Category that checks for any base data chunks (not links) that no longer contain actual storage references.
– MISSING_CHUNKS Category that checks for an object that has been deduplicated but no longer has entries in the database deduplication catalog.
– MISSING_CHUNKS_EXTENDED Category that performs additional resource intensive checks beyond what the MISSING_CHUNKS symptom covers. – ALL The ALL category will go through all of the above symptoms plus a few others that are not documented above. This category should only be run as part of a health check and can run for a long period of time, depending on the size of the database and the deduplicated storage pool.
Audit Phase Options: * The deduplication catalog can be analyzed by either auditing a symptom or by processing a previously generated audit file
Recovery Phase Options: * The following cleanup phase options are available. The cleanup phase occurs after the analysis
is completed: – Either an audit file can be generated, for future processing, or the script can enter a recovery procedure. If the recovery procedure is chosen, the following modes are available: – Restore/Delete
All invalid data is attempted for restoration. If data cannot be restored it will then be removed from the Tivoli Storage Manager server. – Restore Only All invalid data is attempted for restoration from a copy pool. – Delete Only All invalid data will be removed from the Tivoli Storage Manager server. – Preview All recovery steps are taken, including restoration and deletion, but no action is performed. The console will display each step that would be covered if the Recovery was executed without PREVIEW.

Script Examples

Hide details for AUDIT OF ANR4895E SYMPTOM (Invalid Links Found) AUDIT OF ANR4895E SYMPTOM (Invalid Links Found)
***********************************************************************
Welcome to dedupAuditTool! The runtime perl version: 5.010001
Attention!!
Before proceeding with the cleanup procedure, make sure the following statements are true:
– A current FULL database backup is available!
– All data movement and deletion activity in the deduplicated pool has been quiesced! This would include the following processes:
Expiration Reclamation Migration Move Data Move NodeData
– The Tivoli Storage Manager server is at 6.3.4.210 higher!
************************************************************************
Continue with the deduplication audit? (Y/N): <def: N>: Y

***********************************************************
Phase 1: Environment Setup
***********************************************************
Input the db2 database name <def: TSMDB1>: MGSUNC
Input the db2 schema name <def: TSMDB1>: TSMDB1
The connection to DB2 succeeded Input the dsmadmc path(double quote the path) < def: “/opt/tivoli/tsm/client/ba/bin64”>:
Input the admin name <def: admin>:admin Input the password for admin <def: admin>:admin Input the option file(double quote the path) < def: “/opt/tivoli/tsm/client/ba/bin64/dsm.opt”>: Connection to DISCO_A_SRV succeeded.
Finding all deduplicated storage pools
. .
***********************************
DEDUPLICATED STORAGE POOLS:
FILEPOOL
FILENEXTPOOL ***********************************
Input the poolid or poolname to process <def: deduppool>: FILEPOOL The DB2 command has started The DB2 command succeeded Conversion between poolid(4) AND poolname(FILEPOOL) succeeded
***********************************************************
Phase 2: Dedup Audit Tool Setup
***********************************************************
Audit with script[A] or by previously generated audit file[F]? <def: A>: A
Would you like to see all available symptoms that can be audited? <def: Y>: Y
*********AUDIT SYMPTOMS********* ALL (INCLUDES EVERYTHING BELOW): ANR4895E ANR1165E ANR1529I
ANR1162W MISSING_EXTENT_ENTRIES
MISSING_SEGMENT_ENTRIES MISSING_AF_ENTRIES
ORPHANED_EXTENT MISSING_VOL_ENTRIES
INVALIDATED_LINKS ZERO_LENGTH_CHUNKS
MISSING_CHUNKS MISSING_CHUNKS_EXTENDED
*********AUDIT SYMPTOMS*********
Input the symptom, or error message, you want to check <def: ANR4895E>: ANR4895E
Checking to see if there are active dedup deletions occurring.
If there are, the audit will wait until the deletions are finished.

Command #1 to perform is: tsm show dedupdeleteinfo
The TSM command has started The TSM command succeeded
Deduplicated chunks are not being deleted at this time, proceeding….
***********************************************************
Phase 3: Auditing Deduplicated Storage Pool FILEPOOL
***********************************************************
The dedup audit tool is running in ANR4895E mode The audit is in processing phase 1: ZERO_LENGTH_BASE_CHUNKS The DB2 command has started The DB2 command succeeded The audit scan found nothing wrong in phase 1: ZERO_LENGTH_BASE_CHUNKS
The audit is in processing phase 2: INVALIDATED_DEDUP_LINKS The DB2 command has started The DB2 command succeeded The audit scan has detected a potential problem in phase 2: INVALIDATED_DEDUP_LINKS!
The audit is in processing phase 3: DAMAGED_DATA The DB2 command has started The DB2 command succeeded The audit scan found nothing wrong in phase 3: DAMAGED_DATA

***********************************************************
Phase 4: Cleanup of the Deduplicated Storage Pool Cleanup procedures to perform include: INVALIDATED_DEDUP_LINKS
***********************************************************
Perform recovery procedure[RP] or generate an audit file[AF] <def: AF>: AF Input the report file name <def: reportdedup.out>: The report file has been generated.
Check reportdedup.out for the results.

Hide details for EXAMPLE OF AUDIT FILE (Invalid Links Found) EXAMPLE OF AUDIT FILE (Invalid Links Found)
******************************************************************
SYMPTOM: INVALIDATED_DEDUP_LINKS DB2SQL: “select count(*) from bf_bitfile_extents bfbe where bfbe.srvid=0 and bfb
e.poolid=4 and bfbe.linkbfid=9223372036854775807″ RESULT: 10 STORAGE POOL NAME(POOLID): FILEPOOL(4)
******************************************************************

Hide details for RECOVERY OF INVALID LINKS (Using Audit File) RECOVERY OF INVALID LINKS (Using Audit File)
***********************************************************************
Welcome to dedupAuditTool! The runtime perl version: 5.010001
Attention!!
Before proceeding with the cleanup procedure, make sure the following statements are true:
– A current FULL database backup is available!
– All data movement and deletion activity in the deduplicated pool has been quiesced! This would include the following processes:
Expiration Reclamation Migration Move Data Move NodeData
– The Tivoli Storage Manager server is at 6.3.4.210 higher!
************************************************************************
Continue with the deduplication audit? (Y/N): <def: N>: Y

***********************************************************
Phase 1: Environment Setup
***********************************************************
Input the db2 database name <def: TSMDB1>: MGSUNC
Input the db2 schema name <def: TSMDB1>: TSMDB1
The connection to DB2 succeeded Input the dsmadmc path(double quote the path) < def: “/opt/tivoli/tsm/client/ba/bin64”>:
Input the admin name <def: admin>: ADMIN Input the password for admin <def: admin>: ADMIN Input the option file(double quote the path) < def: “/opt/tivoli/tsm/client/ba/bin64/dsm.opt”>: Connection to MSISCO_A_SRV succeeded.
Finding all deduplicated storage pools . .
*********************************** DEDUPLICATED STORAGE POOLS:
FILEPOOL FILENEXTPOOL
***********************************
Input the poolid or poolname to process <def: deduppool>: FILEPOOL The DB2 command has started The DB2 command succeeded Conversion between poolid(4) AND poolname(FILEPOOL) succeeded
***********************************************************
Phase 2: Dedup Audit Tool Setup
***********************************************************
Audit with script[A] or by previously generated audit file[F]? <def: A>: F Input the report file name < def: reportdedup.out>: reportdedup.out Read report file reportdedup.out … The report file( reportdedup.out ) processing was successful
***********************************************************
Phase 4: Cleanup of the Deduplicated Storage Pool Cleanup procedures to perform include: INVALIDATED_DEDUP_LINKS
***********************************************************
Perform recovery procedure now? <def: N>: Y
showdamagedoutput.out exists from previous audit. Remove the file? (Y/N): <def: Y>: Y
Perform INVALIDATED_DEDUP_LINKS recovery procedure? (Y/N):< def: Y> Y
***********************************************************
Starting recovery procedure for INVALIDATED_DEDUP_LINKS
***********************************************************
Perform restore/delete[B], restore[R], delete[D], preview[P] operation <def: R>: D
Performing DB setup required for recovery operation of the INVALIDATED_DEDUP_LINKS.
Command #1 to perform is: db2 “select ‘show bfo ‘ || bfid from af_damaged where poolid=4” => showobject.mac The DB2 command has started
The DB2 command succeeded
Command #2 to perform is:
tsm -itemcommit macro showobject.mac => showdamagedoutput.out
The TSM command has started The TSM command succeeded
Command #3 to perform is: db2 “delete from AF_DAMAGED where srvid=0 and poolid=4” The DB2 command has started
The DB2 command succeeded
Check the showdamagedoutput.out file for any previously marked damaged files.
Performing the recovery operation for INVALIDATED_DEDUP_LINKS corruption

Command #4 to perform is: db2 “delete from AF_DAMAGED where srvid=0 and poolid=4” The DB2 command has started
The DB2 command succeeded
Command #5 to perform is:
tsm restore stgpool FILEPOOL w=y The TSM command has started The TSM command succeeded
Command #6 to perform is: db2 “select ‘delete object ‘ || cast( bfbf.owner as char(24) ) || ‘ force=yes’ from bf_bitfile_extents bfbe left join bf_aggregated_bitfiles bfbf on ( bfbe.srvid=bfbf.srvid and bfbe.bfid=bfbf.bfid and bfbe.superbfid=bfbf.superbfid ) where bfbe.srvid=0 and bfbe.poolid=4 and bfbe.linkbfid=9223372036854775807 and bfbf.srvid is not NULL group by bfbf.owner” => deleteobject.mac The DB2 command has started
The DB2 command succeeded
Command #7 to perform is:
tsm -itemcommit macro deleteobject.mac The TSM command has started The TSM command succeeded
Command #8 to perform is: tsm show dedupdeleteinfo The TSM command has started The TSM command succeeded Deduplicated chunks are not being deleted at this time, proceeding….
Command #9 to perform is: db2 “insert into AF_DAMAGED ( srvid, bfid, poolid, updator ) (select distinct 0, superbfid, poolid, 2 from bf_bitfile_extents where srvid=0 and poolid=4 and linkbfid=9223372036854775807 and bfid!=9223372036854775807 )” The DB2 command has started
The DB2 command succeeded
Command #10 to perform is:
db2 “select ‘show bfo ‘ || bfid from af_damaged where poolid=4” => showobject.mac The DB2 command has started The DB2 command succeeded
Command #11 to perform is: tsm -itemcommit macro showobject.mac => showdamagedoutput.out The TSM command has started The TSM command succeeded
Check current entries in showdamagedoutput.out for any remaining invalid objects.

Hide details for AUDIT OF ANR4895E SYMPTOM (No Problems Found) AUDIT OF ANR4895E SYMPTOM (No Problems Found)
***********************************************************************
Welcome to dedupAuditTool! The runtime perl version: 5.010001
Attention!!
Before proceeding with the cleanup procedure, make sure the following statements are true:
– A current FULL database backup is available!
– All data movement and deletion activity in the deduplicated pool has been quiesced! This would include the following processes:
Expiration Reclamation Migration Move Data Move NodeData
– The Tivoli Storage Manager server is at 6.3.4.210 higher!
************************************************************************
Continue with the deduplication audit? (Y/N): <def: N>: Y

***********************************************************
Phase 1: Environment Setup
***********************************************************
Input the db2 database name <def: TSMDB1>: MGSUNC
Input the db2 schema name <def: TSMDB1>: TSMDB1
The connection to DB2 succeeded Input the dsmadmc path(double quote the path) < def: “/opt/tivoli/tsm/client/ba/bin64”>:
Input the admin name <def: admin>: ADMIN Input the password for ADMIN <def: admin>: ADMIN Input the option file(double quote the path) < def: “/opt/tivoli/tsm/client/ba/bin64/dsm.opt”>: Connection to MSISCO_A_SRV succeeded.
Finding all deduplicated storage pools . .
*********************************** DEDUPLICATED STORAGE POOLS:
FILEPOOL FILENEXTPOOL
***********************************
Input the poolid or poolname to process <def: deduppool>: FILEPOOL The DB2 command has started The DB2 command succeeded Conversion between poolid(4) AND poolname(FILEPOOL) succeeded
***********************************************************
Phase 2: Dedup Audit Tool Setup
***********************************************************
Audit with script[A] or by previously generated audit file[F]? <def: A>: A
Would you like to see all available symptoms that can be audited? <def: Y>: Y
*********AUDIT SYMPTOMS********* ALL (INCLUDES EVERYTHING BELOW): ANR4895E ANR1165E ANR1529I
ANR1162W MISSING_EXTENT_ENTRIES
MISSING_SEGMENT_ENTRIES MISSING_AF_ENTRIES
ORPHANED_EXTENT MISSING_VOL_ENTRIES
INVALIDATED_LINKS ZERO_LENGTH_CHUNKS
MISSING_CHUNKS MISSING_CHUNKS_EXTENDED
*********AUDIT SYMPTOMS*********
Input the symptom, or error message, you want to check <def: ANR4895E>: ANR4895E
Checking to see if there are active dedup deletions occurring.
If there are, the audit will wait until the deletions are finished.

Command #1 to perform is: tsm show dedupdeleteinfo
The TSM command has started The TSM command succeeded
Deduplicated chunks are not being deleted at this time, proceeding….
***********************************************************
Phase 3: Auditing Deduplicated Storage Pool FILEPOOL
***********************************************************
The dedup audit tool is running in ANR4895E mode The audit is in processing phase 1: ZERO_LENGTH_BASE_CHUNKS The DB2 command has started The DB2 command succeeded The audit scan found nothing wrong in phase 1: ZERO_LENGTH_BASE_CHUNKS
The audit is in processing phase 2: INVALIDATED_DEDUP_LINKS The DB2 command has started The DB2 command succeeded The audit scan found nothing wrong in phase 2: INVALIDATED_DEDUP_LINKS
The audit is in processing phase 3: DAMAGED_DATA The DB2 command has started
The DB2 command succeeded The audit scan found nothing wrong in phase 3: DAMAGED_DATA
No problems were detected by the dedup audit tool for FILEPOOL!

Attached Scripts
dedupAuditTool.pl – Script that requires PERL version 5.10 or higher
dedupAuditTool_v580.pl – Script that requires PERL version 5.8.0 – 5.10
NOTE: If is recommended that dedupAuditTool.pl is used if the 5.10 or higher PERL interpreter is available for the given operating environment.

written by Bosse

Mar 05

Iscsilog.dll is not included in the system state backup files on a Windows Server 2008 R2 SP1-based computer

Windows 2008 Comments Off on Iscsilog.dll is not included in the system state backup files on a Windows Server 2008 R2 SP1-based computer

Error description

  • The TSM client may issue the following warning message during a
    backup of the boot drive (usually C:).
    
      ANS1417W Protected system state file
      'C:\Windows\System32\iscsilog.dll' is backed up
      to the drive file space, not system state file space.
    
    This specific file is protected by Windows Resource Protection
    (WRP) but is not enumerated for backup by the Windows VSS System
    Writer. Since files protected by WRP should be part of the VSS
    System Writer, which is part of system state, TSM issues the
    message.
    
    The problem is introduced when Service Pack 1 (SP1) is applied
    to Windows 2008 R2. The problem is detected by the TSM backup-
    archive client as a result of the system state integrity
    verification enhancement made for APAR IC72446.
    
    Initial Impact: Low
    Platform Affected:
    Windows 2008 R2 Sp1

Fix

http://support.microsoft.com/kb/2680097/en-US

 

written by Bosse