Apr 11

Considerations about DIRMC processing

Versions files and directories Comments Off on Considerations about DIRMC processing

Question
Is a specific storage pool required to keep directory information on disk?

Answer
The DIRMC option had a big impact on restore performance in earlier versions of Tivoli Storage Manager (pre V3), where directories were restored first, followed by the files. It was therefore an advantage to have directory information cached so that it was permanently on disk, and best practices design was to create small storage pools specifically for directories. The DIRMC directive in the client options file was used to bind directories to a management class pointing to this storage pool.

Restore processing has changed since then: during the process directories will be created with default attributes and the correct attributes and ACL information is applied once the data is read from the media. Therefore the original reason to cache directories on disk no longer applies.

Nevertheless, the DIRMC option is still useful. If you do not specify this option to associate a management class with directories, the client, during backup, uses the management class in the active policy set of your policy domain with the longest retention period, which could point to a storage pool on tape. This might result in unwanted mount requests during backup/restore.

Having the DIRMC point to a disk storage pool separate from the backup objects might cause the symptom as described with technote swg21247892 (Unexpected high number of EndTxn processing during back up, see link section below)

Related information

Unexpected high number of EndTxn processing during back

written by Bosse

Apr 11

Unexpected high number of EndTxn processing during back up

Versions files and directories Comments Off on Unexpected high number of EndTxn processing during back up

Problem(Abstract)
This technote describes a scenario where it can come to performance degradation during Tivoli Storage Manager client backup because an unexpected high number of transactions is processed.

Resolving the problem
In this scenario, the following settings have been defined:

Server->TxnGroupMax 256

Client-> TXNBYTELIMIT 25600

The following statistics have been reported by a client instrumentation trace:

Server Version 5, Release 3, Level 3.0
Data compression forced off by the server
Server date/time: 29-06-2006 12:00:05 Last access: 29-06-2006 09:15:07

Current command: Incremental
Total number of objects inspected: 74.819
Total number of objects backed up: 74.814
Total number of objects updated: 0
Total number of journal objects: 79.687
Total number of objects rebound: 0
Total number of objects deleted: 0
Total number of objects expired: 4.265
Total number of objects failed: 0
Total number of bytes transferred: 13,96 GB
LanFree data bytes: 0 B
Server-Free data bytes: 0 B
Data transfer time: 5.295,51 sec
Network data transfer rate: 2.766,04 KB/sec
Aggregate data transfer rate: 699,58 KB/sec
Total number of bytes pre-compress: 14.996.313.174
Total number of bytes post-compress: 14.996.313.174
Objects compressed by: 0%
Elapsed processing time: 05:48:57
Average file size: 195,74 KB

——————————————————————

Detailed Instrumentation statistics for
Thread: 6328 Elapsed time 20937,580 sec
Section Actual (sec) Average(msec) Frequency used

———————————————————
Process Dirs 0,000 0,0 0
Solve Tree 0,000 0,0 0
Compute 1,657 0,0 270984
BeginTxn Verb 0,141 0,0 11167
Transaction 48,856 4,4 11167
File I/O 299,004 1,0 307988
Compression 0,000 0,0 0
Encryption 0,000 0,0 0
CRC 0,000 0,0 0
Delta 0,000 0,0 0
Data Verb 2726,280 10,1 270984
Confirm Verb 0,000 0,0 1
EndTxn Verb 17755,910 1590,0 11167
Sleep 0,000 0,0 0
Thread Wait 85,075 2430,7 35
Other 20,657 0,0 0

——————————————————————

Detailed Instrumentation statistics for

Thread: 6508 Elapsed time 20908,581 sec
Section Actual (sec) Average(msec) Frequency used

——————————————————————
Process Dirs 0,000 0,0 0
Solve Tree 0,000 0,0 0
Compute 1,633 0,0 250649
BeginTxn Verb 0,076 0,0 11370
Transaction 49,845 4,4 11370
File I/O 304,188 1,1 288521
Compression 0,000 0,0 0
Encryption 0,000 0,0 0
CRC 0,000 0,0 0
Delta 0,000 0,0 0
Data Verb 2563,864 10,2 250649
Confirm Verb 0,110 110,0 1
EndTxn Verb 17901,403 1574,4 11370
Sleep 0,000 0,0 0
Thread Wait 72,014 1636,7 44
Other 15,448 0,0 0

——————————————————————
From above client instrument:detail stats:

1) total # of files backed up: 74814
2) # of EndTxn’s:
thread 6328: 11167
thread 6508: 11370
Total EndTxns: 22537 or avg # files/txn: 3.3.

With TXNBYTELIMIT=25600 KB , TxnGroupMax=256 and Average file size: 195,74 KB, it seems that the above stats show that Tivoli Storage Manager client is requesting way too many DB commits (EndTxn’s) from Tivoli Storage Manager server.

Avg # of files/txn should be closer to 256, rather than 3.3 as calculated above.

Further investigation did show that different management classes with different copy destinations were configured for files and directories with this setup.

In this case, the directories on the fileservers changed just as much as the files, forcing Tivoli Storage Manager client to request EndTxn’s every time it encountered files/dirs being backed up that are destined for a different MC during backup operation.

The change implemented in this case was to use the same management class for files and directories. The above example should give you a starting point to verify if you are facing this condition when running into performance problems during back up.

For more information on client instrumentation traces see 15.1.2 Tivoli Storage Manager client performance tracing in the IBM Tivoli Storage Manager Implementation Guide:
http://www.redbooks.ibm.com/redbooks/pdfs/sg245416.pdf

written by Bosse

Oct 28

Data binding to wrong managment class

Versions files and directories Comments Off on Data binding to wrong managment class

Problem(Abstract)

Client data is bound to an incorrect management class even though there is a include statement in place.

Resolving the problem

Different include statements have different effects on the data being backed up.

  • DIRMC management_class1
    This binds all the directory structures to the ‘management_class1’.
  • INCLUDE c:\test\* management_class2
    This binds all the files in the directory \test to ‘management_class2’ which includes files with extensions and files with no extensions.
  • INCLUDE c:\test\*.* management_class3
    This binds all the files in the directory \test that have an extension to management ‘management_class3’ but files with no extensions will still bind to the default management class.

written by Bosse

Oct 07

Distinguish the Management Class a Directory Will Be Bound To

Versions files and directories Comments Off on Distinguish the Management Class a Directory Will Be Bound To
Question
Since directories are bound to the management class with the longest retention setting (if dirmc is not set), what happens if there are several management classes with the same retention settings? Which management class will the directories be bound to?
 
 
Answer
Unless the DIRMC option is set in the Tivoli Storage Manager client options file, the directories for a Tivoli Storage Manager client backup will be bound to the management class in the active policy set containing the copygroup with the largest retention settings (highest RETONLY value).
If there are several management classes with the same retention settings, then the directories will be bound to the copygroup of the management class listed LAST in alphabetical order.Here is an example:In the domain STANDARD, this is a list of management classes and copygroups defined to the server for the active policy set:
The default management class is STANDARD. When an incremental backup of directory c:\temp is done, the directory is bound to the XYZ management class. To verify which management class the directory is bound to, the following select statement can be run from a Tivoli Storage Manager administrative client commandline:select class_name from backups where node_name=’YOURNODENAME’ and type=’DIR’ group by class_name

The output from the select statement in this example is:
CLASS_NAME
——————
XYZ

This shows that even though there are several management classes which have the same retention settings, the directories are bound to the last alphabetical listing.

written by Bosse

Oct 07

Directories not displayed for point in time restores using GUI client

Versions files and directories Comments Off on Directories not displayed for point in time restores using GUI client
Problem(Abstract)
Tivoli Storage Manager client fails to show folders in the GUI or web client GUI using a point in time restore
 
 
Cause
Tivoli Storage Manager server policy management for directories is preventing the Tivoli Storage Manager client GUI from properly displaying them.
 
 
Resolving the problem
The Tivoli Storage Manager point in time restore function is used to restore a file space, directory, or file to a previous condition. While a point in time restore can be useful for restoring the state of a system to an earlier date, it also carries a restriction – it can indirectly restrict what files and folders are selectable for the restore (such as from the GUI or web interface). Take the following example, which assumes that directories are bound to the same management class as files:
Filespace:
C:\
|
|-folder1
|
|-file1.txt(static)
|
|-file2.txt(changes before each backup)
|
|-folder2(static)Copygroup settings for this client (for both directories and files):
Verexists = 7 versions of a file(file exists on the original system)
Verdelete = 7 versions of a file(file no longer on the original system)
Retextra = 7 days
Retonly = 7 daysActions:
Assume that file2.txt changes every day, and a schedule backs up this filespace every day. When each incremental backup occurs, it sends a new active version of file2.txt, and a NEW ACTIVE version of folder1. In this example, the following table would result. This shows the backups that are available on Day 8, looking back for each of the prior 8 days (including Day 8). Active copies of the backed up file and folder are represented by the letter “A”, inactive copies by the letter “I”, and versions deleted from the filespace by the letter “X”, as shown below.

Active/Inactive/Deleted
———————————————-
Day 1 | A I I I I I I X
Day 2 | A I I I I I I
Day 3 | A I I I I I
Day 4 | A I I I I
Day 5 | A I I I
Day 6 | A I I
Day 7 | A I
Day 8 | A

On Day 8, if a point in time restore of either file1.txt or folder2, using Day 1 as the point in time criteria, it would fail to see any folders under C:\ (as viewed by the GUI or web client). That is, folder1 and any of its contents are not visible for selection under a point in time restore.

Why is this so?

The Tivoli Storage Manager client GUI/web client GUI point in time restore requires an active or inactive version of that folder to exist (at the point in time specified by the user) so that it can display items from within that folder to restore. Since folder1 now has a point in time scope going back only to Day 2, folder1 and its contents are unavailable because a backup version for that folder no longer exists at that point in time. Thus, nothing within that folder (active or not) can be selected for restore from the Tivoli Storage Manager client GUI/web client GUI.

To restore a folder or file to a particular point in time (such as file1.txt or folder2), use the ‘display active/inactive files’ option instead. With the Tivoli Storage Manager client GUI/web client GUI, select the ‘View’ menu, then select the ‘Display active/inactive files’ option. This is the equivalent of using the Tivoli Storage Manager command line client command ‘restore c:\* -pick -inactive -subdir=yes’. Select the version to be restored, based on the date and time criteria. Refer to the Tivoli Storage Manager B/A client manual for further information regarding the use of the client command line and its available options.

Alternatively, restore the filespace using the Tivoli Storage Manager client command line, which does not have this restriction. Use the PITDATE and PITTIME options to specify the desired point in date/time.

Managing proper versions of directories can be maintained using the DIRMC option in the dsm.opt file. See the Tivoli Storage Manager client manual for details. Also review technote solution 1447142 for learning how this works with Tivoli Storage Manager server policy management.

written by Bosse