The extremely low cost of tape and tape cartridge media compared to their relatively large storage capacity, as well as their small physical size, makes them highly desirable for use in backup and interchange applications. Yet there is no current proposal to include them in an extension to the DICOM PS 3 standard.
Such a proposal requires definition of a media physical format, as well as a logical format to be included in an application format. The choice of a physical media characteristics are chosen below to leverage off existing physical media standards, such as the DAT DDS formats, QIC formats, and Exabyte 8, format in order to maximize the opportunity for interchangeability. Where a standard is unsuitable, such as the ANSI standards for 9-track tape headers, which does not include long enough entries for DICOM file identifiers, and was only ever intended for the interchange of text not binary information, it has been discarded, despite its use in ACR/NEMA PS 1, which was never widely adopted.
The choice of physical block sizes is based on a compromise between the need for efficient transfer of large chunks of data between the application and the tape driver software and device, the relatively universal availability of the variable length read and write facility, for DATs at least (which reblock and/or compress the data before writing anyway), and the unnatural but common inability of software drivers to separate physical block size from bus transfer size from software driver transfer size (see read(2),write(2) on unix systems). A value of 64512 was chosen that will work with most drivers which set an upper limit of 65534 bytes, yet is a multiple of 1024 for those devices that insist on a multiple of 512 (QIC) or 1024 (8mm). This is common practice amongst existing tape users (remember tar -b 126b ?).
The inclusion of short headers before each data file is similar in intent to the ANSI tape label standards, and has been adopted in order to make the possibility of recovery after media errors feasible on those drives that support reading past a medium error ... each significant length of tape is self-describing, and hence some data recovery should theoretically be possible. It is hoped that this choice will not adversely affect streaming behaviour.
The choice of individual files separated by tape marks seems to be common practice amongst proprietary media storage formats from medical imaging equipment vendors here. Contrast this choice with that of common general formats such as tar(5) or cpio(5) which write monolithic files of concatenated data, with an internal header structure. The choice was made in this case to allow easy appending of data files on to the end of an existing tape if the File Set Updater role is supported, and to allow easy access to components of the tape data stream by common utilities such as mt(1) and dd(1) on Unix systems, allowing the construction of simple File Set Reader and File Set Creator applications with shell scripts. Again, it is hoped that this choice will not adversely affect streaming behaviour.
The use of common general formats such as tar(5) or cpio(5) is eschewed for these reasons, particularly the common inability of streaming tape software drivers to append to these formats. The use of such a de facto industry standard initially seems attractive, and would be feasible if simple limits on file identifier names and choice of order of inclusion of files (where to put DICOMDIR ?) were adopted, but the need to manage serial sets of tar archives if updating were permitted, and the absence of a clearly defined directory, necessitating scanning the entire tape contents to determine what files are present and where, negates many of the advantages of using such a format. Besides, compared to implementing the rest of DICOM, writing a handler for a new tape format is straightforward.
The fixed layout of the proposed logical file system directory and the various headers is chosen for simplicity over using a tag based scheme like the rest of DICOM. It is preceived that this is valid for what is really a layer beneath the rest of DICOM, where efficiency and simplicity are an advantage. The existing Parts 10,11 and 12 did not try to redfine a logical file system for disk but rather adopted the MSDOS file system, which has a fixed layout at this level, and hence the precedent for such a choice has been established.
The biggest difficulty with definition of an updateable file format that is applicable to more than one form of tape, is the choice of where to put the directory information. Experiments and consultation with those whose know these things reveal that though 9-track tape drives can rewrite over initial gap files, DAT's and QIC's certainly can't. Hence the approach used by PS 1 and many proprietary 9-track solutions won't work with other types of tape.
Those who defined the DDS DAT formats perceived the need for this type of behaviour by allowing the option of a two partition DAT, with a relatively short fixed up front Partition 1, and the remainder of the tape as Partition 0, the default partition available to those readers who don't know about partitions. Unfortunately, the selection and formatting of such partitions requires the use of utilities which may not be available on many platforms, or direct access to SCSI commands (Mode Select and Locate) which is also not available on many platforms. In order to make the proposed format portable to such platforms, a means to allow support for Partition 1 where available, but with sufficient information stored redundantly in Partition 0 also, is recommended. Thus a File Set Reader that is Partition 1 aware can access the directory quickly, others must spool to the end of media and find the trailing Partition 0 directory of the leading Partition 0 directory is not in use (the medium is updateable).
Thus a proposal that is portable across platforms and yet is shared between media types can offer extensions for specific media types and platforms without compromising interchangability or efficiency.
The question of the inclusion or exclusion of a compression Transfer Syntax is a vexed one. Many modern devices provide in-built compression on the fly (DDS-2 for example) yet this may not optimally compress medical images about which a priori information is known that may allow a better choice of compression algorithm. The limited options in the standard at present, make the inclusion of lossless JPEG seem attractive, as is used in the cardiology CD-R application profile for example. Yet a working group is currently seeking proposals for other compression options. Alternatively, having defined the physical media, it is relatively straightforward to extend one of the Application Profiles described to use another Transfer Syntax and hence the issue if left open for a future extension to this proposal.
The restriction to storage classes of CT and MR type, with limits on size, is made to satisfy a particular perceived clinical need, and avoid overburdening complying applications with the need to handle larger images or a wider range of storage classes. Other Application Profiles for other applications will undoubtedly follow.
Notice that the Secondary Capture storage class has specifically NOT been included, in order to avoid encouraging implementors who are increasingly commonly circumventing the need to construct or store complete CT and MR information objects with all the required attributes by constructing or storing simpler, but considerably less useful, SC objects, and exporting them from their acquisition devices or workstations.
This Annex defines a series of Application Profiles for recording of CT and MR images on sequential media. The identifiers for this series of application profiles are classified according to medium and are shown below.
Application Profile Identifier Basic CT/MR on DAT DDS-1 60m STD-SQCTMR-DDS-1-60 Basic CT/MR on DAT DDS-1 90m STD-SQCTMR-DDS-1-90 Basic CT/MR on DAT DDS-DC 60m STD-SQCTMR-DDS-DC-60 Basic CT/MR on DAT DDS-DC 90m STD-SQCTMR-DDS-DC-90 Basic CT/MR on DAT DDS-2 120m *STD-SQCTMR-DDS-2-120 Basic CT/MR on Exabyte 8mm 2GB STD-SQCTMR-8MM-2 Basic CT/MR on Exabyte 8mm 5GB STD-SQCTMR-8MM-5 Basic CT/MR on QIC-11 STD-SQCTMR-QIC-11 Basic CT/MR on QIC-24 STD-SQCTMR-QIC-24 Basic CT/MR on QIC-120 STD-SQCTMR-QIC-120 Basic CT/MR on QIC-150 STD-SQCTMR-QIC-150 Basic CT/MR on QIC-250 STD-SQCTMR-QIC-250 Basic CT/MR on 9-track 800 cpi STD-SQCTMR-9T-800 Basic CT/MR on 9-track 1600 cpi STD-SQCTMR-9T-1600 Basic CT/MR on 9-track 6250 cpi STD-SQCTMR-9T-6250
All of these application profiles handle single frame CT or MR images up to a maximum resolution of 1024 byte 1024 pixels with a pixel depth of 12 or 16 bits.
Note that though a wide range of media is potentially supportable in order to allow existing equipment to become compliant, use of any Application Profile (and corresponding medium) other than STD-SQCTMR-DDS-2-120 is deprecated in order to facilitate evolution towards a commonly interchangeable medium.
These Application Profile Classes facilitate the interchange of CT and MR images on a wide range of low cost sequential media. Typical interchanges would be from an acquisition device to workstations or archive devices, or between workstations and archive devices, or as the primary archival medium for an acquisition device.
The operational use of the media transfer is potentially both intra-institutional and inter-institutional.
This Application Profile Class uses the Media Storage Service Class defined in PS 3.4 with the Interchange Option.
The Application Entity shall support one or more of the roles of File-set Creator, File-set Reader, and File-set Updater, defined in PS 3.10.
The Application Entity acting as a File-set Creator generates a File Set under one of the specified Application Profile Classes in this Annex. Typical entities adopting this role would include acquistion devices archiving images, archive devices preparing images for interchange, or workstations preparing modified images for interchange.
File-set Creators shall be able to generate one or more of the SOP Classes defined for the specific Application Profile for which a Conformance Statement is made.
File-set Creators may, but shall not be required to, generate the Basic Directory SOP Class in the DICOMDIR file with all types of Directory records related to the SOP Classes stored in the File-set.
File-set Creators may, but shall not be required to, make use of a Logical File System Directory, to facilitate subsequent reading or updating.
File-set Creators may, but shall not be required to, generate a logical format that may subsequently be updated, either by the same Application Entity acting in the File-set Updater role, or by another Application Entity, by leaving the Leading Logical File System Directory and DICOMDIR unused.
File-set Creators may, but shall not be required to, make use of a Directory Partition if supported by the underlying physical medium, to facilitate subsequent reading or updating, but such medium will then only be updateable by Application Entities that also support multiple partitions.
The role of File-set Reader is adopted by Application Entities which receive a transferred File-set. Typical entities using this role would include display workstations and archive systems. File-set Readers shall be able to read all the SOP Classes defined for the specific Application Profile for which a Conformance Statement is made, using all the defined Transfer Syntaxes.
File-set Readers may, but shall not be required to, make use of a Logical File System Directory, if present.
File-set Creators may, but shall not be required to, make use of a Directory Partition if supported by the underlying physical medium.
The role of File-set Updater is adopted by Application Entities which update a File-set by the addition of information. Typical entities using this role would include acquisition or archival devices adding newly acquired images to a previously existing File-set of related images, or workstations and adding processed images.
File-set Updaters do not have to read the images.
File-set Updaters shall be able to generate one or more of the SOP Classes defined for the specific Application Profile for which a Conformance Statement is made, and to read and update the DICOMDIR file.
File-set Updaters shall be required to update any Logical File System Directory present, or else will not attempt to update the medium. The action taken in this event shall be documented in th Conformance Statement.
File-set Updaters shall be required to make use of a Directory Partition if supported by the underlying physical medium and if present, or else will not attempt to update the medium. The action taken in this event shall be documented in the Conformance Statement.
SOP Classes and corresponding Transfer Syntaxes common to all profiles in this Annex are as follows:
IOD & SOP Class UID Transfer Syntax & UID Required for
FSC FSR FSU
CT Image Explicit VR Little Endian Optional Mandatory Optional
Uncompressed
1.2.840.10008.5.1.4.1.1.2 1.2.840.10008.1.2.1
MR Image Explicit VR Little Endian Optional Mandatory Optional
Uncompressed
1.2.840.10008.5.1.4.1.1.4 1.2.840.10008.1.2.1
The Application Profiles contained in this Annex require the corresponding physical media specified in the appropriate Annex to PS 3.12.
Conformant Application Entities may, but are not required to, include in the DICOMDIR file a Basic Directory IOD containing Directory Records at the Patient and subsidiary levels appropriate to the SOP Classes in the File-set, in which case all DICOM files in the File-set incorporating SOP Instances defined for the specific Application Profile shall be referenced by Directory Records.
NOTE: DICOMDIRs with no directory information are allowed by these Application Profiles.
Any additional keys deemed necessary by the Application Entity may be included in the DICOMDIR, and all File-set Readers must be able to gracefully ignore those that are not recognized. Such additional keys will be specified in the Conformance Statement.
This section defines other parameters common to all specific Application Profiles in the classes in this Annex which need to be specified in order to ensure interoperable information interchange.
The attribute values listed below are used within the CT Image and MR Image files and shall take on values as specified:
Attribute Tag Value
Modality (0008,0060) MR or CT
Rows (0028,0010) <= 1024
Columns (0028,0011) <= 1024
BitsAllocated (0028,0100) 12 or 16
BitsStored (0028,0101) 16
HighBit (0028,????) 11 if BitsAllocated 12, else 15
An FSC or FSU, when creating a File-set, shall generate values outside these ranges. An FSR or FSU, when reading a File-set, shall not be required to read values outside these ranges.
Sequential media utilize the Sequential Media File System. For any such medium, the following rules apply.
Only one DICOM File-set shall be stored on a single piece of sequential medium.
The entire DICOM File-set shall be stored on a single piece of sequential medium.
Files that are not DICOM File-sets may also be recorded.
The recording format for each medium that makes use of the Sequential Media File System shall be specified in a separate Annex.
A volume may be separated into two partitions, if supported by the underlying Physical Medium and allowed by the Media Storage Application Profile that utilizes this Logical Format. If so the two partitions will be referred to as the Data Partition and the Directory Partition. The Data Partition is that which will be visible to a File Set Reader that doesn't support two partitions, but is offered a medium formatted with two partitions.
If the underlying Physical Medium does not support two partitions, then the entire volume will have the format of a Data Partition.
Partitions consists of a series of files separated by tape marks.
The Logical End of Partition is signified either by the presence of two sequential Tape Marks without intervening data, or whatever abstraction for Logical End of Partition is specified at the Physical Media layer for the medium in question.
All data between Tape Marks are stored as a series of blocks each with length equal to the Fixed Block Length, except for the last block, which, when the total length is not an exact multiple of the Fixed Block Length, shall be exactly the length of the remaining data, padded if necessary to be a multiple of some value that is specified for each medium in a separate Annex.
The Fixed Block Length, which shall not be less than 8192 bytes, nor greater than 64,512 bytes, shall be specified for each medium in a separate Annex.
File-set Readers, Creators and Updators must therefore be able to read and write variable length blocks as required.
The presence of a Data Partition is mandatory.
The Data Partition consists of a series of files separated by tape marks. The first four of these files are mandatory and consist of a Volume Header, a Leading Logical File System Directory, and a Data File Header and Leading DICOMDIR file. The last three files before the end of medium are mandatory and consist of a Data File Header and Trailing DICOMDIR file, and a Trailing Logical File System Directory.
Between these files, following the Leading DICOMDIR File, and preceding the Trailing DICOMDIR Data File Header, the Data Files are recorded, each preceded by a Data File Header.
Data Partition Format:
---- Tape Mark Logical End Of Partition - Two Tape Marks ---
| |
-------------------------------------------------------------------------------------------------
VOLHDR | LFSD | HDRD | DICOMDIR | HDR1 | FILE1 | HDR2 | FILE2 | ...... | HDRD | DICOMDIR | LFSD ||
-------------------------------------------------------------------------------------------------
^ ^ ^ ^ ^ ^ ^ ^ ^
| | | | | | | | |
| | | | | | DICOMDIR Data File Header -- | |
| | | | | | Trailing DICOMDIR File ------------ |
| | | | | | Trailing Logical File System Directory -------
| | | | | |
| | | | | --- First Data File
| | | | ---------- First Data File Header
| | | |
| | | |
| | | ---------------------------------- Leading DICOMDIR File
| | ----------------------------------------- DICOMDIR Data File Header
| ----------------------------------------------- Logical File System Directory
------------------------------------------------------- Volume Header
The presence of the Volume Header is mandatory.
The Volume Header consists of 512 bytes in the following fixed format:
0-11 Identifier "DICOMVOLHDR\0" 12-25 Partitions enumerated values "ONEPARTITION\0\0" or "TWOPARTITIONS\0" 26-27 reserved 00H 28-31 FixedBlockLength 32 bit little endian unsigned binary integer 32-511 reserved 00H
If the medium is formatted with a Directory Partition present then the Partitions field will be specified as "TWOPARTITIONS", else it will be specified as "ONEPARTITION".
The FixedBlockLength value is an indication of the maximum size of a block of data that may be read from or written to the tape, and is an upper bound on the size of data transfers to be performed, and does not necessarily correspond to the physical record size on the medium.
The presence of the Leading and Trailing Logical File System Directories (LFSD) is mandatory. The Leading LFSD may be left unused, but if it is present shall be an exact duplicate of the Trailing LFSD. The Directory Partition LFSD, if supported by the medium and if present, shall also be an exact duplicate of the Trailing LFSD. The Media Storage Application Profile may specify that it is permitted that the LFSD not be utilized, in which case the Trailing LFSD shall be present but unused.
If utilized, the LFSD contains sufficient information to determine how much data is recorded on the medium, how many Data Files are present, whether each Data File is a DICOM File-Set or a non-DICOM File, and the correspondence between the Data File Number and the File ID which, in the case of a DICOM File-Set, corresponds to a File ID as contained in the DICOMDIR File.
The LFSD consists of a fixed format header, followed by a series of fixed length File Number Entries, one for each Data File present, listed sequentially in the order in which the Data Files are recorded.
The fixed initial portion of the file is structured as follows:
0-13 Identifier "DICOMMEDIADIR\0" 14-20 UseField enumerated values "INUSE\0\0" or "UNUSED\0" 21-23 reserved 00H 24-27 TotalNumberOfFiles 32 bit little endian unsigned binary integer 28-31 TotalNumberOfDicomFiles 32 bit little endian unsigned binary integer 32-35 DICOMDIRFileNumber 32 bit little endian unsigned binary integer 36-39 TotalBytesInFiles 32 bit little endian unsigned binary integer 40-511 reserved 00H
If the LFSD is not utilized, the UseField will be specified as "UNUSED", the remaining fixed field values will be ignored and should be filled with 00H, and no File Number Entries will be present. Otherwise the UseField will be specified as "INUSE", and all the remaining fields will contain valid values.
Each File Number Entry shall be 128 bytes long and of the form:
0-3 FileNumber 32 bit little endian unsigned binary integer 4-7 FileLengthInBytes 32 bit little endian unsigned binary integer 8-79 FileID 80-85 FileType enumerated values "DICOM\0" or "OTHER\0" 86-127 reserved 00H
The FileNumber, FileID, and FileType values shall have the same values as in the Data File Header (see section (n).1.3.2.4) preceding the Data File referred to.
The FileLengthInBytes in the LFSD must be valid, even though the FileLengthInBytes in the corresponding Data File Header may be zero (unknown). The final record of a Data File may be padded, in which case the FileLengthInBytes value of the File Number Entry in the LFSD may be used to determine the exact length of the data file.
The presence of the Leading and Trailing DICOMDIR files and Data File Headers is mandatory. The Leading DICOMDIR may be unused, but if it is present shall be an exact duplicate of the Trailing DICOMDIR. The Directory Partition DICOMDIR, if supported by the medium and if present, shall also be an exact duplicate of the Trailing DICOMDIR. The Media Storage Application Profile may specify that it is permitted that the DICOMDIR not be utilized, in which case the Trailing DICOMDIR shall be present but unused. If a DICOMDIR is unused, it is of non-zero length, containing only a File-set Identification Module.
It shall conform to the description in PS 3.3 section B.X.3 Basic Directory Information Object Definition.
It shall be recorded as specified in PS 3.10 section 8.6.
The requirement as to the contents of the DICOMDIR, and whether or not it is utilized, shall be defined by Media Storage Application Profile that makes use of the Sequential Media File System.
The Data File Header consists of 512 bytes in the following fixed format:
0-12 Identifier "DICOMFILEHDR\0" 13-15 reserved 00H 16-19 FileNumber 32 bit little endian unsigned binary integer 20-23 FileLengthInBytes 32 bit little endian unsigned binary integer 24-95 FileID 95-100 FileType enumerated values "DICOM\0" or "OTHER\0" 101-511 reserved 00HThe FileNumber is the sequential number of the Data File starting from 1.
The FileLengthInBytes may have the value zero, which indicates that at the time when the Data File Header was written, the length of the Data File was not known.
The FileID shall be a 00H terminated string of up to 8 components, each component up to 8 characters long, drawn from the character set described in PS 3.10 section 8.5, separated by backslashes, without a leading backslash, as described in PS 3.10 section 8.2.
The FileType will have the value "DICOM" if the Data File is a DICOM File-set, otherwise it shall have the value "OTHER".
The Data File shall contain a stream of bytes representing a valid DICOM File as described in PS 3.10 section 7, including the DICOM File Meta Information, and a Data Set containing a single encapsulated SOP Instance, encoded using the Transfer Syntax identified in the DICOM File Meta Information.
The allowable Transfer Syntaxes shall be specified in the Media Storage Application Profile that makes use of the Sequential Media File System.
A Data File may nothave a length of zero bytes, otherwise two Tape Marks would occur sequentially, which would indicate the Logical End of Partition.
It is not permissable for a File Set Reader not to be able to determine the exact length of a Data File. An Application Entity shall expect that if the FileLengthInBytes value in the Data File Header is zero, then either an LFSD is present, utilized, and contains a valid FileLengthInBytes value for this file, or all LFSDs are unused, and exactly the correct number of bytes has been written, and that the last block has not been padded to meet some implementation dependent limitation on physical block size.
The presence of a Directory Partition is optional, and is only permissable if supported by the underlying Physical Medium and allowed by the Media Storage Application Profile that utilizes this Logical Format.
If present, the Directory Partition consists of a series of four mandatory files separated by tape marks, consisting of a Volume Header, a Logical File System Directory, and a Data File Header and DICOMDIR file. Each of these exactly duplicates the Volume Header, Trailing Logical File System Directory, and a Data File Header and Trailing DICOMDIR file of the Data Partition.
The Partitions field in the Volume Header shall have the value "TWOPARTITIONS".
Directory Partition Format:
---- Tape Mark --- Logical End Of Partition - Two Tape Marks
| |
---------------------------------
VOLHDR | LFSD | HDRD | DICOMDIR ||
---------------------------------
^ ^ ^ ^
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | ------- DICOMDIR File
| | -------------- DICOMDIR Data File Header
| -------------------- Logical File System Directory
---------------------------- Volume Header
Only one DICOM File-set shall be stored on a single piece of sequential medium.
The media format comprises two distinct components:
The magnetic recording format for each of the following media:
shall comply with the formats specified in:
The magnetic recording format for each of the following media:
shall comply with the formats specified in:
The magnetic recording format for each of the following media:
shall comply with the formats specified in:
The magnetic recording format for each of the following media:
shall comply with the formats specified in:
Note that ANSI Standard X3.27 "Magnetic Tape Labels and File Structure for Information Interchange" is not utilized by this standard.
The Fixed Block Length is an indication of the maximum size of a block of data that may be read from or written to the tape, and is an upper bound on the size of data transfers to be performed, and does not necessarily correspond to the physical record size on the medium.
Application Entities supporting this physical medium must be able to read and write media that support physical record lengths up to but not exceeding the Fixed Block Length.
The Fixed Block Length for each medium shall be as follows:
The largest block sizes are chosen to be the largest multiple of 1024 < 65,535, as some Application Entities may have difficulty reading and writing blocks that are larger than this.
Drives for DAT media must support variable-length record mode.
Drives for 0.5" 9-track media must support variable-length record mode.
Drives for QIC media will not accept blocks that are not multiples of 512 bytes, and hence short blocks will be padded to this length. These blocks will be written as a series of fixed 512 byte physical records.
Drives for 8mm media will not accept blocks that are not multiples of 1024 bytes, and hence short blocks will be padded to this length.
No such padding shall be performed for other forms of media.
DAT media may optionally utilize two partitions, if supported by the file system and Application Profile that make reference to this Annex, in which case the first physical partition, Partition 1, on the tape will be formatted with a length of 10 Megabytes, and the second partition, Partion 0 will occupy the remainder of the medium.
Partition 0 will be the partition visible to File-set Readers that do not have the capability of reading multiple partitions.
A File-set Creator may choose not to, or not be capable of, formatting DAT media with multiple partitions, in which case only a single partition shall be created.
File-set Updaters that cannot utilize multiple partitions shall not update DAT media formatted with multiple partitions.
The physical media shall be as specified in those standards referred to in section (n+1).2.3 Recording Format.