next up previous
Next: Example: Data Compression
Up: ADDITIONAL FACILITIES
Previous: Specifying Where the Native NDF is Stored

Efficiency Considerations  

When deciding where to store the native NDF format copy of a dataset, it is wise to specify a location on a local file system wherever possible. This is, of course, always good practice where large datasets are concerned, as access to remote files is usually far less efficient and can generate considerable network traffic that may interfere with other people's work.

With NDF format conversion facilities, local file access is even more important. This is because the temporary datasets involved are always read immediately after being written, and very frequently deleted immediately after that. In this situation, an operating system with good file caching will often not actually write the data to a local file at all, but merely copy it to and from memory. This is far faster than waiting for actual data transfer to take place, which is what will normally happen if remote files are involved.

For this reason, you are recommended to configure format conversion software so that temporary datasets are stored in the user's default directory, in the expectation that this directory, at least, will be chosen sensibly and reside on a local file system. Users may, however, still need to be reminded of the need for this (e.g. in documentation). You may also need to explain how to change this behaviour if, for example, access to larger amounts of space for temporary files becomes necessary.

Note that, by default, temporary NDF datasets are stored in the standard HDS scratch file, which resides in a directory specified by the HDS_SCRATCH environment variable. If this variable is not explicitly set, the user's default directory is used.



next up previous
Next: Example: Data Compression
Up: ADDITIONAL FACILITIES
Previous: Specifying Where the Native NDF is Stored


Starlink System Note 20
R.F.Warren-Smith & D.S.Berry
17th July 2000
E-mail:ussc@star.rl.ac.uk

Copyright © 2000 Council for the Central Laboratory of the Research Councils