next up previous
Next: Chunking
Up: NDF SECTIONS
Previous: Restrictions on Mapped Access to Sections

More Advanced Use: Partitioning   

In times gone by, when computers had little memory, it was common for image-processing applications to read their data one line at a time, and to operate on that line before passing on to the next. Nowadays, there is little need for this approach and the programming complications it causes, because reasonably-sized images can be accommodated entirely in memory.

Nevertheless, limitations on the available memory can still be important in some situations. For instance, when processing extremely large files, it is still possible to find that the memory available (or the memory quota allocated by your system manager) is insufficient. Even when it appears possible to accommodate a large array in memory, it is wise to remember that with a virtual memory operating system the information may not actually reside in the physical memory of the computer, and this may result in substantial inefficiencies.

This sort of consideration can be ignored for most types of work, but steps must sometimes be taken to reduce the amount of memory required when accessing large NDFs. This will often result in improvements in efficiency even if actual memory limitations are unimportant, because a reduction in memory requirements in general tends to make more memory available for other system activities, which therefore run more efficiently.

To allow memory usage to be limited, the NDF_ system provides facilities for partitioning NDFs so that they may be processed in pieces. Two partitioning methods are available, termed chunking and blocking. Chunking is appropriate when the NDF can be regarded simply as a 1-dimensional sequence of values (i.e. when the actual shape is unimportant) while blocking is used if the shape and positional relationship between the pixels is significant. Both of these techniques work by dividing an NDF into sections. They are described in turn below.

Note that these techniques may also have applications in parallel processing, where it is often necessary to partition a large array and then to pass the resulting pieces to separate processors.



next up previous
Next: Chunking
Up: NDF SECTIONS
Previous: Restrictions on Mapped Access to Sections


Starlink User Note 33
R.F. Warren-Smith
11th January 2000
E-mail:rfws@star.rl.ac.uk

Copyright © 2000 Council for the Central Laboratory of the Research Councils