NetCDF  4.6.3
 All Data Structures Files Functions Variables Typedefs Macros Modules Pages
inmemory.md
1 NetCDF In-Memory Support
2 ====================================
3 
4 <!-- double header is needed to workaround doxygen bug -->
5 
6 NetCDF In-Memory Support {#inmemory}
7 ====================================
8 
9 [TOC]
10 
11 Introduction {#inmemory_intro}
12 --------------
13 
14 It can be convenient to operate on a netcdf file whose
15 content is held in memory instead of in a disk file.
16 The netcdf API has been modified in a number of ways
17 to support this capability.
18 
19 Actually, three distinct but related capabilities are provided.
20 
21 1. DISKLESS -- Read a file into memory, operate on it, and optionally
22 write it back out to disk when nc_close() is called.
23 2. INMEMORY -- Tell the netcdf-c library to treat a provided block
24 of memory as if it were a netcdf file. At close, it is possible to ask
25 for the final contents of the memory chunk. Be warned that there is
26 some complexity to this as described below.
27 4. MMAP -- (deprecated) Tell the netcdf-c library to use the *mmap()* operating
28 system functionality to access a file.
29 
30 The first two capabilities are intertwined in the sense that the
31 *diskless* capability makes use internally of the *inmemory*
32 capability (for netcdf classic only). But, the *inmemory*
33 capability can be used independently of the *diskless*
34 capability.
35 
36 The *mmap()* capability provides a capability similar to *diskless* but
37 using special capabilities of the underlying operating system. It turns out
38 that the mmap capability has seen no significant use, so its use is deprecated
39 and will be removed at some point in the future.
40 
41 Note also that *diskless* and *inmemory* can be used for both
42 *netcdf-3* (classic) and *netcdf-4* (enhanced) data. The *mmap*
43 capability can only be used with *netcdf-3*.
44 
45 Enabling Diskless File Access {#Enable_Diskless}
46 --------------
47 The *diskless* capability can be used relatively transparently
48 using the *NC_DISKLESS* mode flag.
49 
50 Note that since the file is stored in memory, size limitations apply.
51 If you are on using a 32-bit pointer then the file size must be less than 2^32
52 bytes in length. On a 64-bit machine, the size must be less than 2^64 bytes.
53 
54 Also note that for a diskless file, there are two notions of
55 *write* with respect to the file. The first notion is that the
56 file is read-only through the netCDF API. For example, if the file
57 is read-only, then a call to, for example, _nc_def_dim()_ will fail.
58 The second notion of *write* refers to the file on disk to which
59 the contents of memory might be persisted.
60 
61 WARNING: control of the two kinds of *write* has changed since
62 release 4.6.1.
63 
64 The mode flag NC_WRITE determines the first kind of *write*.
65 If set, then NC_WRITE means that the file can be modified through
66 the netCDF API, otherwise it is read-only. This is a change since
67 release 4.6.1.
68 
69 The new mode flag NC_PERSIST now determines the second kind of
70 *write*. If set, then NC_PERSIST means that the memory contents
71 will be persisted to disk, possibly overwriting the previous
72 file contents. Otherwise, the default is to throw away the
73 in-memory contents.
74 
75 ### Diskless File Open
76 Calling *nc_open()* using the mode flag *NC_DISKLESS* will cause
77 the file being opened to be read into memory. When calling *nc_close()*,
78 the file will optionally be re-written (aka "persisted") to disk. This
79 persist capability will be invoked if and only if *NC_PERSIST* is specified
80 in the mode flags at the call to *nc_open()*.
81 
82 ### Diskless File Create
83 Calling *nc_create()* using the mode flag *NC_DISKLESS* will cause
84 the file to initially be created and kept in memory.
85 When calling *nc_close()*, the file will be written
86 to disk if and only if *NC_PERSIST* is specified
87 in the mode flags at the call to *nc_create()*.
88 
89 Enabling Inmemory File Access {#Enable_Inmemory}
90 --------------
91 
92 The netcdf API has been extended to support the inmemory capability.
93 The relevant API is defined in the file `netcdf_mem.h`.
94 
95 The important data structure to use is `NC_memio`.
96 ````
97 typedef struct NC_memio {
98  size_t size;
99  void* memory;
100  int flags;
101 } NC_memio;
102 
103 ````
104 An instance of this data structure is used when providing or
105 retrieving a block of data. It specifies the memory and its size
106 and also some relevant flags that define how to manage the memory.
107 
108 Current only one flag is defined -- *NC_MEMIO_LOCKED*.
109 This tells the netcdf library that it should never try to
110 *realloc()* the memory nor to *free()* the memory. Note
111 that this does not mean that the memory cannot be modified, but
112 only that the modifications will be within the confines of the provided
113 memory. If doing such modifications is impossible without
114 reallocating the memory, then the modification will fail.
115 
116 ### In-Memory API
117 
118 The new API consists of the following functions.
119 ````
120 int nc_open_mem(const char* path, int mode, size_t size, void* memory, int* ncidp);
121 
122 int nc_create_mem(const char* path, int mode, size_t initialsize, int* ncidp);
123 
124 int nc_open_memio(const char* path, int mode, NC_memio* info, int* ncidp);
125 
126 int nc_close_memio(int ncid, NC_memio* info);
127 
128 ````
129 ### The **nc_open_mem** Function
130 
131 The *nc_open_mem()* function is actually a convenience
132 function that internally invokes *nc_open_memio()*.
133 It essentially provides simple read-only access to a chunk of memory
134 of some specified size.
135 
136 ### The **nc_open_memio** Function
137 
138 This function provides a more general read/write capability with respect
139 to a chunk of memory. It has a number of constraints and its
140 semantics are somewhat complex. This is primarily due to limitations
141 imposed by the underlying HDF5 library.
142 
143 The constraints are as follows.
144 
145 1. If the *NC_MEMIO_LOCKED* flag is set, then the netcdf library will
146 make no attempt to reallocate or free the provided memory.
147 If the caller invokes the *nc_close_memio()* function to retrieve the
148 final memory block, it should be the same
149 memory block as was provided when *nc_open_memio* was called.
150 Note that it is still possible to modify the in-memory file if the NC_WRITE
151 mode flag was set. However, failures can occur if an operation
152 cannot complete because the memory needs to be expanded.
153 2. If the *NC_MEMIO_LOCKED* flag is <b>not</b> set, then
154 the netcdf library will take control of the incoming memory.
155 This means that the user should not make any attempt to free
156 or even read the incoming memory block in this case.
157 The newcdf library is free to reallocate the incomming
158 memory block to obtain a larger block when an attempt to modify
159 the in-memory file requires more space. Note that implicit in this
160 is that the old block -- the one originally provided -- may be
161 free'd as a side effect of re-allocating the memory using the
162 *realloc()* function.
163 The caller may invoke the *nc_close_memio()* function to retrieve the
164 final memory block, which may not be the same as the originally block
165 provided by the caller. In any case, the returned block must always be freed
166 by the caller and the original block should not be freed.
167 
168 ### The **nc_create_mem** Function
169 
170 This function allows a user to create an in-memory file, write to it,
171 and then retrieve the final memory using *nc_close_memio()*.
172 The *initialsize* argument to *nc_create_mem()* tells the library
173 how much initial memory to allocate. Technically, this is advisory only
174 because it may be ignored by the underlying HDF5 library.
175 It is used, however, for netcdf-3 files.
176 
177 ### The **nc_close_memio** Function
178 
179 The ordinary *nc_close()* function can be called to close an in-memory file.
180 However, it is often desirable to obtain the final size and memory block
181 for the in-memory file when that file has been modified.
182 The *nc_close_memio()* function provides a means to do this.
183 Its second argument is a pointer to an *NC_memio* object
184 into which the final memory and size are stored. WARNING,
185 the returned memory is owned by the caller and so the caller
186 is responsible for calling *free()* on that returned memory.
187 
188 ### Support for Writing with *NC_MEMIO_LOCKED*
189 
190 When the NC_MEMIO_LOCKED flag is set in the *NC_memio* object
191 passed to *nc_open_memio()*, it is still possible to modify
192 the opened in-memory file (using the NC_WRITE mode flag).
193 
194 The big problem is that any changes must fit into the memory provided
195 by the caller via the *NC_memio* object. This problem can be
196 mitigated, however, by using the "trick" of overallocating
197 the caller supplied memory. That is, if the original file is, say, 300 bytes,
198 then it is possible to allocate, say, 65000 bytes and copy the original file
199 into the first 300 bytes of the larger memory block. This will allow
200 the netcdf-c library to add to the file up to that 65000 byte limit.
201 In this way, it is possible to avoid memory reallocation while still
202 allowing modifications to the file. You will still need to call
203 *nc_close_memio()* to obtain the size of the final, modified, file.
204 
205 Enabling MMAP File Access (Deprecated) {#Enable_MMAP}
206 --------------
207 
208 The MMAP functionality is deprecated.
209 
210 Some operating systems provide a capability called MMAP.
211 This allows disk files to automatically be mapped to chunks of memory.
212 It operates in a fashion somewhat similar to operating system virtual
213 memory, except with respect to a file.
214 
215 By setting mode flag NC_MMAP, it is possible to do the equivalent
216 of NC_DISKLESS but using the operating system's mmap capabilities.
217 
218 Currently, MMAP support is only available when using netcdf-3 or cdf5
219 files.
220 
221 Known Bugs {#Inmemory_Bugs}
222 --------------
223 
224 1. If you are modifying a locked memory chunk (using
225  NC_MEMIO_LOCKED) and are accessing it as a netcdf-4 file, and
226  you overrun the available space, then the HDF5 library will
227  fail with a segmentation fault.
228 
229 2. You will get an HDF5 error under the following conditions.
230 
231  1. You call nc_open on a file with the flags NC_DISKLESS|NC_WRITE
232  but without NC_PERSIST.
233  2. The file to be read is read-only (i.e. mode 0444).
234 
235  Note that this should be ok because the modifications to the file
236  are not intended to pushed back into the disk file. However, the
237  HDF5 core driver does not allow this.
238 
239 References {#Inmemory_References}
240 --------------
241 
242 1. https://support.hdfgroup.org/HDF5/doc1.8/Advanced/FileImageOperations/HDF5FileImageOperations.pdf
243 
244 Point of Contact
245 --------------
246 
247 __Author__: Dennis Heimbigner<br>
248 __Email__: dmh at ucar dot edu
249 __Initial Version__: 2/3/2018<br>
250 __Last Revised__: 2/5/2018
251 
252 

Return to the Main Unidata NetCDF page.
Generated on Sat Apr 6 2019 08:19:00 for NetCDF. NetCDF is a Unidata library.