Tutorial | Background and Further Reading: |
What is NetCDF? | NetCDF is a set of data formats, programming interfaces, and software libraries that help read and write scientific data files |
The NetCDF Utilities | The ncdump command reads a netCDF file and outputs text in a format called CDL |
Tools for Manipulating NetCDF Files | Many existing software applications can read and manipulate netCDF files |
The NetCDF Programming APIs | Unidata supports netCDF APIs in C, C++, Fortran 77, Fortran 90, and Java |
NetCDF Documentation | Language specific programming guides are available for netCDF for the C, C++, Fortran 77, Fortran 90, and Java APIs: |
netcdf_data_model | The netCDF data model is the way that we think about data |
The Classic Model | The classic netCDF data model consists of variables, dimensions, and attributes |
The Enhanced Data Model | With netCDF-4, the netCDF data model has been extended, in a backwards compatible way |
Fill Values | Sometimes there are missing values in the data, and some value is needed to represent them |
Error Handling | Each netCDF function in the C, Fortran 77, and Fortran 90 APIs returns 0 on success, in the tradition of C |
Unlimited Dimensions | Sometimes you don't know the size of all dimensions when you create a file, or you would like to arbitrarily extend the file along one of the dimensions |
NetCDF Example Programs | The netCDF example programs show how to use netCDF |
The simple_xy Example | This example is an unrealistically simple netCDF file, to demonstrate the minimum operation of the netCDF APIs |
The sfc_pres_temp Example | This example has been constructed for the meteorological mind |
The pres_temp_4D Example | This example expands on the previous example by making our two-dimensional data into four-dimensional data, adding a vertical level axis and an unlimited time step axis |
The simple_nc4 Example | This example, like the simple_xy netCDF-3 example above, is an overly simplified example which demonstrates how to use groups in a netCDF-4 file |
The simple_xy_nc4 Example | This example, like the simple_xy netCDF-3 example above, is an overly simplified example |
Interoperability with HDF5 | NetCDF-4 allows some interoperability with HDF5 |
Groups | NetCDF-4 files can store attributes, variables, and dimensions in hierarchical groups |
User Defined Types | NetCDF supported six data types through version 3.6.0 (char, byte, short, int, float, and double) |
Strings | Use the NC_STRING type to store arrays of strings |
Parallel I/O with NetCDF-4 | NetCDF-4 provides access to HDF5 parallel I/O features for netCDF-4/HDF5 files |
Numbering of NetCDF IDs | In C, Fortran 77, and Fortran 90, netCDF objects are identified by an integer: the ID |
Creating New Files and Metadata, an Overview | To construct a netCDF file you need to: |
Reading NetCDF Files of Known Structure | To read a netCDF file of known structure, you need to: |
Reading NetCDF Files of Unknown Structure | Perhaps you would like to write your software to handle more general cases, so that you don't have to adjust your source every time the grid size changes, or a variable is added to the file |
Reading and Writing Subsets of Data | Usually users are interested in reading or writing subsets of variables in a netCDF data file |
Getting NetCDF | The easiest way to get netCDF is through a package management program, such as rpm, yum, adept, and others |
Building NetCDF | The netCDF-C library and utilities require third-party libraries for full functionality |
Building with NetCDF-4 and the Remote Data Client | The usual way of building netCDF requires the HDF5, zlib, and curl libraries |
Building NetCDF with Classic Library Only | It is possible to build the netCDF C libraries and utilities so that only the netCDF classic and 64-bit offset formats are supported, or the remote data access client is not built |
Building with HDF4 Support | The netCDF-4 library can (since version 4.1) read HDF4 data files, if they were created with the SD (Scientific Data) API |
Building with Parallel I/O Support | For parallel I/O to work, HDF5 must be installed with –enable-parallel, and an MPI library (and related libraries) must be made available to the HDF5 configure |
Linking to NetCDF | For static build, to use netCDF-4 you must link to all the libraries, netCDF, HDF5, zlib, szip (if used with HDF5 build), and curl (if the remote access client has not been disabled) |
./configure options | Note: –disable prefix indicates that the option is normally enabled |
Building-with-CMake | |
Build Instructions for netCDF-C using CMake | |
windows-binaries | |
Installing and Using netCDF-C Libraries in a Windows Environment | |
The NetCDF Users Guide | Summary |
Summary | The purpose of the Network Common Data Form (netCDF) interface is to allow you to create, access, and share array-oriented data in a form that is self-describing and portable |
The NetCDF Interface | The Network Common Data Form, or netCDF, is an interface to a library of data access functions for storing and retrieving data in the form of arrays |
The netCDF File Format | Until version 3.6.0, all versions of netCDF employed only one binary data format, now referred to as netCDF classic format |
What about Performance? | One of the goals of netCDF is to support efficient access to small subsets of large datasets |
Creating Self-Describing Data conforming to Conventions | The mere use of netCDF is not sufficient to make data "self-describing" and meaningful to both humans and machines |
Limitations of NetCDF | The netCDF classic data model is widely applicable to data that can be organized into a collection of named array variables with named attributes, but there are some important limitations to the model and its implementation in software |
The Data Model | A netCDF dataset contains dimensions, variables, and attributes, which all have both a name and an ID number by which they are identified |
Name | |
Is NetCDF a Good Archive Format? | NetCDF classic or 64-bit offset formats can be used as a general-purpose archive format for storing arrays |
Attribute Conventions | Attribute conventions are assumed by some netCDF generic applications, e.g., ‘units’ as the name for a string attribute that gives the units for a netCDF variable |
Background and Evolution of the NetCDF Interface | The development of the netCDF interface began with a modest goal related to Unidata's needs: to provide a common interface between Unidata applications and real-time meteorological data |
The Remote Data Access Client | |
Data Access | To access (read or write) netCDF data you specify an open netCDF dataset, a netCDF variable, and information (e.g., indices) identifying elements of the variable |
Dimensions | A dimension may be used to represent a real physical dimension, for example, time, latitude, longitude, or height |
Variables | Variables are used to store the bulk of the data in a netCDF dataset |
Coordinate Variables | It is legal for a variable to have the same name as a dimension |
Attributes | NetCDF attributes are used to store data about the data (ancillary data or metadata), similar in many ways to the information stored in data dictionaries and schema in conventional database systems |
Differences between Attributes and Variables | In contrast to variables, which are intended for bulk data, attributes are intended for ancillary data, or information about the data |
Parts of a NetCDF Classic File | A netCDF classic or 64-bit offset dataset is stored as a single file comprising two parts: |
Parts of a NetCDF-4 HDF5 File | NetCDF-4 files are created with the HDF5 library, and are HDF5 files in every way, and can be read without the netCDF-4 interface |
The Extended XDR Layer | XDR is a standard for describing and encoding data and a library of functions for external data representation, allowing programmers to encode data structures in a machine-independent way |
Large File Support | It is possible to write netCDF files that exceed 2 GiByte on platforms that have "Large File Support" (LFS) |
NetCDF 64-bit Offset Format Limitations | Although the 64-bit offset format allows the creation of much larger netCDF files than was possible with the classic format, there are still some restrictions on the size of variables |
NetCDF Classic Format Limitations | There are important constraints on the structure of large netCDF classic files that result from the 32-bit relative offsets that are part of the netCDF classic file format: |
The NetCDF-3 I/O Layer | The following discussion applies only to netCDF classic and 64-bit offset files |
Parallel Access with NetCDF-4 | Use the special parallel open (or create) calls to open (or create) a file, and then to use parallel I/O to read or write that file (see nc_open_par()) |
Interoperability with HDF5 | To create HDF5 files that can be read by netCDF-4, use the latest in the HDF5 1.8.x series |
DAP Support | Beginning with netCDF version 4.1, optional support is provided for accessing data through OPeNDAP servers using the DAP protocol |
The Chunk Cache | When data are first read or written to a netCDF-4/HDF5 variable, the HDF5 library opens a cache for that variable |
The Default Chunking Scheme in version 4.1 | (and 4.1.1) |
The Default Chunking Scheme in version 4.0.1 | In the 4.0.1 release, the default chunk sizes were chosen with a different scheme, as demonstrated in the following C code: |
Chunking and Parallel I/O | When files are opened for read/write parallel I/O access, the chunk cache is not used |
A Utility to Help Benchmark Results: bm_file | The bm_file utility may be used to copy files, from one netCDF format to another, changing chunking, filter, parallel I/O, and other parameters |
CDL Syntax | Below is an example of CDL, describing a netCDF classic format file with several named dimensions (lat, lon, time), variables (z, t, p, rh, lat, lon, time), variable attributes (units, _FillValue, valid_range), and some data |
CDL Data Types | The CDL primitive data types for the classic model are: |
CDL Notation for Data Constants | This section describes the CDL notation for constants |
ncgen | The ncgen tool generates a netCDF file or a C or FORTRAN program that creates a netCDF dataset |
ncdump | The ncdump utility generates a text representation of a specified netCDF file on standard output, optionally excluding some or all of the variable data in the output |
nccopy | The nccopy utility copies an input netCDF file in any supported format variant to an output netCDF file, optionally converting the output to any compatible netCDF format variant, compressing the data, or rechunking the data |
ncgen3 | The ncgen3 tool is the new name for the older, original ncgen utility |
The NetCDF Classic Format Specification | To present the format more formally, we use a BNF grammar notation |
Notes on Computing File Offsets | The offset (position within the file) of a specified data value in a classic format or 64-bit offset data file is completely determined by the variable start location (the offset in the begin field), the external type of the variable (the nc_type field), and the dimension indices (one for each of the variable's dimensions) of the value desired |
Examples | By using the grammar above, we can derive the smallest valid netCDF file, having no dimensions, no variables, no attributes, and hence, no data |
The 64-bit Offset Format | The netCDF 64-bit offset format differs from the classic format only in the VERSION byte, ‘\x02’ instead of ‘\x01’, and the OFFSET entity, a 64-bit instead of a 32-bit offset from the beginning of the file |
The NetCDF-4 Format | The netCDF-4 format implements and expands the netCDF-3 data model by using an enhanced version of HDF5 as the storage layer |
The NetCDF-4 Classic Model Format | Every classic and 64-bit offset file can be represented as a netCDF-4 file, with no loss of information |
HDF4 SD Format | Starting with version 4.1, the netCDF libraries can read HDF4 SD (Scientific Dataset) files |
Data Type | |
External Data Types | The atomic external types supported by the netCDF interface are: |
Data Structures in Classic and 64-bit Offset Files | The only kind of data structure directly supported by the netCDF classic (and 64-bit offset) abstraction is a collection of named arrays with attached vector attributes |
Type Conversion | Each netCDF variable has an external type, specified when the variable is first defined |
Notes | |
Ignored if NULL | Many of the argurments of netCDF functions are pointers |
Specify a Hyperslab | The NetCDF allows specification of hyperslabs to be read or written with vectors which specify the start, count, stride, and mapping |
NetCDF ID | Most netCDF function require the netCDF ID as a first parameter |
CDL | |
NetCDF Library Architecture | |
Copyright | Copyright 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 University Corporation for Atmospheric Research/Unidata |
\b ncdump tool - Convert netCDF file to text form (CDL) | |
\b nccopy tool - Copy a netCDF file, optionally changing format, compression, or chunking in the output. | |
Deprecated List | |