1 NetCDF Authorization Support
2 ======================================
4 <!-- double header is needed to workaround doxygen bug -->
6 NetCDF Authorization Support {#auth}
7 ====================================
11 ## Introduction {#auth_intro}
13 netCDF can support user authorization using the facilities provided by the curl
14 library. This includes basic password authentication as well as
15 certificate-based authorization.
17 At the moment, this document only applies to DAP2 and DAP4 access
18 because they are (for now) the only parts of the netCDF-C library
21 With some exceptions (e.g. see the section on <a href="#REDIR">redirection</a>)
22 The libcurl authorization mechanisms can be accessed in two ways
24 1. Inserting the username and password into the url, or
25 2. Accessing information from a so-called _rc_ file named either
28 ## URL-Based Authentication {#auth_url}
30 For simple password based authentication, it is possible to
31 directly insert the username and the password into a url in this form.
33 http://username:password@host/...
35 This username and password will be used if the server asks for
36 authentication. Note that only simple password authentication
37 is supported in this format.
39 Specifically note that [redirection-based](#REDIR)
40 authorization may work with this but it is a security risk.
41 This is because the username and password
42 may be sent to each server in the redirection chain.
44 Note also that the `user:password` form may contain characters that must be
45 escaped. See the <a href="#USERPWDESCAPE">password escaping</a> section to see
46 how to properly escape the user and password.
48 ## RC File Authentication {#auth_dodsrc}
49 The netcdf library supports an _rc_ file mechanism to allow the passing
50 of a number of parameters to libnetcdf and libcurl.
51 Locating the _rc_ file is a multi-step process.
55 The file must be called one of the following names:
56 ".daprc" or ".dodsrc".
57 If both ".daprc" and ".dodsrc" exist, then
58 the ".daprc" file will take precedence.
60 It is strongly suggested that you pick one of the two names
61 and use it always. Otherwise you may observe unexpected results
62 when the netcdf-c library finds one that you did not intend.
64 The search for an _rc_ file looks in the following places in this order.
66 1. Check for the environment variable named _DAPRCFILE_.
67 This will specify the full path for the _rc_ file
68 (not just the containing directory).
69 2. Search the current working directory (`./`) looking
70 for (in order) .daprc or .dodsrc.
71 3. Search the HOME directory (`$HOME`) looking
72 for (in order) .daprc or .dodsrc. The HOME environment
73 variable is used to define the directory in which to search.
75 It is strongly suggested that you pick a uniform location
76 and use it always. Otherwise you may observe unexpected results
77 when the netcdf-c library get an rc file you did not expect.
81 The rc file format is a series of lines of the general form:
83 [<host:port>]<key>=<value>
85 where the bracket-enclosed host:port is optional.
87 ### URL Constrained RC File Entries
89 Each line of the rc file can begin with
90 a host+port enclosed in square brackets.
91 The form is "host:port".
92 If the port is not specified
93 then the form is just "host".
94 The reason that more of the url is not used is that
95 libcurl's authorization grain is not any finer than host level.
99 [remotetest.unidata.ucar.edu]HTTP.VERBOSE=1
103 [fake.ucar.edu:9090]HTTP.VERBOSE=0
105 If the url request from, say, the _netcdf_open_ method
106 has a host+port matching one of the prefixes in the rc file, then
107 the corresponding entry will be used, otherwise ignored.
108 This means that an entry with a matching host+port will take
109 precedence over an entry without a host+port.
113 http://remotetest.unidata.ucar.edu/thredds/dodsC/testdata/testData.nc
115 will have HTTP.VERBOSE set to 1 because its host matches the example above.
119 http://fake.ucar.edu:9090/dts/test.01
121 will have HTTP.VERBOSE set to 0 because its host+port matches the example above.
123 ## Authorization-Related Keys {#auth_keys}
125 The currently defined set of authorization-related keys are as follows.
126 The second column is the affected curl_easy_setopt option(s), if any
129 <tr><th>Key</th><th>Affected curl_easy_setopt Options</th><th>Notes</th>
130 <tr><td>HTTP.COOKIEJAR</td><td>CURLOPT_COOKIEJAR</td>
131 <tr><td>HTTP.COOKIEFILE</td><td>CURLOPT_COOKIEJAR</td><td>Alias for CURLOPT_COOKIEJAR</td>
132 <tr><td>HTTP.PROXY.SERVER</td><td>CURLOPT_PROXY, CURLOPT_PROXYPORT, CURLOPT_PROXYUSERPWD</td>
133 <tr><td>HTTP.PROXY_SERVER</td><td>CURLOPT_PROXY, CURLOPT_PROXYPORT, CURLOPT_PROXYUSERPWD</td><td>Decprecated: use HTTP.PROXY.SERVER</td>
134 <tr><td>HTTP.SSL.CERTIFICATE</td><td>CURLOPT_SSLCERT</td>
135 <tr><td>HTTP.SSL.KEY</td><td>CURLOPT_SSLKEY</td>
136 <tr><td>HTTP.SSL.KEYPASSWORD</td><td>CURLOPT_KEYPASSWORD</td>
137 <tr><td>HTTP.SSL.CAINFO</td><td>CURLOPT_CAINFO</td>
138 <tr><td>HTTP.SSL.CAPATH</td><td>CURLOPT_CAPATH</td>
139 <tr><td>HTTP.SSL.VERIFYPEER</td><td>CURLOPT_SSL_VERIFYPEER</td>
140 <tr><td>HTTP.SSL.VALIDATE</td><td>CURLOPT_SSL_VERIFYPEER, CURLOPT_SSL_VERIFYHOST</td>
141 <tr><td>HTTP.CREDENTIALS.USERPASSWORD</td><td>CURLOPT_USERPASSWORD</td>
142 <tr><td>HTTP.CREDENTIALS.USERNAME</td><td>CURLOPT_USERNAME</td>
143 <tr><td>HTTP.CREDENTIALS.PASSWORD</td><td>CURLOPT_PASSWORD</td>
144 <tr><td>HTTP.NETRC</td><td>N.A.</td><td>Specify path of the .netrc file</td>
147 ### Password Authentication
150 HTTP.CREDENTIALS.USERPASSWORD
151 can be used to set the simple password authentication.
152 This is an alternative to setting it in the url.
153 The value must be of the form "username:password".
154 See the <a href="#USERPWDESCAPE">password escaping</a> section
155 to see how this value must escape certain characters.
156 Also see <a href="#REDIR">redirection authorization</a>
157 for important additional information.
160 HTTP.CREDENTIALS.USERNAME and HTTP.CREDENTIALS.PASSWORD
161 can be used as an alternative to HTTP.CREDENTIALS.USERPASSWORD
162 to set the simple password authentication.
163 If present, they take precedence over HTTP.CREDENTIALS.USERPASSWORD.
164 The values do not need to be escaped.
165 See <a href="#REDIR">redirection authorization</a>
166 for important additional information.
170 The HTTP.COOKIEJAR key
171 specifies the name of file from which
172 to read cookies (CURLOPT_COOKIEJAR) and also
173 the file into which to store cookies (CURLOPT_COOKIEFILE).
174 The same value is used for both CURLOPT values.
175 It defaults to in-memory storage.
176 See [redirection authorization](#REDIR)
177 for important additional information.
179 ### Certificate Authentication
182 specifies a file path for a file containing a PEM cerficate.
183 This is typically used for client-side authentication.
185 HTTP.SSL.KEY is essentially the same as HTTP.SSL.CERTIFICATE
186 and should always have the same value.
189 specifies the password for accessing the HTTP.SSL.CERTIFICAT/HTTP.SSL.key file.
192 specifies the path to a directory containing
193 trusted certificates for validating server certificates.
194 See reference #2 for more info.
197 is a boolean (1/0) value that if true (1)
198 specifies that the client should verify the server's presented certificate.
201 specifies the url for accessing the proxy:
202 e.g. *http://[username:password@]host[:port]*
205 deprecated; use HTTP.PROXY.SERVER
208 specifies the absolute path of the .netrc file.
209 See [redirection authorization](#REDIR)
210 for information about using .netrc.
212 ## Password Escaping {#auth_userpwdescape}
214 With current password rules, it is is not unlikely that the password
215 will contain characters that need to be escaped. Similarly, the user
216 may contain characters such as '@' that need to be escaped. To support this,
217 it is assumed that all occurrences of `user:password` use URL (i.e. %%XX)
218 escaping for at least the characters in the table below.
220 The minimum set of characters that must be escaped depends on the location.
221 If the user+pwd is embedded in the URL, then '@' and ':' __must__ be escaped.
222 If the user+pwd is the value for
223 the HTTP.CREDENTIALS.USERPASSWORD key in the _rc_ file, then
224 ':' __must__ be escaped.
225 Escaping should __not__ be used in the `.netrc` file nor in
226 HTTP.CREDENTIALS.USERNAME or HTTPCREDENTIALS.PASSWORD.
228 The relevant escape codes are as follows.
230 <tr><th>Character</th><th>Escaped Form</th>
231 <tr><td>'@'</td><td>%40</td>
232 <tr><td>':'</td><td>%3a</td>
234 Additional characters can be escaped if desired.
236 ## Redirection-Based Authentication {#auth_redir}
238 Some sites provide authentication by using a third party site
239 to do the authentication. Examples include ESG, URS, RDA, and most oauth2-based
242 The process is usually as follows.
244 1. The client contacts the server of interest (SOI), the actual data provider
245 using, typically _http_ protocol.
246 2. The SOI sends a redirect to the client to connect to the e.g. URS system
247 using the _https_ protocol (note the use of _https_ instead of _http_).
248 3. The client authenticates with URS.
249 4. URS sends a redirect (with authorization information) to send
250 the client back to the SOI to actually obtain the data.
252 It turns out that libcurl, by default, uses the password in the
253 `.daprc` file (or from the url) for all connections that request
254 a password. This causes problems because only the the specific
255 redirected connection is the one that actually requires the password.
256 This is where the `.netrc` file comes in. Libcurl will use `.netrc`
257 for the redirected connection. It is possible to cause libcurl
258 to use the `.daprc` password always, but this introduces a
259 security hole because it may send the initial user+pwd to every
260 server in the redirection chain.
261 In summary, if you are using redirection, then you are
262 ''strongly'' encouraged to create a `.netrc` file to hold the
263 password for the site to which the redirection is sent.
265 The format of this `.netrc` file will contain lines that
266 typically look like this.
268 machine mmmmmm login xxxxxx password yyyyyy
270 where the machine, mmmmmm, is the hostname of the machine to
271 which the client is redirected for authorization, and the
272 login and password are those needed to authenticate on that machine.
274 The location of the `.netrc` file can be specified by
275 putting the following line in your `.daprc`/`.dodsrc` file.
277 HTTP.NETRC=<path to netrc file>
279 If not specified, then libcurl will look first in the current
280 directory, and then in the HOME directory.
282 One final note. In using this, you MUST
283 to specify a real file in the file system to act as the
284 cookie jar file (HTTP.COOKIEJAR) so that the
285 redirect site can properly pass back authorization information.
287 ## Client-Side Certificates {#auth_clientcerts}
289 Some systems, notably ESG (Earth System Grid), requires
290 the use of client-side certificates, as well as being
291 [re-direction based](#REDIR).
292 This requires setting the following entries:
294 - HTTP.COOKIEJAR — a file path for storing cookies across re-direction.
295 - HTTP.NETRC — the path to the netrc file.
296 - HTTP.SSL.CERTIFICATE — the file path for the client side certificate file.
297 - HTTP.SSL.KEY — this should have the same value as HTTP.SSL.CERTIFICATE.
298 - HTTP.SSL.CAPATH — the path to a "certificates" directory.
299 - HTTP.SSL.VALIDATE — force validation of the server certificate.
301 Note that the first two are there to support re-direction based authentication.
305 1. https://curl.haxx.se/libcurl/c/curl_easy_setopt.html
306 2. https://curl.haxx.se/docs/ssl-compared.html
308 ## Appendix A. All RC-File Keys {#auth_allkeys}
310 For completeness, this is the list of all rc-file keys.
311 If this documentation is out of date with respect to the actual code,
312 the code is definitive.
314 <tr><th>Key</th><th>curl_easy_setopt Option</th>
315 <tr valign="top"><td>HTTP.DEFLATE</td><td>CUROPT_DEFLATE<br>with value "deflate,gzip"</td>
316 <tr><td>HTTP.VERBOSE</td><td>CUROPT_VERBOSE</td>
317 <tr><td>HTTP.TIMEOUT</td><td>CUROPT_TIMEOUT</td>
318 <tr><td>HTTP.USERAGENT</td><td>CUROPT_USERAGENT</td>
319 <tr><td>HTTP.COOKIEJAR</td><td>CUROPT_COOKIEJAR</td>
320 <tr><td>HTTP.COOKIE_JAR</td><td>CUROPT_COOKIEJAR</td>
321 <tr valign="top"><td>HTTP.PROXY.SERVER</td><td>CURLOPT_PROXY,<br>CURLOPT_PROXYPORT,<br>CURLOPT_PROXYUSERPWD</td>
322 <tr valign="top"><td>HTTP.PROXY_SERVER</td><td>CURLOPT_PROXY,<br>CURLOPT_PROXYPORT,<br>CURLOPT_PROXYUSERPWD</td>
323 <tr><td>HTTP.SSL.CERTIFICATE</td><td>CUROPT_SSLCERT</td>
324 <tr><td>HTTP.SSL.KEY</td><td>CUROPT_SSLKEY</td>
325 <tr><td>HTTP.SSL.KEYPASSWORD</td><td>CUROPT_KEYPASSWORD</td>
326 <tr><td>HTTP.SSL.CAINFO</td><td>CUROPT_CAINFO</td>
327 <tr><td>HTTP.SSL.CAPATH</td><td>CUROPT_CAPATH</td>
328 <tr><td>HTTP.SSL.VERIFYPEER</td><td>CUROPT_SSL_VERIFYPEER</td>
329 <tr><td>HTTP.CREDENTIALS.USERPASSWORD</td><td>CUROPT_USERPASSWORD</td>
330 <tr><td>HTTP.CREDENTIALS.USERNAME</td><td>CUROPT_USERNAME</td>
331 <tr><td>HTTP.CREDENTIALS.PASSWORD</td><td>CUROPT_PASSWORD</td>
332 <tr><td>HTTP.NETRC</td><td>CURLOPT_NETRC,CURLOPT_NETRC_FILE</td>
335 ## Appendix B. URS Access in Detail {#auth_ursdetail}
337 It is possible to use the NASA Earthdata Login System (URS)
338 with netcdf by using using the process specified in the
339 [redirection based authorization section](#REDIR).
340 In order to access URS controlled datasets, however, it is necessary to
341 register as a user with NASA at this website (subject to change):
343 https://uat.urs.earthdata.nasa.gov/
345 ## Appendix C. ESG Access in Detail {#auth_esgdetail}
347 It is possible to access Earth Systems Grid (ESG) datasets
348 from ESG servers through the netCDF API using the techniques
349 described in the section on [Client-Side Certificates](#CLIENTCERTS).
351 In order to access ESG datasets, however, it is necessary to
352 register as a user with ESG and to setup your environment
353 so that proper authentication is established between an netcdf
354 client program and the ESG data server. Specifically, it
355 is necessary to use what is called "client-side keys" to
356 enable this authentication. Normally, when a client accesses
357 a server in a secure fashion (using "https"), the server
358 provides an authentication certificate to the client.
359 With client-side keys, the client must also provide a
360 certificate to the server so that the server can know with
361 whom it is communicating. Note that this section is subject
362 to change as ESG changes its procedures.
364 The netcdf library uses the _curl_ library and it is that
365 underlying library that must be properly configured.
369 The key elements for client-side keys requires the constructions of
370 two "stores" on the client side.
372 * Keystore - a repository to hold the client side key.
373 * Truststore - a repository to hold a chain of certificates
374 that can be used to validate the certificate
375 sent by the server to the client.
377 The server actually has a similar set of stores, but the client
378 need not be concerned with those.
382 The first step is to obtain authorization from ESG.
383 Note that this information may evolve over time, and
385 This discussion is in terms of BADC and NCSA. You will need
386 to substitute as necessary.
388 1. Register at http://badc.nerc.ac.uk/register
389 to obtain access to badc and to obtain an openid,
390 which will looks something like:
391 <pre>https://ceda.ac.uk/openid/Firstname.Lastname</pre>
393 2. Ask BADC for access to whatever datasets are of interest.
395 3. Obtain short term credentials at
396 _http://grid.ncsa.illinois.edu/myproxy/MyProxyLogon/_
397 You will need to download and run the MyProxyLogon program.
398 This will create a keyfile in, typically, the directory ".globus".
399 The keyfile will have a name similar to this: "x509up_u13615"
400 The other elements in ".globus" are certificates to use in
401 validating the certificate your client gets from the server.
403 4. Obtain the program source ImportKey.java
404 from this location: _http://www.agentbob.info/agentbob/79-AB.html_
405 (read the whole page, it will help you understand the remaining steps).
407 ### Building the KeyStore
409 You will have to modify the keyfile in the previous step
410 and then create a keystore and install the key and a certificate.
411 The commands are these:
413 openssl pkcs8 -topk8 -nocrypt -in x509up_u13615 -inform PEM -out key.der -outform DER
414 openssl x509 -in x509up_u13615 -inform PEM -out cert.der -outform DER
415 java -classpath <path to ImportKey.class> -Dkeypassword="<password>" -Dkeystore=./<keystorefilename> key.der cert.der
417 Note, the file names "key.der" and "cert.der" can be whatever you choose.
418 It is probably best to leave the .der extension, though.
420 ### Building the TrustStore
422 Building the truststore is a bit tricky because as provided, the
423 certificates in ".globus" need some massaging. See the script below
424 for the details. The primary command is this, which is executed for every
425 certificate, c, in globus. It sticks the certificate into the file
428 keytool -trustcacerts -storepass "password" -v -keystore "truststore" -importcert -file "${c}"
430 ### Running the C Client
432 Refer to the section on [Client-Side Certificates](#CLIENTCERTS).
433 The keys specified there must be set in the rc file to support ESG access.
435 - HTTP.COOKIEJAR=~/.dods_cookies
436 - HTTP.NETRC=~/.netrc
437 - HTTP.SSL.CERTIFICATE=~/esgkeystore
438 - HTTP.SSL.KEY=~/esgkeystore
439 - HTTP.SSL.CAPATH=~/.globus
440 - HTTP.SSL.VALIDATE=1
442 Of course, the file paths above are suggestions only;
443 you can modify as needed.
444 The HTTP.SSL.CERTIFICATE and HTTP.SSL.KEY
445 entries should have same value, which is the file path for the
446 certificate produced by MyProxyLogon. The HTTP.SSL.CAPATH entry
447 should be the path to the "certificates" directory produced by
450 As noted, ESG also uses re-direction based authentication.
451 So, when it receives an initial connection from a client, it
452 redirects to a separate authentication server. When that
453 server has authenticated the client, it redirects back to
454 the original url to complete the request.
456 ### Script for creating Stores
458 The following script shows in detail how to actually construct the key
459 and trust stores. It is specific to the format of the globus file
460 as it was when ESG support was first added. It may have changed
461 since then, in which case, you will need to seek some help
462 in fixing this script. It would help if you communicated
463 what you changed to the author so this document can be updated.
466 KEYSTORE="esgkeystore"
467 TRUSTSTORE="esgtruststore"
469 TRUSTROOT="certificates"
471 TRUSTROOTPATH="$GLOBUS/$TRUSTROOT"
472 CERTFILE="$GLOBUS/$CERT"
476 CCP="bcprov-jdk16-145.jar"
480 # Initialize needed directories
488 # Compile MyProxyCmd and ImportKey
489 javac -d ./build -classpath "$CCP" *.java
490 javac -d ./build ImportKey.java
493 java -cp "$CP myproxy.MyProxyCmd
496 openssl pkcs8 -topk8 -nocrypt -in $CERTFILE -inform PEM -out key.der -outform DER
497 openssl x509 -in $CERTFILE -inform PEM -out cert.der -outform DER
498 java -Dkeypassword=$PWD -Dkeystore=./${KEYSTORE} -cp ./build ImportKey key.der cert.der
500 # Clean up the certificates in the globus directory
501 for c in ${TRUSTROOTPATH}/*.0 ; do
502 alias=`basename $c .0`
503 sed -e '0,/---/d' <$c >/tmp/${alias}
504 echo "-----BEGIN CERTIFICATE-----" >$c
505 cat /tmp/${alias} >>$c
508 # Build the truststore
509 for c in ${TRUSTROOTPATH}/*.0 ; do
510 alias=`basename $c .0`
511 echo "adding: $TRUSTROOTPATH/${c}"
513 yes | keytool -trustcacerts -storepass "$PWD" -v -keystore ./$TRUSTSTORE -alias $alias -importcert -file "${c}"
519 __Author__: Dennis Heimbigner<br>
520 __Email__: dmh at ucar dot edu
521 __Initial Version__: 11/21/2014<br>
522 __Last Revised__: 08/24/2017