Reading HDF5 File Metadata

Read a selection of standard PSI data products in HDF5 format and examine their metadata.

This example demonstrates how to use read_hdf_meta() to inspect the metadata of various standard PSI data files; these example files are utilized elsewhere in the documentation and examples; hopefully this example will help users understand the structure of these files and how to work with them using PSI I/O.

from pathlib import Path
from psi_io import read_hdf_meta, data

Read in example 1D data file, viz. a vignette function (used for radial scaling).

data1d_filepath = data.get_1d_data()
meta1d = read_hdf_meta(data1d_filepath)

Datasets for file: rscale.h5
+-----------------+-----------------+-----------------+
| Name            | Shape           | Type            |
+-----------------+-----------------+-----------------+
| Data            | (121,)          | float32         |
+-----------------+-----------------+-----------------+

Scales for file: rscale.h5
+------------+------------+------------+------------+------------+------------+------------+
| Dataset    | Dim        | Name       | Shape      | Type       | Min        | Max        |
+------------+------------+------------+------------+------------+------------+------------+
| Data       | 0          | dim1       | (121,)     | float32    | 1.0        | 20.0       |
+------------+------------+------------+------------+------------+------------+------------+

Read in example 2D data file, viz. a long-lat coronal hole map.

data2d_filepath = data.get_2d_data()
meta2d = read_hdf_meta(data2d_filepath)

Datasets for file: chmap.h5
+-----------------+-----------------+-----------------+
| Name            | Shape           | Type            |
+-----------------+-----------------+-----------------+
| Data            | (721, 361)      | float32         |
+-----------------+-----------------+-----------------+

Scales for file: chmap.h5
+------------+------------+------------+------------+------------+------------+------------+
| Dataset    | Dim        | Name       | Shape      | Type       | Min        | Max        |
+------------+------------+------------+------------+------------+------------+------------+
| Data       | 0          | dim1       | (361,)     | float32    | 0.0        | 3.1415927  |
| Data       | 1          | dim2       | (721,)     | float32    | 0.0        | 6.2831855  |
+------------+------------+------------+------------+------------+------------+------------+

Read in example 3D data file, viz. the radial component of the magnetic field.

data3d_filepath = data.get_3d_data()
meta3d = read_hdf_meta(data3d_filepath)

Datasets for file: br.h5
+-----------------+-----------------+-----------------+
| Name            | Shape           | Type            |
+-----------------+-----------------+-----------------+
| Data            | (181, 100, 151) | float32         |
+-----------------+-----------------+-----------------+

Scales for file: br.h5
+------------+------------+------------+------------+------------+------------+------------+
| Dataset    | Dim        | Name       | Shape      | Type       | Min        | Max        |
+------------+------------+------------+------------+------------+------------+------------+
| Data       | 0          | dim1       | (151,)     | float32    | 0.99956024 | 30.511646  |
| Data       | 1          | dim2       | (100,)     | float32    | 0.0        | 3.1415927  |
| Data       | 2          | dim3       | (181,)     | float32    | 0.0        | 6.2831855  |
+------------+------------+------------+------------+------------+------------+------------+

Read in example fieldline trace data file – a 2D dataset without coordinate variables.

datafieldline_filepath = data.get_fieldline_data()
metafieldline = read_hdf_meta(datafieldline_filepath)

Datasets for file: fieldline.h5
+-----------------+-----------------+-----------------+
| Name            | Shape           | Type            |
+-----------------+-----------------+-----------------+
| Data            | (2578, 3)       | float32         |
+-----------------+-----------------+-----------------+

Scales for file: fieldline.h5
     No scales found in this file.

Read in example file from PSI’s Coronal Hole Map Database

Note

This file contains not only a 2D coronal hole map (with its associated coordinate variables) but also a variety of additional metadata datasets describing the observation and processing used to create the map.

datasynchronic_map_filepath = data.get_synchronic_map_data()
metasynchronic_map = read_hdf_meta(datasynchronic_map_filepath)

Datasets for file: synchronic_map.h5
+-----------------+-----------------+-----------------+
| Name            | Shape           | Type            |
+-----------------+-----------------+-----------------+
| Data            | (640, 1600)     | float32         |
| chd             | (640, 1600)     | float16         |
| mu              | (640, 1600)     | float16         |
| origin_image    | (640, 1600)     | uint32          |
+-----------------+-----------------+-----------------+

Scales for file: synchronic_map.h5
+------------+------------+------------+------------+------------+------------+------------+
| Dataset    | Dim        | Name       | Shape      | Type       | Min        | Max        |
+------------+------------+------------+------------+------------+------------+------------+
| Data       | 0          | dim1       | (1600,)    | float32    | 0.0        | 6.283185   |
| Data       | 1          | dim2       | (640,)     | float32    | -1.0       | 1.0        |
+------------+------------+------------+------------+------------+------------+------------+

Except for the last example file (derived from the Coronal Hole Map Database), each of these files are available in both HDF4 and HDF5 formats. You can read the metadata from either format using the same read_hdf_meta() function. To illustrate how these files differ (internally) between HDF4 and HDF5, we can read the metadata from the HDF4 version of the 3D data file using the same function:

data3d_hdf4_filepath = data.get_3d_data(hdf=".hdf")
meta3d_hdf4 = read_hdf_meta(data3d_hdf4_filepath)

Datasets for file: br.hdf
+-----------------+-----------------+-----------------+
| Name            | Shape           | Type            |
+-----------------+-----------------+-----------------+
| Data-Set-2      | (181, 100, 151) | float32         |
+-----------------+-----------------+-----------------+

Scales for file: br.hdf
+------------+------------+------------+------------+------------+------------+------------+
| Dataset    | Dim        | Name       | Shape      | Type       | Min        | Max        |
+------------+------------+------------+------------+------------+------------+------------+
| Data-Set-2 | 0          | fakeDim0   | (181,)     | float32    | 0.0        | 6.2831854820251465 |
| Data-Set-2 | 1          | fakeDim1   | (100,)     | float32    | 0.0        | 3.1415927410125732 |
| Data-Set-2 | 2          | fakeDim2   | (151,)     | float32    | 0.9995602369308472 | 30.511646270751953 |
+------------+------------+------------+------------+------------+------------+------------+

Compared to the HDF5 version of the same file, the HDF4 version has slightly different dataset naming conventions (e.g. the primary dataset is labeled “Data-Set-2” instead of “Data”, while its scales are named “fakeDim0”, “fakeDim1”, and “fakeDim2” instead of “dim1”, “dim2”, “dim3”). Also – and perhaps more importantly – the HDF4 version does not respect the inherent Fortran ordering of the underlying data arrays

Note

For more information regarding the differences between HDF4 and HDF5, see Overview.