Data Structure^•

Hierarchical Data Format (HDF) #

• Designed to store and organize large amounts of data.
• Store multiple data files in a single data file!
  • Different types of information.
  • Self describing (metadata included in the file)
• Properties^[ref]:
  • Datasets (numpy arrays): fast slicing, compression.
  • Group (dictionaries): nesting, POSIX path syntax.
  • Attributrs (metadata): datasets/group, key-value.
• HDF5 is row based and really effient than csv for very large file size^[ref].
• Extensions: .h5, .hdf, .hdf4, ...
• Tool: HDFView
• Example^[ref]:

An example HDF5 file structure which contains groups, datasets and associated metadata.

import h5py

f = h5py.File('mytestfile.hdf5', 'r') # read a file
# h5py.File acts like Python dict
dset = f['mydataset']
dset.attrs # attribute

t-digest #

later

^•Notes with this notation aren't good enough. They are being updated.