Skip to content

Instantly share code, notes, and snippets.

@ila057
Last active August 23, 2016 18:34
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ila057/c60c70a152fb86e6f5f11230700dcd9c to your computer and use it in GitHub Desktop.
Save ila057/c60c70a152fb86e6f5f11230700dcd9c to your computer and use it in GitHub Desktop.
odML-HDF5-Converter is a library for conversion of any metadata from odML to HDF5 format,and vice versa; along with transfer of raw data to NIX data model.

#odML-HDF5-Converter

###Github Repository for Library https://github.com/ila057/odML-HDF5-Converter/

###List of Commits

  1. https://github.com/ila057/odML-HDF5-Converter/commits/master?author=ila057
  2. https://github.com/ila057/odML-HDF5-Converter/commits/basicConverter?author=ila057
  3. https://github.com/INCF/eeg-database/commits/hdf5Converter?author=ila057

###Background The data produced by experiments of Electroencephalography (EEG) and Event­-related potentials (ERP) is stored in web portal of EEGBase, by implementing a system of templates. These templates allow the users to store metadata in odML format, and the data is stored its raw format(.eeg or .avg). However, the initiatives within the neuroinformatics community have proposed unification of data models by using open standardized formats such as HDF5 or odML. This concept of opendata necessitates the development of this library, which shall allow the conversion to either odML or HDF5 as per the requirements.

This library is enabled to be integrated with EEGBase, such that the user is provided with the option of downloading the data-package in HDF5 format along with the currently existing option of downloading the odML metadata and raw data.

###Working The library has been developed in two versions:

  • Basic-Converter, which contains the basic crux of complete-converter, and takes in the
    • name of the HDF5 file to be created,
    • metadata.xml
    • .eeg/.avg file containing raw data
    • header file name
    • marker file name

This Basic-Converter extracts relevant information from the metadata.xml, .eeg/.avg file, header file, and marker and creates the HDF5 file of the given name. This can be used separately to be integrated in another piece of software, as in the sample code : ```java ODMLParserImpl odmlParser = new ODMLParserImpl("Experiment_208_Driver's_attention_with_visual_stimulation_and_audio_disturbance", "metadata.xml", "LED_26_3_2014_0004.eeg", "LED_26_3_2014_0004.vhdr", "LED_26_3_2014_0004.vmrk"); ``` Link to the code of Basic-Converter : https://github.com/ila057/odML-HDF5-Converter/tree/basicConverter
  • Complete-Converter, which does everything that the Basic-Convertor does, but is developed with the intention of integration into EEGBase. It takes in a zip data-package, downloaded from EEGBase. This zip package must follow the directory structure templates used in EEGBase. The converter extracts the data files, marker and header files and converts them into HDF5 format, retaining the same directory structure and the scenario in which the experiment was conducted. Example usage:
  String finalConvertedFolderToBeZipped = dataProcessor.generateConvertedDataSet("EEG_ERP.zip");

Link to code of Complete-Converter : https://github.com/ila057/odML-HDF5-Converter/

###Implementation

Though the code is documented, explaining the role of each component, a few salient points explaining the working, and the motivation behind making things work in a certain way are explained below:

  1. The working of Complete-Converter starts from generateConvertedDataSet(String inputZipFile), from wherein it unzips the data-package and handovers its contents to processAllDataSetsFinal(). This function is responsible for extracting the data files(eeg/avg files, header and marker files) and furthering them for conversion to HDF5. On analysis of the data-packages created by EEGBase, it was found that it follows one of the directory structures below:
1. <Data_Package.zip>/<Several_Experiments/Data/<Several_Experiment_Datasets.zip>/<data_files>
2. <Data_Package.zip>/<Several_Experiments/Data/<Several_Experiment_Datasets.zip>/<Experiment_Dataset_Name>/<data_files>
3. <Data_Package.zip>/<Several_Experiments/Data/<data_files>

The <data_files> must be converted to HDF5 and are hence passed to ODMLParser. The objective of ODMLParser is to call DataParser and MetadataParser, which are responsible for convertion of data and metadata into HDF5 file respectively.

  1. The DataParser converts the raw data from <file_name.eeg/.avg> and stores it as double values in HDF5. The MetadataParser must read through three files to consolidate the entire metadata:

  2. metadata.xml : Metadata corresponding to a dataset

  3. <file_name>.vmrk : Marker File

  4. <file_name>.vhdr : Header File

  5. One can also convert the converted HDF5 file back to the set of odML metadata and data-files if required. Sample usage is as below:

MetadataCreator metadataCreator = new MetadataCreatorImpl(hdf5FileName);
m.createOdml(odMLMetadataFile);

###Limitations This library is only compatible with versions of java 8 or greater, and needs 64-bit OS to work properly. These hardware and java restrictions are actually imposed by the nix-java bindings which is used as a dependency for the project.

###Future Work

  • The library eegloader is used to extract the channel and marker information from .vhdr and .vmrk files respectively. This does not extract all the information from those files, but has support for following information:
    • Channel
      • Name
      • Number
      • Units
      • Resolution
    • Marker
      • Name
      • Position
      • Stimulus

When eegloader is extended, this library can also be extended to include further information from header and marker files.
  • EEGBase Live server currently runs on Java 6. This imposes a restriction on merging our library as well as some commits in EEGBase unless its servers are moved to Java 8. Thus, it remains to merge the library with EEGBase after its migration to Java 8.

  • This library's dependencies (nix-java bindings) require 64-bit OS to work. The existing VM for development and testing of EEGBase on any system is 32 bit, and hence end-to-end testing with EEGBase could not be established, which remains as a future work. However, the library is tested separately, and the code for enabling integration in EEGBase has also been thoroughly tested.

commit f51926c0b85e8bc7040e3db267cce5d62d827b6f
Author: ipsita <ipsitabits@gmail.com>
Date: Sun Aug 21 21:43:04 2016 +0530
Added downloaded support for HDF5 Files
commit 03447051187b32a9f7cd06f04459b55e124f7841
Author: ipsita <ipsitabits@gmail.com>
Date: Sun Aug 21 18:56:20 2016 +0530
let jar be called converter.jar
commit 459710102d651015a761b1e5182568e3c6636427
Author: ipsita <ipsitabits@gmail.com>
Date: Sun Aug 21 18:27:12 2016 +0530
removed bugs, added support to take a zip file and convert all the corresponding datasets inside it to HDF5
commit 4fe96f56ea8acd2bfb01e28dcddae07f813b5343
Author: ipsita <ipsitabits@gmail.com>
Date: Tue Aug 16 23:26:17 2016 +0530
added support for vmrk and vhdr files
commit df373547a5693d3b6289b478f1208fdce0ff42d6
Author: ipsita <ipsitabits@gmail.com>
Date: Sun Jul 31 22:30:48 2016 +0530
handled exception and created interfaces
commit 491f0af4a3c8ae31937823ac89d9777347f9fd1f
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Sun Jul 31 22:02:59 2016 +0530
Update README.md
commit 55e9397e8f912f6c5ba48c73eb546b6ab1e7d742
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Fri Jul 29 12:44:38 2016 +0530
Update pom.xml
commit fe846603e3483ecd0b53e8a22dd9a39f46abfa5b
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Thu Jul 28 16:44:04 2016 +0530
Update pom.xml
Made corrections to pom.xml.
commit 877a0da665612337305a60d82cbd7df885a4d79f
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Wed Jul 27 15:38:42 2016 +0530
Delete OdMLReader.java
commit d35ed5290f72c7b19cfa8a5c4384d359b1997f34
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Wed Jul 27 15:38:31 2016 +0530
Delete MainClass.java
commit 52dccd4803466142a3fe9e90532bf77bc647cc16
Author: Ipsita Samal <ipsita@IpsitaSamal-INL.local>
Date: Wed Jul 27 14:05:51 2016 +0530
Remove production classes
commit 2b6e8cc0ff079808f7afd20a8f196c2cc7dceded
Author: Ipsita Samal <ipsita@IpsitaSamal-INL.local>
Date: Wed Jul 27 14:04:56 2016 +0530
Remove targer classes
commit 8069c898c83efc960d621723788359161bb30d02
Author: Ipsita Samal <ipsita@IpsitaSamal-INL.local>
Date: Wed Jul 27 13:58:02 2016 +0530
Mavenised the project, and added support for converting from HDF5 file, adding logging
commit a7f2ce83f6ec0534ed16d875a7aa8791474a7886
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Fri Jul 15 14:25:20 2016 +0530
Update MetadataParserImpl.java
commit 8a83fa304f1b97574c9850113f9f344e5d4c8411
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Fri Jul 15 14:24:36 2016 +0530
Delete OdMLReader.java
commit 0c1c1d2395b3a05d187d7c3474a5bef1bb33e3ff
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Fri Jul 15 12:55:48 2016 +0530
Delete metadata.xml
commit c5a870c95e1343a23432a9a4251edad08c8bd03a
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Fri Jul 15 12:55:27 2016 +0530
Delete LICENSE~
commit 202c212b843550a5ee23827df6b47ac2f647f180
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Fri Jul 15 12:54:59 2016 +0530
Update README.md
commit 1594a2b5fd938361b4f9694684dc66775d9dbc51
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Fri Jul 15 12:54:37 2016 +0530
Update README.md
commit 80c1ec097d020e0cdb14bc91d0219789f78029cf
Author: Ipsita Samal <ipsita@IpsitaSamal-INL.local>
Date: Fri Jul 15 12:50:53 2016 +0530
Commit
commit a0446e718c2f71a19df21a4698c647a701b81115
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Fri Jun 24 21:34:50 2016 +0530
Update README.md
commit da4affe0608adb5b12758f7207b77fbd5979d2c6
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Fri Jun 24 21:33:29 2016 +0530
Update README.md
commit 97f5a3809f63104b82165934ee2c83fe54ce8f12
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Fri Jun 24 21:31:37 2016 +0530
Update MetadataParser.java
added code to use odml-java-lib, fixed bug regarding recursive sections, added support for gui-namespaces
commit 09bc33e8613373bff864e53fdb1ce1e0c812a5e2
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Thu Jun 23 14:58:00 2016 +0530
Add files via upload
commit f5c1cbe6df68e71585adbadeb0c73591539e8da8
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Thu Jun 23 13:52:26 2016 +0530
Add files via upload
index.html is the homepage of the documentation
commit eb8d2b584ee43e5e01215ce60733476dd1939330
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Thu Jun 23 13:48:08 2016 +0530
Delete ParseMetadata.java
commit e518ef02a63d79f81f0f8a6dadbd6dc69d39cb89
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Thu Jun 23 13:47:50 2016 +0530
Add files via upload
commit 508c8522c75937e88ea00730304107e054de795e
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Thu Jun 23 13:02:44 2016 +0530
Update README.md
commit 0885df72fa477f64a73864c8bca91c08f99ecda7
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Thu Jun 23 12:59:35 2016 +0530
Update README.md
commit 6a6bab4432c01e4e0e97b99f47f387b5e4bf4771
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Thu Jun 23 12:57:33 2016 +0530
Update README.md
commit 436555ed46670d90a3809423ca1c11e1169d696a
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Thu Jun 23 12:34:22 2016 +0530
Add files via upload
commit 0dfd3e07d28a811fa733c733813d3451e896d3cc
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Thu Jun 23 12:31:54 2016 +0530
Create LICENSE
commit 3c9b86d46641d8872da86f966571837016e94e1b
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Mon Jun 20 14:52:16 2016 +0530
Delete test
commit 3087e6bf11e8f6d29a00952b66b1db99237c1dc2
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Mon Jun 20 14:51:57 2016 +0530
Test file to check metadata parser
commit 4aa046c83356aaab8e772050bbf5e0499bc0f997
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Mon Jun 20 14:51:31 2016 +0530
Create test
commit 5d140a94a3372acfbae30735b8139b2c632eb0eb
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Mon Jun 20 14:50:57 2016 +0530
Delete test
commit 7640621d8886fc0fa262c8a2cbeb42cbd8290ae7
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Mon Jun 20 14:50:45 2016 +0530
odML metadata to HDF5
commit 6b531dac98140e8b7226caff1841e2f0ca829c05
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Mon Jun 20 14:50:02 2016 +0530
Create test
commit 37ff2deb1e2d36b9c02fd0e1ac5cae34f955ab28
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Mon Jun 20 14:48:59 2016 +0530
Delete ParseMetadata.java
commit 063fdd5fd4855de7a0128ac70f17d0b5cb7ac877
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Mon Jun 20 14:48:20 2016 +0530
Commit
commit 505bb134a86497b44ff2724c8aec0a4a9697c4d1
Author: Ipsita Samal <ipsitabits@gmail.com>
Date: Mon Jun 20 14:15:47 2016 +0530
Initial commit
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment