Skip to content

bobkocisko/netcdf-challenge

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

21 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Introduction

This repository is my attempt at solving a programming challenge related to the netCDF file format.

The ultimate result is a set of visualizations of concentration data as pictured here:

animated visualization

cpp

A c++ REST API that serves information about the specified netCDF file. Currently this is configured to use a netCDF file present in the source, but since it uses the command line to specify the desired netCDF file, this could be rather easily reconfigured to work for other netCDF files as well.

python

This python project is simply here to track my usage of python's netcdf4 plugin to examine the netCDF file provided and print general information. I decided to use python initially for this because I figured it would be a quick path to iterate around browsing the data without waiting to setup the dependencies and compilation environment of c++, and without depending on the compile/run cycle.

Setup

This repository uses a combination of git submodules and docker to manage dependencies and to provide the dev container for the VS Code ide.

  1. git clone --recurse-submodules [email protected]:bobkocisko/netcdf-challenge.git

  2. cd cpp

  3. docker compose up -d (start containers in background)

  4. docker compose watch netcdf-api (automatically rebuild and restart container on any code changes, and display build output)

  5. Test the endpoints available:

  6. To get full intellisense support in VSCode:

    1. Choose Attach to running container...
    2. Then select /netcdf-api-ide from the options

Known Issues

When querying the /get-image endpoint in "force-refresh" mode (holding down SHIFT while clicking Refresh in the browser), the api is sometimes unresponsive. No logs are generated by crow during the unresponsive time period. The only solution is to cancel the request in the browser. Some research revealed these issues which may be related. Other endpoints do not display this behavior.

When querying /get-image?time_index=7&z_index=0, the api doesn't respond due to the matplot++ library delaying indefinitely. This must be due to the fact that all data values are 0.0, but I ran out of time to figure out and resolve the issue.

Reference

netCDF file format

Fundamentals of netCDF data storage

NetCDF: Introduction and Overview

The NetCDF User's Guide

netCDF library usage

netCDF C++ Interface Guide

NetCDF: The NetCDF-C Tutorial

NetCDF: Reading NetCDF/HDF5 Format NetCDF Files of Unknown Structure (groups/types don't appear to be present for the sample file we are working with)

Thread safety concerns

ncdump.c: handling NC_CHAR data type (source code for ncdump utility, this link is to a particular function that parses NC_CHAR data types)

Memory management for NC_STRING types

Crow REST library

Crow REST library docs

Docker reference

Setting up a dev environment with Docker/Compose

Understanding multi-stage builds

PID 1 Signal Handling in Docker

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published