AMGeO 2.0: Crafting an API for Geospace Data Scientists#

Authors#

Author1 = { “name”: “Willem Mirkovich”, “affiliation”: “University of Colorado Boulder, Smead Aerospace Engineering Sciences”, “email”: “willem.mirkovich@colorado.edu”, “orcid”: “0000-0003-0955-8281” }

Author2 = { “name”: “Tomoko Matsuo”, “affiliation”: “University of Colorado Boulder, Smead Aerospace Engineering Sciences”, “email”: “tomoko.matsuo@colorado.edu”, “orcid”: “0000-0002-2754-1224” }

Author3 = { “name”: “Liam Kilcommons”, “affiliation”: “University of Colorado Boulder, Smead Aerospace Engineering Sciences”, “email”: “Liam.Kilcommons@colorado.edu”, “orcid”: “0000-0002-4980-3045” }

1 AMGeO 2.0: Crafting an API for Geospace Data Scientists
2 Setup
- 2.1 Library imports
3 Parameter definitions
4 Data import
5 Data processing and analysis
6 References

Purpose#

The purpose of this notebook is to introduce Geospace scientists to the new AMGeO 2.0 API, as well as demonstrate functionality with other popular python packages.

Technical contributions#

Introduces geospace researchers to AMGeO’s new API
Showcases power of using Xarray to enable research of multidimensional datasets
Gives examples of using the API with various other Python packages, including
- Numpy
- Matplotlib
- Apexpy
- Cartopy

Methodology#

The Assimilative Mapping of Geospace Observations (AMGeO) is a data science tool for the geospace science community that automates labor-intensive data acquisition and processing, combining observations from various sensors into assimilative maps of the high-latitude ionosphere. While AMGeO offers a highly configurable toolset, it currently requires both domain expertise and familiarity with Python to use it effectively for scientific research.

To remove hurdles for novice users and empower all AMGeO users, we have recently introduced a new Application Programming Interface (API) focused on enhanced user-experience, including better compatibility with Jupyter Notebooks, improved data manipulation with Xarray and more flexible data generation. This notebook will demonstrate the functionality offered by the new API and how to use AMGeO in conjunction with other popular Python research tools in order to accelerate geospace data science processes.

Funding#

AMGeO is supported by the NSF EarthCube grants ICER 1928403 to the University of Colorado Boulder, ICER 1928327 to the Virginia Tech, and ICER 1928358 to the Johns Hopkins University Applied Physics Laboratory.

Keywords#

keywords=[“AMGeO”, “Xarray”, “Python”, “API”, “Geospace”]

Citation#

Willem Mirkovich, Liam Kilcommons, Tomoko Matsuo, 2022. AMGeO, Xarray, Python, API, Geospace. Accessed 4/15/2022 at https://github.com/willemmirkovich/Earthcube-Meeting-2022

Work in progress#

While this notebook is in a complete state, these are some notable additions that would be nice to add:

Apexpy example of converting to base vectors
Cartopy example of time-series
Cartopy example of vector plots

Suggested next steps#

Please get started with AMGeO here, we would love any feedback to continue improving our software.

Acknowledgements#

We would like to thank our data providers:

Setup#

Library imports#

Below are the necessary libraries needed to run this notebook:

Python 3.8
AMGeO 2.0.2 (comes with Matplotlib, Numpy, Xarray)
Apexpy 1.0.1
Cartopy 0.20.2

NOTE: If AMGeO is installed locally, upon import, if you have not configured AMGeO with your API key, SuperMAG username or AMPERE username, this will be asked here (with instructions on how to get each). This notebook bypasses this step.

# Ignore uunecessary warnings from AMGeO dependencies
import warnings
warnings.filterwarnings('ignore')

# AMGeO's API class
from AMGeO.api import AMGeOApi

# python's datetime module
from datetime import datetime, date

# matplotlib tools
import matplotlib.pyplot as plt
import numpy as np

# cartopy
import cartopy.crs as crs

# apexpy
from apexpy import Apex

Solar wind data files will be saved to /home/jovyan/.local/share/nasaomnireader

------------IMPORTANT----------------------------
Unable to import spacepy. Will fall back to
using Omni text files, which may have slightly
different data and incomplete metadata
-------------------------------------------------

Parameter definitions#

During the duration of this notebook, there are some commone placeholder variables that will be used.

api will be used to store AMGeO’s API instance

api = AMGeOApi()
api

AMGeO API instance
current configuration: default
current output directory: /workspaces/Earthcube-Meeting-2022/amgeo_out

controller will be used to generate and load assimilative maps using AMGeO. This will be described in more detail in Creating an AMGeO controller

controller = api.get_controller()
controller

Default AMGeO Controller
Creates Assimilative Maps using SuperMAG and SuperDARN observations
Does not predict magnetic potential nor field-aligned current

Data import#

Since this notebook is a technical overview of AMGeO’s new API, all Data used will be imported/generated during the course of the Data Processing section. To learn more about AMGeO and the data we generate, please go to our website

Data processing and analysis#

Compatability with matplotlib#

Xarray DataArrays also make quick plotting much easier, with coordinates and metadata accessible from the same data structure

def _polar2dial(ax):
    """
    Turns a matplotlib axes polar plot into a dial plot
    """
    #Rotate the plot so that noon is at the top and midnight
    #is at the bottom, and fix the labels so radial direction
    #is latitude and azimuthal direction is local time in hours
    ax.set_theta_zero_location('S')
    theta_label_values = np.array([0.,3.,6.,9.,12.,15.,18.,21.])*180./12
    theta_labels = ['%d' % (int(th/180.*12)) for th in theta_label_values.flatten().tolist()]
    ax.set_thetagrids(theta_label_values,labels=theta_labels)

    r_label_values = 90.-np.array([80.,70.,60.,50.,40.])
    r_labels = [r'$%d^{o}$' % (int(90.-rv)) for rv in r_label_values.flatten().tolist()]
    ax.set_rgrids(r_label_values,labels=r_labels)
    ax.set_rlim([0.,40.])
    
def plot_epot_map(fig, lats, lons, epot):
    ax = fig.add_subplot(111, projection='polar')
    _polar2dial(ax)
    
    # plotting
    r = 90.-lats
    th = np.radians(lons)
    v = 30000
    levels=np.linspace(-v,v,30)
    cb = ax.contourf(th,r,epot,levels=levels,cmap='RdBu_r', vmin=-v, vmax=v,extend='both')
    
    # metadata attributes accessible on a DataArray
    units = epot.attrs['units']
    description = epot.attrs['longname']
    
    fig.colorbar(cb,label=f'{description} [{units}]')
    return ax

fig = plt.figure(figsize=(4, 4))

# grab 24x37 grid of Electo-static potentials
epot = ds['epot'][1]
# grab the time for this grid
t = epot.time.values
# plot epots with correct lat/lon coords
ax = plot_epot_map(fig, epot.lat, epot.lon, epot)
ax.set_title('Electric Potential map for %s' % t, pad=15, size=20)
plt.show()

Compatability with numpy#

Xarray has native support for quite a few Numpy compatable operations

Look here for a full list of support for Numpy

It is also possible to convert a DataArray to a ndarray with the values property

# convert from DataArray to Numpy Array
arr = ds['joule_heat'].values
type(arr)

numpy.ndarray

Compatability with apexpy#

ApexPy is a fully fleshed out library for geospatial coordinate systems conversions such as Geodetic, Magnetic Local Time, and Apex.

AMGeO’s latitude and longitude are in Magnetic Apex Latitude and Magnetic Local Time in Degrees, respectively, which is simple enough to transform to other coordinate systems using this package.

# first, create an apex_out instance

# for mlt, need specific date, so lets load up a dataset with a specific date in mine
dates = controller.browse('N')
dates[0]

datetime.date(2013, 1, 6)

d = dates[0]
ds = controller.load(d, 'N')

# from here, we can create an apex_out instance for coordinate conversion
apex_out = Apex(date=d)
apex_out

<apexpy.apex.Apex at 0x7fc966dfd2b0>

AMGeO to geodetic#

For plots using the common mercator projection, Geodetic is what we will be looking for

# need to create 24x37 grid of lat & lons for conversion based on 24x37 lat lon grid

amgeo_lats, amgeo_lons = np.zeros((24, 37)), np.zeros((24, 37))
for i in range(24):
    for j in range(37):
        amgeo_lats[i][j] = ds.lat.values[i]
        amgeo_lons[i][j] = ds.lon.values[j]
amgeo_lats, amgeo_lons = np.array(amgeo_lats), np.array(amgeo_lons)
amgeo_lats.shape, amgeo_lons.shape

((24, 37), (24, 37))

# NOTE: AMGeO lons are mlt in degrees, so simple transform to move to apex mlt
mlt = amgeo_lons / 180 * 12

# get specific reference date for conversion (needed for mlt conversion)
dt = controller.browse(d, 'N')[0]

# now, can convert to geodetic, AMGeO uses a reference height of 110km (the nominal F layer)
geo_lat, geo_lon = apex_out.convert(amgeo_lats, mlt, 'mlt', 'geo', datetime=dt, height=110)
geo_lat.shape, geo_lon.shape

((24, 37), (24, 37))

Compatability with cartopy#

CartoPy is a popular python package that allows for plotting onto Geographic projections of the world.

AMGeO’s API lends itself well to mapping its data easily to these plotting tools.

Plotting AMGeO electostatic potential on orthographic projection#

Using the convenient geo_lat and geo_lon values we just generated, we can use Cartopy’s PlatteCarree transform to map these values onto an Orthographic projection.

The Orhtographic projection will suit nicely for visualzing an AMGeO grid of electostatic potentials on the northern hemisphere, which will spread from the magnetic north pole.

fig = plt.figure(figsize=[10, 5])

idx = 1

# Northern Hemisphere
ax = fig.add_subplot(1, 2, 1, projection=crs.Orthographic(0, 90))

ax.coastlines(zorder=3)
ax.stock_img()
ax.gridlines()

ax.scatter(geo_lon, geo_lat, c=ds.epot[idx], cmap='coolwarm', transform=crs.PlateCarree())

ax.set_title(f'Epots for {ds.time.values[idx]}')

plt.show()

Pandas compatability#

While not necessarily easier, since AMGeO works with multi-dimensional data, Xarray provides compatability with pandas. A more comprehensive overview can be found here

# convert AMGeO Xarray DataSet to Pandas DataFrame
df = ds.to_dataframe()
print(type(df))
df

<class 'pandas.core.frame.DataFrame'>

			E_ph	E_th	cond_hall	cond_ped	epot	int_joule_heat	joule_heat	v_ph	v_th
time	lat	lon
2013-01-06 16:30:00	88.333323	0.0	-0.010193	0.026999	4.0	4.0	-6755.328259	9.622610	3.331331	459.399493	173.436070
		10.0	-0.009465	0.025080	4.0	4.0	-6419.514908	9.622610	2.874432	426.756367	161.045790
		20.0	-0.007984	0.023245	4.0	4.0	-6113.721553	9.622610	2.416306	395.527306	135.848700
		30.0	-0.005780	0.021667	4.0	4.0	-5862.900304	9.622610	2.011470	368.673123	98.356825
		40.0	-0.003063	0.020531	4.0	4.0	-5688.033704	9.622610	1.723594	349.343274	52.117439
...	...	...	...	...	...	...	...	...	...	...	...
2013-01-06 17:30:00	49.999754	320.0	0.000019	-0.000030	4.0	4.0	313.325279	27.863884	0.000005	-0.614055	-0.385016
		330.0	-0.000059	-0.000013	4.0	4.0	313.296195	27.863884	0.000015	-0.263458	1.207273
		340.0	0.000009	0.000095	4.0	4.0	338.207580	27.863884	0.000036	1.939544	-0.191665
		350.0	0.000146	-0.000018	4.0	4.0	267.731952	27.863884	0.000087	-0.369148	-2.998625
		360.0	-0.000003	-0.000166	4.0	4.0	196.514563	27.863884	0.000111	-3.403963	0.068048

1776 rows × 9 columns

Conclusion#

This notebook showcases the standalone utility of AMGeO’s new API, as well as the various ways it can be used in tandem with other popular python packages.

Please get started with AMGeO here, we would love any feedback to continue improving our software.

References#

AMGeO Collaboration (2019), A Collaborative Data Science Platform for the Geospace Community: Assimilative Mapping of Geospace Observations (AMGeO) v1.0.0, http://doi.org/10.5281/zenodo.3564914.

EarthCube 2022 Call for Notebooks

AMGeO 2.0: Crafting an API for Geospace Data Scientists

Contents

AMGeO 2.0: Crafting an API for Geospace Data Scientists#

Authors#

Table of Contents

Purpose#

Technical contributions#

Methodology#

Funding#

Keywords#

Citation#

Work in progress#

Suggested next steps#

Acknowledgements#

Setup#

Library imports#

Parameter definitions#

Data import#

Data processing and analysis#

AMGeO’s new API#

Creating an AMGeO API instance#

Setting our output directory for AMGeO data#

Creating an AMGeO controller#

Generating AMGeO maps#

Browsing AMGeO maps#

Loading AMGeO maps#

AMGeO’s xarray datasets#

Data variables#

Coordinates/Dimensions#

Dataset metadata#