AT 419, the first week

Week of 1/7/2019

Here on forward, this blog will act as a weekly progress log for the second capstone course, AT419.
This capstone will be a group effort as described in previous posts in this blog, in particular the posts in or near December. Being in the heart of winter with limited flying days and an exceptional need to design the workflows and communication infrastructure around the project, The first few weeks of this blog going forward will certainly be structured around organization of the project. Being that I am the primary member in charge of data, my initial tasks have been to build up the data infrastructure and processes so that once we have collected imagery, it can be organized and handled with the greatest efficiency. We have so many distinct roles in this project that I must be sure to communicate the structure of the data management effectively and provide the opportunity for ground members to give me feedback about what they would like to have access to and if my proposed processes are intelligible.

Wednesday Meeting and the Public Access Pipeline

Before our second meeting of the class, I had sketched notes regarding ideas for data management in my field notebook (Post Index, Figure 1). Using these notes, I led a discussion with the class to identify how to best store data within our two server drives, research and classes. We determined that it is likely best for everyone to have read access to everything. This is ideal so that people can see the state of processing, have download access to any information or data they might like at any state from collection to final deliverable, and check the data-team's work. However, we all agreed that not everyone needs write access to the research drive. The reason for this is not so much out of a lack of trust, but instead out of data integrity concerns. Accidents happen, and if someone deletes, moves, alters, or otherwise disturbs data or folder structure in the path where processing and analysis is occurring, it may be very difficult to correct the issue.

The address this problem, a common data dump will be available in the classes drive, within an AT419 folder.  Within the AT 419 folder is a folder for missions, where we are defining a mission as a single data gathering operation regardless of how many flights were needed to gather the requisite data, as long as the same aircraft and sensor was used (I.E. regardless of battery swaps). Within the mission folder, each mission is saved with the following naming structure:

mmddyy_ac_sensor_altitude

mmddyy = month/day/year
ac = aircraft
Sensor = sensor carried on aircraft
altitude = height the aircraft was flying agl

within the mission specific folders is a folder for each given flight, simply titled:

Fx_time

where x is the number, in order of conducted flights, needed to complete the mission. So the second flight after a battery change would have the designation, F2. The time is the 2400 clock time for which the flight began. We may be doing high temporal analysis so time values are being used.

Within each flight folder, meta data, imagery, and aircraft files specific to any given flight will be saved.

A visualization of this process can be seen in Post Index, Figure 1. Two diagrams created during this meeting showing two different options we were considering for folder structure can be seen in embedded figures 1 and 2.

After data is left by any team member in the data dump, myself or any future data team members will move the data over to the research drive where the data processing and analysis, as well as long term data storage, will be conducted. All other members will have access to the research drive to read or download, but not write, data. The research drive has not been fully mapped and the final research drive design will be finished in a later week.

Figure 1: A folder structure design with limited class access. Image also includes early metadata concept.

Figure 2: A later design showing the beginnings of the closed research drive architecture (left side dashed box) and how the data dump will act as the intermediate step where field crews drop data. The mission naming structure and internal file structure also drawn.

Metadata Setup

The other major task to be completed this week was the creation of a metadata format. The inspiration, but not final, version of the metadata format is seen in figure 1, or figure 1 of the post index. I created the final metadata values needed for each operation after it was discussed in meeting. In addition, I developed a guide describing how each metadata category should be filled out. To keep everything consistent, especially if some of the future analysis is run through script, some of the metadata categories have a code, such as the sensors. These codes will also be used in the file naming structure. After creating the metadata form and guide, I also created an example metadata sheet so that people can see how to best fill out the form. The metadata form, guide, and example are available for download here:

Meta Data Form, Guide, and Example Download

Next Steps

The next step on this topic is to finalize a research drive folder structure and create a guide to saving data with Ryan and Ian of the systems integration team. Being that they will have the say as to how the equipment is put away after a mission, a combined guide as to then store the data would be useful. However, neither of these are of the highest priority in the short term. In the next update, a completed or mostly started rendition of operation zones within our research area should be the primary topic. I will be using arcmap generated shapefiles to accomplish the zoning of flight areas in accordance with Kyle, the operations manager.

Post Index

 

 

(P.I.Figure 1): Notebook page for this week showing folder structure and metadata format