Geospatial Video: A Powerful and Underappreciated Tool in UAS

Outside of military surveillance operations, geospatial video is a data format that is likely unheard of for most commercial UAS operators. If not unheard of, at least poorly understood. Geospatial video is the inherent combination of geospatial information and video into a single data product. Essentially, geospatial video allows pilots and data analyst to understand where a UAV is positioned in space at any point along a recorded video. Depending on the video format, even the frame of view of the video can be spatially located along a basemap. At first glance, this may seem like a minor alteration to a given UAV video with little use, but this technology fills an important gap in the capabilities of UAS remote sensing.

Perhaps the most common data product created with UAS data, the orthophoto mosaic, has temporal and computational drawbacks. Comprised of an assortment of images "stitched" together to form a point cloud, orthomosaics take an exceptionally long time to process and can not take into account moving objects within the test area. Within many fields, a target topic of study may act on temporal scales too quick to be assessed in an orthomosaic or even too quick for a UAV flying a standard grid pattern. Two examples that come to mind in my beloved field of ecology are animal surveys and wildfire tracking. Animals that move at all are likely to be subtracted or duplicated from an orthomosaic. This effectively makes animal surveys through this method impossible. Even if you were looking through individual images that are used to generate a mosaic, the non-continuous nature of the data could make repeat observations more likely.  Wildfires, especially if tracking wildfire dynamics in real time, is another topic area that could benefit from the geospatial video technology. Either for the purpose of fire fighting or ecology, many of the concerns and interactions caused by wildfires occur at extremely fine temporal scales. Scales that the drone is capable of flying, but mosaic software can not account for. Many other topics of research could benefit from geospatial video within ecology include algal bloom research, population and community level interactions, and simply using drones as a recon tool to find and locate features ahead of time without the limited scaling on a typical grid transect pattern.

Generally speaking, faster image analysis is possible with geospatial video technology. Shape files could be created based on the aircraft path, so a nadir video following a physical feature could help inform cartographic and volumetric analysis. Real time analysis is possible with this method which could be critical for activities in conservation. Even just allowing for shorter field campaigns by enabling some of the observations to be made in the air makes this technology beneficial for the wildlife biologist.

While payware and better video cameras streamline the processing for this technology, simple UAVs that log GPS data can be used with freeware geospatial video software to produce professional quality product. I used Video GeoTagger FREE to investigate the ease of use of the technology and how it might be useful to the ecological research community.

Guide to use and my Experience

The geotagger software is available for free here.

After the software is downloaded and installed, the intro screen looks like below.

Conveniently,  a Youtube video is included in the top left corner that provides guidance on at least the most basic steps. The software is targeted for ESRI users as this can be used as an extension, but the steps are essentially the same.

The first thing to do is to load data points. Go to file and click "open video for geotagging".

The following prompt will be produced. Fill in the open categories with the desired video and GPS file. The file can be a .txt, .gpx, .SRT, among others.

 

At this point, the video (viewable in the top left panel when the Youtube video is not selected) and GPS data (viewable as points along the base map) are present. However, this data is not yet merged, they are two separate entities. You must select a point (preferably the first or last) from the GPS data on the basement and then select the "Geotag Video" option in the bottom left corner. This effectively merges the data set.

The interface after data has been imported but before it has merged

Once the data is merged, the media browser tab should be selected, and the icon for the video should be selected. Multiple merged videos can be present at one time, so it is important to select the correct video. The following image shows what this should look like. You may also notice the four new icons on the right side of the basemap.

After this step, when the video is played, the active location of the UAS is tracked in time with the video. The green dot on the basemap (in the following image) represents the UAV as it travels across the flight path. As the video plays, the four indicators on the right of the map also change showing heading, time, altitude, as well as general metadata (location in space as well as time and altitude) in descending order. At any point the video can be paused and the information for that point can be recorded. This is a massive advantage of this technology in that observations found in the video can be tied directly with metadata for that individual point in time (and thus feature). The following two images show a zoomed-in perspective of the flight tracker screen and the corresponding video clip.

Use and Weaknesses

As is hopefully apparent, the ability to pair metadata with video is crucial for tasks that need quick assessment or need to be monitored in real time. This tool is even more powerful when using the MISB video format that enables a bounding box of the camera extent to be created along the basemap as well as UAV position. I can also see this as being the primary means of UAV data collection in research groups without the infrastructure to use more expensive software or hardware.

However, there are some weaknesses to this approach, at least with the freeware version. Precision of point is a significant concern. As you may tell in the above image, the gps points in space are not entirely continuous. There is some lag in the continuous video among the stream of points. Meaning that if I pause the video twice in a row within a very short amount of time, both pauses may claim the same GPS point and associated metadata. Obviously this is not possible as the video stream pause locations, while similar, are not exactly the same. This may limit the technology somewhat for observations that require an immensely accurate GPS point. While none come to mind immediately, (perhaps an assessment of extremely heterogeneous landscapes) missions that require that much precision are likely better suited for different data products.

The other issue is scaling and validation of the basemap-GPS combination. While different basemap formats can be selected, including imagery, the natural resolution of UAS video may create a problem. If the drone data, which is extremely high resolution, does not match the lower resolution basemap, it may be difficult to interpret the exact location of points in space. How much this is an issue is difficult to quantify with a freeware software, but should be a consideration. Another problem with this software may come in the form of correcting for differences in the video and the basemap, or identifying differences in the least. This would especially be an issue when the video is taken of homogeneous landscapes. If the video is taken over a coniferous forest away from identifying man made features, for example, it could be extremely difficult to validate if the GPS points on the basemap are accurate at all. In this video, we had access to a road and we can clearly see the road in the basemap is also present in the video. Operators may not always have that luxury.

Strategy Sessions and Project Plan

Over the past few weeks, us capstone students have met to discuss the joint UAS capstone project. These meetings have really been focused on a few components:
 1) Where do we want to perform our capstone project
 2) What types of questions could we answer there
 3) What skills will be needed
 4) What type of Data would we like to collect
 5) What roles we will need and who will fill these roles
This report is the accumulation of the strategy session talks and what will happen going forward.

The Project

As stated in my bibliography and timeline blog post, our project will be based at the amphitheater park in the Lafayette Greater Area (The Tippecanoe County Amphitheater park). This site was chosen for the large variety of features we could analyze. It includes a very large parking lot, a full large scale amphitheater, hardwood forests, large grass areas (which will also help for launching fixed wing aircraft), hilly topography, streams, and a section of the Wabash River. Also, this site is phenologically dynamic because the Wabash river always floods in the Winter and Spring in that general area. We are also fortunate in that, during the week, this site is typically completely empty of pedestrians. This early in the process, we have decided not to make any fundamental decisions regarding what our final paper will answer (as we are mostly setting out to develop standardized practices for collecting extremely high spatial and temporal resolution data for a specific site). However, we are fairly certain we will try to analyze the impact of flooding at this site over time, perhaps developing models to predict flooding and water presence based on topography and land cover class. To do this, we are planning to collect data extremely frequently (several times a week) with a variety of sensors/data-types (multi-spectral, RGB, geospatial video, thermal, and maybe LiDAR) and an assortment of platforms (various multi-rotor aircraft and C-Astral Bramor ppX). Understandably, this will create an enormous quantity of data to analyze and will hopefully allow future groups to ask and answer questions for years. We will be writing a formal research publication to a not yet determined journal. 

Core Skills

- Effective and efficient flight operations and CRM
- Proper and informed data management and analysis
- Multi-faceted group organization 

Roles

We have made two main conclusions regarding roles for this project during our sessions. The first is that we need everyone to have a unique permanent role, but everyone should have a rotating secondary role.

The presence of a permanent role is critical for an assortment of reasons. The first is due to the scale of this project. We are dealing with so many systems, sensors, and data types for a very heterogeneous test site over a large amount of time. We are also the first to use this equipment and lab space so checklists, manuals, procedures, training protocols, tests, and data management structure all need to be developed. To achieve this, permanent roles structured around the skills of our diverse group will allow for standardization for each major component of the project. In effect, each person is in charge of leading all the developments in their section of the project. This will allow our group to finish a successful project by May despite the enormity of tasks that need to be accomplished. The permanent roles are ones that would be confusing, or perhaps even harmful to the overall project, if people switched into that role later in the semester.

Rotating roles are going to be important for each member. The structure of rotation is still yet to be determined, but essentially everyone will have an additional role in addition to their permanent role at any one time. This will allow everyone to get some experience at various aspects of the project and thus the UAS industry. This will also allow some of the permanent roles that are very time intensive to have additional help. Essentially, everyone's secondary rotated role is that of an assistant where they can gain experience at a task (such as data management and analysis) without structurally changing the procedures the permanent members have put in place. Some examples of these roles include being a member of the data team, systems integration, ground control team, or flight training.

In addition to one permanent role and a rotating role, each member will also act as a flight crew member. Each person will likely work in one or two crews at a maximum to promote continuing crew resource management and general operational understanding. This will also simplify scheduling. Each crew may specialize in a particular (or a few types of) aircraft, sensor, mission type, etc. This will likely help repeatability, accurate data collection, ease of training, and safety. However, exactly how this will work is still yet to be determined.

The following is a list of the permanent roles. It is important to recognize that these roles and their descriptions are very much susceptible to change. It is also important to recognize that the role descriptions encompass some, not all, of the tasks a role will strive to complete.

Operations Manager - Kyle Sheehan -
The operations manager is in charge of structuring flight and field operations in a way that is consistent and collects the appropriate data types at the correct time. The operations manager schedules the entirety of the flight operations and works with leaders in other parts of the project to ensure field operations are performing at the requisite level. They are the final authority on many of the documents that will need to be created related to aircraft operations.

Systems Integrator - Ryan Ferguson & Ian Willey -
The systems integrator is the primary member(s) focused on the aircraft and sensor integration and maintenance.  Their job is to fully understand the physical capabilities and limitations of the aircraft, draft appropriate documentation, and prepare the aircraft(s) ahead of each flight. This could include charging batteries, cleaning/maintaining appropriate equipment, integrating sensors to aircraft, performing and staying on top of updates on all equipment, and generally ensuring that flight crews and the data crew have easy access to the aircraft and data when needed. This role will have a rotating assistant member as needed because flight operations may be so frequent that more than two members may need to be present throughout the week. These members are also the primary enforcement for aircraft and aircraft component flight logs.

Lead Pilot/Flight Instructor - Todd Horn -
The lead pilot, otherwise known as flight instructor for this project, is in charge of mastering flight and operation of each aircraft and the required sensors. They will be the authority for determining if each flight crew is appropriately trained on each aircraft and if they could safely fly missions on their own. For that reason, he should work closely with the operations manager so that flight training sessions can be scheduled before the crew is sent on a data gathering flight. The lead pilot is also supposed to maintain knowledge as to the proficiency of each crew and determining if each crew needs more training. For example, if a crew goes several weeks without flying a particular system, it is recommended that the lead pilot goes with the crew to the operation to provide guidance as needed. This role is the critical for long term operational safety. A rotating member may be assigned to this role so that they can become proficient in all systems and then help check to see if other crews are operating correctly and safely.

Data Manager- Evan Hockridge -
This individual is responsible for all components of the data post operation. Tasks include developing a data management, security, processing, and analysis workflows. The data manager must master all the processing and analysis software needed to accomplish the project. This role requires close association with each of the other roles to guarantee the correct data is gathered, placed in the correct location in the correct way, and that the appropriate results can be concluded. Rotating (multiple) members will be assigned to be a part of the data team at any given time. This is because of the massive quantity of data that will be accumulated. The data manager will train the data team to store, process, and analyze the data correctly so that data can be assessed promptly.

Geospatial Crew Leader - Dylan McQueen -
The geospatial crew leader runs a small team of rotating members which perform ground fieldwork. In particular, this individual is responsible for the collect and quality of ground control measurements. The team will collect points at various scales (full-site and sub-site) so that operations can rely on previously collected permanent ground control. Because the site is predicted to be dynamic in nature, the crew leader should plan to recollected ground control at various points in the semester including before and after flood events. They are also in charge of standardizing placement and use of the aeropoint ground control for missions that will use them. Aeropoint procedures are needed because they are non-permanent points, so quality placement recommendations are needed for our site.

Lead Author - Krysta Rolle -
The lead author's primary task is focused towards the back of the semester where they will head the authorship of the inevitable research paper. However, before that point, they will also collect much of the general background research. They may also head up a weekly progress report as collected from each of the permanent group members. They will also help edit all the individual documents generated throughout the semester.

Project Manager - Thoma Gonya -
The project manager may be a semi-permanent role, or have a full rotation where a member who specializes in a particular aspect of the project takes lead when needed (example: Integration PM early on or a Data PM later on). The project manager is to ensure that all the components of the project work together for the common goal. Effectively, they find and minimize weaknesses in our approach and maximize on strengths. They set and adjust the overall project timeline, set subgoals for the various groups within the project, and assign deliverables. The PM also sets important meetings and during meetings, acts as the organizational voice so that all important topics are covered during the meetings.

Likely Early Deliverables

This is a short list of some of the deliverables that will be needed very early in the semester (within the first two weeks or so). By having these finished early, we have the best chance of finishing on time.

1) Pre, during, and post flight checklists
2) data storage and management procedures (including folder pathing)
3) Systems integration guides for each sensor and aircraft (that we have at the beginning of the semester)
4) Flight training procedures and schedule
5) Early flight schedule
6) General semester timeline
7) Early field notes of site
8) preliminary data
9) Ground control point information
10) Preliminary/early flight plans
11) Meta-data saving guide
12) Meeting schedule and plan adjustment points