A User Centered Engineering Journey to Escape GoPro’s Software Shackles
As we get closer to opening up Pixel8earth for more users we’ve been doing lots of dog-fooding. Everyone on the team has a 360 GoPro that we’ve been mapping our respective neighborhoods with. While we love the GoPro hardware and metadata (their telemetry is fabulous) the process of exporting videos from their software is tedious. This post will document our journey to reverse engineer various proprietary GoPro formats and specs so we can go directly from GoPro hardware to Pixel8 point cloud. First, though a quick dive into why we universally found GoPro software so challenging.
To 3D map the globe we are going to need to support a wide variety of cameras. 360 cameras are a great place to start since they provide such great coverage and diversity of perspectives for photogrammetry. The most popular consumer 360 cameras are the GoPro Fusion and Max.
This is where the challenges start. Fusion and Max have two different desktop apps for exporting videos. For the GoPro Fusion there is the Fusion Studio app now called the Exporter for GoPro Fusion. The Fusion hardware generates two MP4’s for each fish eye lens, and then Exporter for Fusion stitches those two MP4 files into a 360 video.
The new GoPro Max has a new proprietary .360 file format and a new exporter app that only runs on Windows. So, we have two different exporter apps and two different file formats. Leaving users to navigate this morass of formats and operating system support was a non-starter for us. Then top that off with waiting hours to stitch larger videos and flaky camera connectivity. There are several blog posts and community bulletin board posts dedicated to the topic. We needed another way.
In order to maximize user’s experience it was best to remove the GoPro middleman. Simply take the video file off your SD card and upload it to explore.pixel8.earth — done. To do so we needed to figure out how to manage GoPro imagery and handle all the metadata, like telemetry, on our own. In the case of the GoPro Fusion, where it generates two MP4’s for each fish-eye lens, this isn’t so bad, but add the proprietary GoPro .360 format and it gets pretty gnarly. A break down by camera.
GoPro Fusion
The primary challenge we ran into with the GoPro Fusion was generating a 360 video in Fusion Studio takes forever ever. Then you have to upload the, usually big, file which also takes a lot of time. Luckily there was a potential work around to save some of that time. Since Pixel8’s goal is creating point clouds we don’t directly need a 360 video. We can pass in the two fish eye videos and use a fish eye camera model instead of the pin hole model, which means we can optimize the SfM construction to be much faster than building a 360 video in Fusion Studio. The downside to this approach is since we loose the 360 videos we just have two cameras views that don’t overlap. That lack of overlap in photogrammetry is critical because it can lead to camera drift and bad 3D models. To avoid this we used Alicevision’s rig constraint that takes into account the cameras front/back geometry to restrict the camera’s relative position and achieve a similar result to overlap by managing camera drift. Also, since we aren’t using Fusion Studio we don’t end up with a 4k down sampled video and we can use the full resolution of the native imagery. The end result is that it is possible to upload raw data from a GoPro Fusion and generate high accuracy point clouds in our platform.
GoPro Max
While getting rid of the software dependencies of Fusion was tricky the Max upped the level of difficulty with a new .360 proprietary format. In addition to the new format GoPro also removed the telemetry track from the 360 video generation. So, we have to copy the telemetry track using the FFMPEG command line interface and then merge it back in with the exported Max 360 video. Fortunately with a bit of sleuthing Pramukta discovered that the GoPro .360 proprietary file is an MP4 containing an equi-angular cube map that is split into two tracks. For a great overview of equi-angular cube maps check out this fabulous post from Google.
In addition, the cube maps generated by GoPro have duplicate pixels that they likely uses for image alignment. It is also worth noting that the overlapping pixels used by GoPro and Google are slightly different. To get around the issue of variable pixel differences we removed the duplicate pixels with a 3D library (pyredner — a differentiable ray tracer). This allows us to reconstruct and render the equiangular cube map as an equirectangular panorama while also correcting for gravity. Then we can select any individual frame from the movie to process rather than having to stitch or reconstruct the whole 360 video. We do this based on the telemetry included with the video. The end results, as with the Fusion, is we can now let users upload the raw .360 file off the GoPro Max without needing a detour through their desktop software.
User Centered Engineering
If you’ve spent time with GoPro’s desktop applications, while contributing to awesome projects like Mapillary, you’ve likely felt the pain and long wait times associated with connecting your camera and generating videos. Since our goal isn’t creating 360 videos we saw an opportunity to invest some engineering time to make life a bit easier for future users. The reality of working with video and generating quite large point clouds and meshes is that nothing is going to happen quickly. So, any chance we get to cut time and get a cool result back to the user faster is something the team jumps on. That is a major goal as we get ready to launch — https://explore.pixel8.earth/. Look forward to sharing more as we get there.
