Terra Populus, or TerraPop, is a cyberinfrastructure project that integrates, preserves, and disseminates massive data collections describing characteristics of the human population and environment over the last six decades. TerraPop has made a number of GIScience advances in the handling of big spatial data to make information interoperable between formats and across scientific communities. In this paper, we describe challenges of these data, or 'deserts in the deluge' of data, that are common to spatial big data more broadly, and explore computational solutions specific to microdata, raster, and vector data models.

The Terra Populus project (TerraPop) addresses a variety of data management, curation, and preservation challenges with respect to spatiotemporal population and environmental data. In this article, we describe our approaches to these challenges, with a particular focus on geospatial data workflows and associated provenance metadata. The goal of TerraPop is to enable research, learning, and policy analysis by providing integrated spatiotemporal data describing people and their environment. To do so, TerraPop is assembling a globe-spanning and temporally extensive collection of high-quality population and environmental data, ensuring good documentation, and developing a Web-based data access system that enables users to assemble customized integrated data sets drawing on a variety of data sources and formats. We describe TerraPop's collection strategies, detail the geospatial workflows involved in preparing data for ingest into the project database and those used to transform data across formats for dissemination, and discuss the system used to capture and manage provenance metadata throughout the project. A key aspect of the project is the development of global current and historical administrative unit boundaries that can be linked to census data. These boundaries serve as the linchpin of TerraPop's data integration strategy, and constitute an important data set in their own right.

Every year research institutions produce petabytes of data. Yet, only a small percent of the data is readily accessible for analysis. Terra Populus acts as the bridge between big data sources and researchers. Researchers are provided convenient web applications that allow them to access, analyze, and tabulate different datasets under a common platform. Terra Populus is developing three unique applications. The first application, Paragon, is a prototype parallel spatial database, which aims to extend the functionality of PostgreSQL and PostGIS onto multinode systems. Terra Populus Tabulator application employs Parquet on Spark to build dynamic queries for analyzing large population survey data. The last application, Terra Explorer, is an exploratory analysis tool for visualizing the spatial datasets within the repository.