Design

The purpose of these design documents is to describe the design of Propagate in enough detail to guide a sustainable implementation (i.e., maintainable in the long term). It also ensures that the team has a shared understanding of the package’s scope and architecture.

Propagate’s design builds on our overall Seedcase design principles and patterns and follows the same principles and patterns described there.

Purpose

The overall aim of Propagate is to streamline requesting and subsetting data from a data package.

The central concept in Propagate is a set of instructions that describe a requested subset of data. These instructions are intended to be both machine-readable and human-readable, allowing them to be shared, reviewed, and archived. They also provide a reproducible specification for regenerating the same data subset at a later time.

Overall, Propagate:

  1. Creates structured instructions for requesting a subset of data.
  2. Provides an interactive interface for creating those instructions from metadata.
  3. Checks the instructions against the metadata before applying them to the data.
  4. Generates reproducible subsets from a data package based on the instructions.

Requirements

Specifically, Propagate must:

Core functionality

  • Define a machine-readable and human-readable instruction specification for requesting data subsets.
  • Store enough information in the instructions to reproduce the subset in the future.
  • Support checking instructions against the metadata before generating a subset.
  • Assume one request per research project per data package.

Data request functionality

  • Generate instructions interactively from the data package metadata via a web interface without downloading or installing software.
  • Support an “export” functionality: completed instructions should be downloadable as a file.
  • Support a “send” functionality: the requester’s email client should be opened with a pre-filled email to the data owner containing the instruction file as an attachment.
  • Allow users to save and resume partially completed requests.
  • Include request metadata in the instructions (e.g., requester name and email).
  • For experienced users, provide a terminal interface to create the request interactively without needing to open a web app.

Data management functionality

  • Generate a web component via a command-line interface for creating instructions interactively from data package metadata.
  • Support embedding the web component in a static website without requiring a server. In the Seedcase ecosystem, this would be the website generated by Template Data Package.
  • Generate data subsets in Parquet format for rectangular data from the instructions and the data from the data package.
  • Support previewing the planned subset operation before execution.
  • Check that the request is correctly structured, that it can be parsed to SQL, and that it has instructions that can be applied to the data package.

Out of scope

Propagate does not:

  • Manage data access approvals.
  • Transfer data subsets to the requester or to different servers.
  • Host web or server services.
  • Modify source data or metadata.
  • Act as a general-purpose query engine.
  • Store or organize the requests, the data, or the metadata.