Rust library interface

This document describes the design of the functions forming the Rust library and tracks the implementation status. All of these functions are external-facing functions that are used internally in the CLI commands.

We use symbols to indicate the status of implementation (see table below). For planned or in-progress work, we might include signatures, docstrings, and pseudocode to clarify the design. Once the interface is implemented, these are replaced by links to the reference documentation.

A table showing the symbols used to indicate the status of interface components, along with their descriptions.

Status	Description
	Interface that has been implemented.
	Interface that is currently being worked on.
	Interface that is planned, but isn’t being worked on currently.

Rap

This struct contains everything needed for later functions by bundling Package and Request structs together. It’s called Rap as an acronym for “Request And Package”. Nearly all internal functions make use of this struct.

read_package()

This function reads in the data package’s metadata file (currently only the datapackage.json file) into a Package struct. While we called it source in the args of many CLI commands, we call it package here to be clearer. “Source” on it’s own is a vague term, “source of what?”, so we wanted to be explicit that it is the data package’s metadata we are reading in. Plus, the struct is called Package. However, the read_package() has the argument called source, which is the location given in the SOURCE arg of many of the CLI commands.

At some point we aim to support other metadata formats, but for now only datapackage.json is supported.

The signature is (with an internal PackageSource enum for the different source locations):

read_package(source: PackageSource) -> Result<Package, Error>

read_request()

This reads in the request.yaml file into the Request struct. This functionality is very similar to read_package(). The signature is:

read_request(path: &str) -> Result<Request, Error>

write_request()

This writes the Request struct to a request.yaml file on disk. The signature is:

write_request(request: &Request, path: &str) -> Result<PathBuf, Error>

check_request()

The check_request() function takes the Rap struct that contains the Request and Package data and outputs whether any checks failed. It takes ownership of Rap in order to consume it as later functions shouldn’t use this object anymore. The output is a set of errors that indicate the problems if there as a failure, otherwise, outputs a CheckedRap object to indicate it has been checked. See the diagram below for an overview of what is going on internally. Once we start implementing it we’ll know how to group and organize these errors.

The approximate signature is:

check_request(rap: Rap) -> Result<CheckedRap, Error>

flowchart TD
    request[/"Request"/]
    package[/"Package"/]
    rap[/"Rap"/]
    output[/"Result\< CheckedRap, Errors \>"/]

    subgraph check ["check_request()"]
      check_package_name["check_package_name()<br>Package name matches metadata"]
      check_package_version["check_package_version()<br>Package version matches metadata"]
      check_resource_names["check_resource_names()<br>Resource names match metadata"]
      check_column_names["check_column_names()<br>Column names match metadata"]
      check_row_filters["check_row_filters()<br>Row filters match metadata columns"]
    end

    request & package --> rap

    rap -->
      check_package_name -->
      check_package_version -->
      check_resource_names -->
      check_column_names -->
      check_row_filters -->
      output

    check:::outer_function
    classDef outer_function fill:none

Figure 1: The flow of input and output through the check_request() function.

subset_resources()

subset_resources() takes the data package metadata and the request details and applies the request to the resources into a data subset. While also visually shown in the diagram below, briefly, it needs to:

1. Keep the selected resources.
2. Read in the resource data.
3. Filter each resource separately and keep the observational unit ID (usually the foreign key, ideally pointing to a central “ids” resource) from matching rows.
4. Combine the observational unit IDs of the kept rows from all resources.
5. Apply the combined IDs as a filter to each resource separately to identify which rows to keep.
6. Keep only the specified columns for each resource.
7. Output the kept data as one table per kept resource, along with the resource’s name (for later processing).

We need to filter all resources by the keys after the first row-filtering in order to create the correct population for the research project.

The signature is:

subset_resources(rap: CheckedRap) -> Result<Arc<[SubsettedResource]>, Errors>

flowchart TD
    CheckedRap[/"CheckedRap"/]
    output[/"Result\< <br>Arc\< [SubsettedResource] \>,<br>Error\>"/]

    CheckedRap --> get_resource_data

    subgraph subset_resources ["subset_resources()"]
      get_resource_data["get_resource_data()"]
      RequestedResource1[/"Vec\< [RequestedResource] \><br>data as mutable reference"/]
      map_keep_requested_rows["map(keep_requested_rows)"]
      RequestedResource2[/"Arc\< [RequestedResource] \><br>row-filtered data"/]
      map_get_ids["map(get_ids)"]
      AAIds[/"Arc\< [ Arc\< [Ids] \> ] \><br>Ids as String"/]
      flatten["flatten()"]
      ArcIds[/"Arc\< [Ids] \><br>Ids as string"/]
      unique["unique()"]
      UniqueIds[/"Arc\< [Ids] \><br>No duplicate Ids"/]
      RequestedResource3[/"Arc\< [RequestedResource] \><br>original data, now with ids field"/]
      map_subset_resource["map(subset_resource)<br>Keep rows with Ids and<br>select columns"]

      get_resource_data -->
        RequestedResource1 -->
        map_keep_requested_rows -->
        RequestedResource2 -->
        map_get_ids -->
        AAIds -->
        flatten -->
        ArcIds -->
        unique -->
        UniqueIds -->|add to struct| RequestedResource3 -->
        map_subset_resource
    end

    map_subset_resource --> output

    subset_resources:::outer_function
    classDef outer_function fill:none

Figure 2: The flow of input and output through the subset_resources() function.