This document outlines the features of the {laminr} package and the roadmap for future development.

## Features

### Connect to an instance

- Connect to a LaminDB instance (`connect()`).
- Handle authentication and authorization.
- Connect to a LaminDB instance without needing to install the `lamin_cli` Python package.

### Query & search

- Query exactly one record (`Registry$get(...)`): Fetch a single record by ID.
- Query sets of records (`Registry$filter()`): Fetch multiple records based on filters.
  - `$df()`: Returns a data frame with each record in a row.
  - `$all()`: Returns all records as a `QuerySet`.
  - `$one()`: Return exactly one record.
  - `$one_or_none()`: Return one record or `NULL`.
- Leverage relationships when querying (`Artifact$filter(created_by__handle__startswith = "testuse")$df()`): Query records based on relationships.
- Comparators: Use comparators in filters.
  - `and`: Example: `Artifact$filter(suffix = ".jpg", created_by = user)`
  - `less than` / `greater than`: Example: `Artifact$filter(size__lt = 1e4)`
  - `in`: Example: `Artifact$filter(suffix_in = [".jpg", ".fastq.gz"])`
  - `order by`: Example: `Artifact$filter().order_by("created_at")`
  - `contains`: Example: `Artifact$filter(name__contains = "test")`
  - `startswith`: Example: `Artifact$filter(name__startswith = "test")`
  - `or`: Example: `...`
  - `not`: Example: `...`
- Search for records (`Registry$search(...)`): Search for records based on a query string.
- Pagination: Support pagination for large query results.
- Field lookups: Provide convenient functions for looking up field values (e.g., `Artifact$lookup("description")`).

### Manage data & metadata

- Create artifacts: Create new artifacts from various data sources (e.g., files, data frames, in-memory objects).
  - `$from_df()`: Create an artifact from a data frame.
  - `$from_path()`: Create an artifact from a path.
  - `$from_anndata()`: Create an artifact from an `AnnData`.
- Save artifacts: Save artifacts to LaminDB with appropriate metadata.
- Load artifacts: Load artifacts from LaminDB into R:
  - `csv`: Load a data frame from a CSV file.
  - `fcs`: Load flow cytometry data.
  - `h5ad`: Load an AnnData from an HDF5 file.
  - `h5mu`: Load a MuData from an HDF5 file.
  - `html`: Load content from an HTML file.
  - `jpg`: Load an image from JPG.
  - `json`: Load data from a JSON file.
  - `parquet`: Load a data frame from a Parquet file.
  - `png`: Load an image from PNG.
  - `rds`: Load an R object from an RDS file.
  - `svg`: Load an image from SVG.
  - `tsv`: Load a data frame from a TSV file.
  - `yaml`: Load data from a YAML file.
  - `zarr`: Load an AnnData from a Zarr store.
- Cache artifacts: Cache artifacts locally for faster access:
  - `s3`: Interact with S3 storage.
  - `gcp`: Interact with Google Cloud Storage.
- Version artifacts: Create new versions of artifacts.
- Delete artifacts: Delete an existing artifact.
- Manage artifact metadata: Add, update, and delete artifact metadata.
- Work with collections: Create, manage, and query collections of artifacts.
- Stream backed artifacts: Connect to file-backed artifacts (`$open`).
  - `tiledbsoma`: Stream TileDB-SOMA objects

### Track notebooks & scripts

- Track code execution: Automatically track the execution of R scripts and notebooks.
- Capture run context: Record information about the execution environment (e.g., package versions, parameters).
- Link code to artifacts: Associate code execution with generated artifacts.
  - Link to artifacts loaded from other instances
- Visualize data lineage: Create visualizations of data lineage and dependencies.
- Finalize tracking: End and save a run.

### Curate datasets

- Validate data: Validate data against predefined schemas or constraints.
- Standardize data: Apply standardization rules to ensure data consistency.
- Annotate data: Add annotations and labels to data.
- Use the Curator class: Implement the `Curator` class for a streamlined curation workflow.

### Access public ontologies

- Access ontology data: Fetch data from public ontologies (e.g., gene names, protein IDs).
- Search ontologies: Search for entities within ontologies.
- Use ontology terms in queries: Use ontology terms to filter and query data.
- Manage ontology versions: Access different versions of ontologies.

### Manage biological registries

- Create and manage records in bionty registries: Add, update, and delete records for genes, proteins, cell types, etc.
- Utilize hierarchical relationships: Navigate and query based on parent-child relationships in ontologies.
- Manage synonyms: Add and use synonyms for biological entities.

### Manage schema modules

- List available modules: Retrieve a list of available modules in an instance.
- Access module registries: Access registries within specific modules.
- (Advanced) Create custom modules: Define and register custom schema modules.

### Transfer data

- Upload data: Upload data files to LaminDB storage.
- Download data: Download data files from LaminDB storage.
- (Advanced) Support zero-copy data transfer: Implement efficient data transfer mechanisms.

## Roadmap

### Version 0.1.0

A first version of the package that allows users to:

- Connect to a LaminDB instance.
- List all records in a registry.
- Fetch one record by ID or UID.
- Cache S3 artifacts locally.
- Load AnnData artifacts.

### Version 0.2.0

- Implement basic data and metadata management features (create, save, load and delete artifacts).
- Expand support for different data formats.
- Implement code tracking.

### Version 0.3.0

- Track input artifacts.
- Support for more storage backends using a reticulate Python backend.

### Version 0.4.0

- Expand support for different storage backends.
- Expand query functionality with comparators, relationships, and pagination.
- Implement data lineage visualization.
- Introduce data curation features (validation, standardization, annotation).
- Enhance support for bionty registries and ontology interactions.
- Connect to TileDB-SOMA artifacts.

### Future versions

- Implement advanced features like custom module creation and zero-copy data transfer.
- Continuously improve performance, usability, and documentation.