Skip to content

Commit

Permalink
Merge pull request #29 from tablelandnetwork/bcalza/rename
Browse files Browse the repository at this point in the history
renaming commands to vaults terminology
  • Loading branch information
brunocalza committed Dec 19, 2023
2 parents 3d16294 + 7576c69 commit 563b4c2
Show file tree
Hide file tree
Showing 19 changed files with 707 additions and 553 deletions.
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
/basin
/vaults
/cover.out
.vscode/*
7 changes: 6 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,14 @@ lint:

# Build
build:
go build -o basin cmd/basin/*
go build -o vaults cmd/vaults/*
.PHONY: build

# Install
install:
go install ./cmd/vaults
.PHONY: install

# Test
test:
go test ./... -short -race -timeout 1m
Expand Down
156 changes: 90 additions & 66 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,47 +1,58 @@
# basin-cli
# vaults-cli

[![License](https://img.shields.io/github/license/tablelandnetwork/basin-cli.svg)](./LICENSE)
[![standard-readme compliant](https://img.shields.io/badge/standard--readme-OK-green.svg)](https://github.com/RichardLitt/standard-readme)

> Continuously publish data from your database to the Tableland network.
> Continuously publish data from your database or file uploads to the Tableland Vaults network.
# Table of Contents
## Table of Contents

- [Install](#install)
- [Postgres Setup](#postgres-setup)
- [Self-hosted](#self-hosted)
- [Amazon RDS](#amazon-rds)
- [Background](#background)
- [Usage](#usage)
- [Install](#install)
- [Postgres Setup](#postgres-setup)
- [Supabase](#supabase)
- [Create a publication](#create-a-publication)
- [Start replicating a publication](#start-replicating-a-publication)
- [Upload a Parquet file](#upload-a-parquet-file)
- [Listing Publications](#listing-publications)
- [Listing Deals](#listing-deals)
- [Running](#running)
- [Run tests](#run-tests)
- [Retrieving](#retrieving)
- [Create a vault](#create-a-vault)
- [Start replicating a database](#start-replicating-a-database)
- [Write a Parquet file](#write-a-parquet-file)
- [Listing Vaults](#listing-vaults)
- [Listing Events](#listing-events)
- [Retrieving](#retrieving)
- [Development](#development)
- [Running](#running)
- [Run tests](#run-tests)
- [Contributing](#contributing)
- [License](#license)

# Background
## Background

Tableland Basin is a secure and verifiable open data platform. The Basin CLI is a tool that allows you to continuously replicate a table or view from your database to the network. Currently, only PostgreSQL is supported.
Textile Vaults is a secure and verifiable open data platform. The Vaults CLI is a tool that allows you to continuously replicate a table or view from your database to the network (currently, only PostgreSQL is supported). Or, you can directly upload files to the vault (currently, parquet is only supported)

> 🚧 Basin is currently not in a production-ready state. Any data that is pushed to the network may be subject to deletion. 🚧
> 🚧 Vaults is currently not in a production-ready state. Any data that is pushed to the network may be subject to deletion. 🚧
# Usage
## Usage

## Install
### Install

You can either install the CLI from the remote source:

```bash
go install github.com/tablelandnetwork/basin-cli/cmd/vaults@latest
```

Or clone from source and run the Makefile `install` command:

```bash
git clone https://github.com/tablelandnetwork/basin-cli.git
cd basin-cli
go install ./cmd/basin
make install
```

## Postgres Setup
### Postgres Setup

### Self-hosted
#### Self-hosted

- Make sure you have access to a superuser role. For example, you can create a new role such as `CREATE ROLE basin WITH PASSWORD NULL LOGIN SUPERUSER;`.
- Make sure you have access to a superuser role. For example, you can create a new role such as `CREATE ROLE vaults WITH PASSWORD NULL LOGIN SUPERUSER;`.
- Check that your Postgres installation has the [wal2json](https://github.com/eulerto/wal2json) plugin installed.
- Check if logical replication is enabled:

Expand All @@ -53,7 +64,7 @@ go install ./cmd/basin

- Restart the database in order for the new `wal_level` to take effect (be careful!).

### Amazon RDS
#### Amazon RDS

- Make sure you have a user with the `rds_superuser` role, and use `psql` to connect to your database.

Expand All @@ -69,7 +80,7 @@ go install ./cmd/basin
WHERE name = 'rds.logical_replication';
```

- If it's on, go to [Create a publication](#create-a-publication)
- If it's on, go to [Create a vault](#create-a-vault)
- If it's off, follow the next steps:
- [Create a custom RDS parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithDBInstanceParamGroups.html#USER_WorkingWithParamGroups.Creating)
- After creation, edit it and set the `rds.logical_replication` parameter to `1`
Expand All @@ -83,127 +94,140 @@ go install ./cmd/basin
- Log into the [Supabase](https://supabase.io/) dashboard and go to your project, or create a new one.
- Check if logical replication is enabled. This should be the default setting, so you shouldn't have to change anything. You can do this in the `SQL Editor` section on the left hand side of the Supabase dashboard by running `SHOW wal_level;` query, which should log `logical`.
- You can find the database connection information on the left hand side under `Project Settings` > `Database`. You will need the `Host`, `Port`, `Database`, `Username`, and `Password` to connect to your database.
- When you create a publication, the `--dburi` should follow this format:
- When you create a vault, the `--dburi` should follow this format:
```sh
postgresql://postgres:[PASSWORD]@db.[PROJECT_ID].supabase.co:5432/postgres
```

## Create a publication
### Create a vault

_Publications_ define the data you are pushing to Basin.
_Vaults_ define the place you push data into.

Basin uses public key authentication, so you will need an Ethereum style (ECDSA, secp256k1) wallet to create a new publication. You can use an existing wallet or set up a new one with `basin wallet create`. Your private key is only used locally for signing.
Vaults uses public key authentication, so you will need an Ethereum style (ECDSA, secp256k1) wallet to create a new vault. You can use an existing wallet or set up a new one with `vaults wallet create`. Your private key is only used locally for signing.

```bash
basin wallet create [FILENAME]
vaults account create [FILENAME]
```

A new private key will be written to `FILENAME`.

The name of a publication contains a `namespace` (e.g. `my_company`) and the name of an existing database relation (e.g. `my_table`), separated by a period (`.`). Use `basin publication create` to create a new publication. See `basin publication create --help` for more info.
The name of a vault contains a `namespace` (e.g. `my_company`) and the name of an existing database relation (e.g. `my_table`), separated by a period (`.`). Use `vaults create` to create a new vault. See `vaults create --help` for more info.

```bash
basin publication create --dburi [DBURI] --address [WALLET_ADDRESS] namespace.relation_name
vaults create --dburi [DBURI] --account [WALLET_ADDRESS] namespace.relation_name
```

🚧 Basin currently only replicates `INSERT` statements, which means that it only replicates append-only data (e.g., log-style data). Row updates and deletes will be ignored. 🚧
🚧 Vaults currently only replicates `INSERT` statements, which means that it only replicates append-only data (e.g., log-style data). Row updates and deletes will be ignored. 🚧

## Start replicating a publication
### Start replicating a database

Use `basin publication start` to start a daemon that will continuously push changes to the underlying table/view to the network. See `basin publication start --help` for more info.
Use `vaults stream` to start a daemon that will continuously push changes to the underlying table/view to the network. See `vaults stream --help` for more info.

```bash
basin publication start --private-key [PRIVATE_KEY] namespace.relation_name
vaults stream --private-key [PRIVATE_KEY] namespace.relation_name
```

## Upload a Parquet file
### Write a Parquet file

Before uploading a Parquet file, you need to [Create a publication](#create-a-publication), if not already created. You can omit the `--dburi` flag, in this case.
Before writing a Parquet file, you need to [Create a vault](#create-a-vault), if not already created. You can omit the `--dburi` flag, in this case.

Then, use `basin publication upload` to upload a Parquet file.
Then, use `vaults write` to write a Parquet file.

```bash
basin publication upload --name [namespace.relation_name] --private-key [PRIVATE_KEY] filepath
vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] filepath
```

You can attach a timestamp to that file upload, e.g.
You can attach a timestamp to that file write, e.g.

```bash
basin publication upload --name [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 1699984703 filepath
vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 1699984703 filepath
# or use data format
basin publication upload --name [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 2006-01-02 filepath
vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 2006-01-02 filepath
# or use RFC3339 format
basin publication upload --name [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 2006-01-02T15:04:05Z07:00 filepath
vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 2006-01-02T15:04:05Z07:00 filepath
```

If a timestamp is not provided, the CLI will assume the timestamp is the current client epoch in UTC.

## Listing Publications
### Listing Vaults

You can list the publications from an address by running:
You can list the vaults from an account by running:

```bash
basin publication list --address [ETH_ADDRESS]
vaults list --account [ETH_ADDRESS]
```

## Listing Deals
### Listing Events

You can list deals of a given publication by running:
You can list events of a given vault by running:

```bash
basin publication deals --publication [PUBLICATION] --latest 5
vaults events --vault [VAULT_NAME] --latest 5
```

Deals command accept `--before`,`--after` , and `--at` flags to filter deals by timestamp
Events command accept `--before`,`--after` , and `--at` flags to filter events by timestamp

```bash
# examples
basin publication deals --publication demotest.data --at 1699569502
basin publication deals --publication demotest.data --before 2023-11-09T19:38:23-03:00
basin publication deals --publication demotest.data --after 2023-11-09
vaults events --vault demotest.data --at 1699569502
vaults events --vault demotest.data --before 2023-11-09T19:38:23-03:00
vaults events --vault demotest.data --after 2023-11-09
```

## Retrieving
### Retrieving

You can retrieve a file from a vault by running:

```bash
basin publication retrieve bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq
vaults retrieve bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq
```

# Development
You can also specify where to save the file:

## Running
```bash
vaults retrieve --output /path/to/dir bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq
```

Or stream the file to stdout the `-` value (note: the short form `-o` is for `--output`), and then pipe it to something like [`car extract`](https://github.com/ipld/go-car) to unpack the CAR file's contents:
```bash
vaults retrieve -o - bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq | car extract
```
## Development
### Running
You can make use of the scripts inside `scripts` to facilitate running the CLI locally without building.
```bash
# Starting the Provider Server
PORT=8888 ./scripts/server.sh
# Create a wallet
./scripts/run.sh wallet create pk.out
# Create an account
./scripts/run.sh account create pk.out
# Start replicating
./scripts/run.sh publication start --private-key [PRIVATE_KEY] namespace.relation_name
./scripts/run.sh vaults stream --private-key [PRIVATE_KEY] namespace.relation_name
```
## Run tests
### Run tests
```bash
make test
```
Note: One of the tests requires Docker Engine to be running.

# Contributing
## Contributing
PRs accepted.
Small note: If editing the README, please conform to the
[standard-readme](https://github.com/RichardLitt/standard-readme) specification.
# License
## License
MIT AND Apache-2.0, © 2021-2023 Tableland Network Contributors
75 changes: 0 additions & 75 deletions cmd/basin/config.go

This file was deleted.

Loading

0 comments on commit 563b4c2

Please sign in to comment.