Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adds cache support #21

Merged
merged 2 commits into from
Nov 28, 2023
Merged

adds cache support #21

merged 2 commits into from
Nov 28, 2023

Conversation

brunocalza
Copy link
Contributor

@brunocalza brunocalza commented Nov 21, 2023

Adds a --cache flag to the publication create command. It means how many minutes the data will stay in cache.

That information is sent to the provider. See tablelandnetwork/basin-provider#16

Also, it adds a new column EXIPIRES AT to the listing of files

+-------------------------------------------------------------+---------+---------------------------+----------+---------------------+
|                             CID                             |  SIZE   |         TIMESTAMP         | ARCHIVED |     EXPIRES AT      |
+-------------------------------------------------------------+---------+---------------------------+----------+---------------------+
| bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq | 1613287 | 2023-11-17T16:29:20-03:00 | Y        | 2023-11-24 05:52:36 |
| bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq | 1613287 | 2023-11-17T16:29:20-03:00 | Y        |                     |
| bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq | 1613287 | 2023-11-17T16:29:20-03:00 | Y        |                     |
+-------------------------------------------------------------+---------+---------------------------+----------+---------------------+

Note: no changes were made to the retrieve command. Theoretically, it should check if the file is in cache or not and make a decision. But there's no way for the CLI to implement that without hitting the provider to check that and download the file from GCS. That would mean implementing a download method in capnp, which seems too much effort for something we're moving away from. We have two options:

  • Make GCS bucket public, so that the CLI can hit the bucket direct and check if it's in cache
  • When adding an HTTP API, add a download endpoint

@brunocalza brunocalza marked this pull request as ready for review November 21, 2023 21:00
Copy link
Member

@sanderpick sanderpick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What will happen if retrieve command is used before the file is archived?

Nit: In the table you showed, it's not clear what is expiring. It could interpretted as if the archived data is expiring. how about "Cache Expiry"

@brunocalza
Copy link
Contributor Author

What will happen if retrieve command is used before the file is archived?

Right, now it will work just fine because the file is also in IPFS.

Nit: In the table you showed, it's not clear what is expiring. It could interpretted as if the archived data is expiring. how about "Cache Expiry"

Good suggestion 👍

Signed-off-by: Bruno Calza <brunoangelicalza@gmail.com>
Signed-off-by: Bruno Calza <brunoangelicalza@gmail.com>
@brunocalza brunocalza self-assigned this Nov 28, 2023
@brunocalza brunocalza merged commit c1a0412 into main Nov 28, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants