Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

More automated Sync/Backup options #338

Open
Goldendawn69 opened this issue Jul 15, 2024 · 19 comments
Open

More automated Sync/Backup options #338

Goldendawn69 opened this issue Jul 15, 2024 · 19 comments
Labels
enhancement New feature or request

Comments

@Goldendawn69
Copy link

Is your feature request related to a problem? Please describe.

Syncing between multiple systems/browsers is not really what interests me here. I am interested in backup capabilities, which I feel are lacking.

I wish there were a way to use the Sync/backup functionality, but I don't want to sync my data to your server due to security concerns. Sorry to say, but being a long-term technology person, I know there are too many situations where data has been leaked and used in situations where it should not be happening.

I am perfectly happy to fund you every month through your Patreon page, but there need to be other options for syncing/backing up data.

Describe the solution you'd like

It would be good if we could sync/backup automatically to Google, Dropbox, or other prime cloud-based drives. We could, in fact, do this now (by backing up local to Google Drive), but there is no automatic backup process.

What would be good is to offer an automatic backup process to backup the content of the GSD (it doesn't have to include Thumbnails) and allow them to be either synced or backed up locally. Again, automatic backup could be offered as part of your monthly Pro fee if you felt the need.
@Goldendawn69 Goldendawn69 added the enhancement New feature or request label Jul 15, 2024
@Juraj-Masiar
Copy link
Member

Juraj-Masiar commented Jul 15, 2024 •
edited
Loading

Hello,
I would love to integrate the popular storage services, but it's not easy, at all.
It requires OAuth implementation on the client (in the addon), and I would have to bundle also some google library in my addon that handles their API, which is not great either.
Additionally, using OAuth means regular updates (usually not breaking though) and various checkups by the provider service (usually only once a year).

So, I would rather focus on some "alternative solutions", maybe some out of box ideas.
This is what I can think of:

  1. addons can create a folder with files inside, in your default downloads folder (this depends where your browser downloads files, so either it's the "user/Downloads" folder, or "user/Desktop".
    For example, I could create some kind of automatic backup in this path: <default downloads folder>/_backups/export_2024_07_09___15_18_01.group_speed_dial.
    You could then let your favorite storage provider to backup data in this folder (if it's supported).
    And if the extra folder bothers you, I think you can mark is as hidden (or as "system"), and it would "disappear". Sadly, in Linux this requires folder name starting with a ".", which is forbidden for addons.

  2. I could build a "End to end" encrypted synchronization, so you would set a strong and long password locally, then it would be saved in the browser so that you don't have enter it each time browser starts, and all data send to my server would be encrypted, so not readable to me or any attacker.
    The main issue is, that if you would lost your device (or if your SSD dies) and you can't remember your password, you would loose your dials data forever.

UPDATE:
There is a new "File System API" that may allow me to store backups in any user-selected folder. If this works (also after restarting PC), I will use it for optional regular backups. This will allow full backups including thumbnails.
I could then create a full backup once a week or so...
And since this targets non-server-using users, I will have to make it free :)
I'll put it on my todo list, somewhere up.

@Goldendawn69
Copy link
Author

I understand the OAuth conversation - though I know many applications that support that model.

But as stated, if there is a local capability to save your file, then people can put it into their local Dropbox, Google folder, etc. They (or like me) could backup the file offsite (I do my backups encrypted into Google Drive and Dropbox) myself, with my own encryption.

I like your approach, and if you implement it, I would support your service anyway because then the solution you are presenting would do everything I need. FVD saves locally, but they haven't been interested in making it automated for Chrome (though I do believe they did have it in Firefox) and are planning on changing that behaviour (at some stage) to force people onto their Eversync service.

Also, I liked version 2 of your suggestion—this would add that level of encryption and safety I feel is missing right now. I know you have encryption, but honestly, because of other sites' behaviours, our data (our history) is important. In all situations where I am sending data to external sites, I have a private token that encrypts the data. And if I "lost my token," that is my problem, as is everything else I have. I am responsible for storing my token and keeping it safe - otherwise, I will lose it all.

(isn't that what that piece of paper is that you store under your keyboard... with your secret information and all your passwords for :D)

Which would be easier to implement - or - would both of them be good?

Because of your response, because of the way you have already thought on this, I have decided to join your Patreon because this is the type of awesome developer behaviour we should support! So thanks.

@Juraj-Masiar
Copy link
Member

Oh, thank you for supporting my work! That's very generous of you.
Now I have to move it to the top of my todo list :)

I will be implementing both options, but I will start with the easier one, so locally created backup files.
I've already read the documentation so I'll start implementing it now, so maybe next week it could be released.

There seems to be one issue though, if I'm seeing it right, only Chromium browsers currently support selecting a directory outside the downloads folder.
But we will see soon, I'm writing proof of concept now.

@Goldendawn69
Copy link
Author

Goldendawn69 commented Jul 16, 2024 via email

@Juraj-Masiar
Copy link
Member

Thanks for the stackoverflow link!
Sadly, it won't help, the polyfill is missing the crucial feature:

Storing a file handle in IndexedDB or sharing it with postMessage isn't currently possible unless you use native.

You see, once the user selects the folder, the app will get access to it ONLY while it holds the "approval" object (the handler).
So in order to be able to do a an automatic backup to that folder, I need to store the handler and then load it when needed.

To be honest, this sounds too good to be true :), and I'm giving it 80% chance it's actually gonna work as expected :). Addons/extensions environment is sometimes too special for new features like this.

In any case, the automatic storing to the Downloads folder (sub-folder) should work, so this is what I'll use for Firefox.
And if it works fine in Chrome, I'll request the missing API in the Firefox bugzilla page.

I still didn't do the proof of concept, I've discovered some random crazy huge memory leak (40GB of RAM!) that requires my attention, as it's likely related to my addon.

Thank you for the words of encouragement :), I'll do my best to make you a happy customer!

@Goldendawn69
Copy link
Author

Goldendawn69 commented Jul 18, 2024 via email

@Juraj-Masiar
Copy link
Member

Actually one of the Firefox engineers is helping me identify the issue, since we were not sure about the cause of the leak.
So far it looks like my addon is trying to store something huge into "in-memory" storage and Firefox should not allow it (it should have a limit of 10MB). Firefox is also very slow releasing used memory as we discovered.

Anyway, I've spend almost whole day yesterday with checking my code so today I'll continue with normal work and once I see the leak again (it's a bit rare right now, I've seen it only on my home PC and only like 5 times), I'll check the store manually to see what monstrosity is there.

@Juraj-Masiar
Copy link
Member

Hello,
I've just finished implementing a proof of concept, and as I feared, it doesn't work as smooth as expected.
The "permission" (the directory handle) is valid only for the current "session", so after restarting browser, the handle stored in IndexedDB is "expired": DOMException: The request is not allowed by the user agent or the platform in the current context.

It seems to be known issue:
https://stackoverflow.com/questions/66500836/domexception-the-request-is-not-allowed-by-the-user-agent-or-the-platform-in-th

HOWEVER, as was writing this, and further testing it, suddenly Chrome asked me, whether I want to permanently allow access to the backup folder! I think I just need to re-request it twice and it will trigger the extra permission popup:
image

And that seems to fix the issue!
I can now create backup, asynchronously, from background script, without any user interaction.
So good news! :)

@Goldendawn69
Copy link
Author

Goldendawn69 commented Jul 26, 2024 via email

@Juraj-Masiar
Copy link
Member

I'm working on the Firefox workaround right now, it will be able to create backups in the default downloads folder, in a specified subfolder.
Chrome version is already finished:
image

This will be creating a full backups, including thumbnails in a specific location (wherever on the disk).

@Juraj-Masiar
Copy link
Member

Done, new version 25.9 is waiting for review and brings folder specific offline backups in Chrome.
In Firefox, it can create backups in the user specified subfolder in the Downloads folder.

The permissions are super strict though (in Chrome). It will ask you to allow access to the folder when you first select it, but then also while making the first backup (after restarting Chrome, week or two later) - at that time it will give you option to always allow access to the folder.

I'm sorry it took me so long, I've been sick for more than a week plus I wanted to fix some issues in this release.

@Goldendawn69
Copy link
Author

Goldendawn69 commented Aug 21, 2024 via email

@Juraj-Masiar
Copy link
Member

Well, I'm preparing a post about new features for my Patreon page.
And I did wanted to make a short videos for the new features too, I've totally forgot about that.
But it should be intuitive enough even without the "manual", I wish I could watch you how you use it, it's always super useful to see how my "intuitive design" is actually a total mess :D.

In any case, the new version is now available for Chrome, if you didn't get it yet, you can force the update:
https://github.com/fastaddons/GroupSpeedDial/wiki/How-to-update-to-latest-version

@Goldendawn69
Copy link
Author

Goldendawn69 commented Sep 22, 2024 via email

@Juraj-Masiar
Copy link
Member

Thanks for the feedback!
Regarding the security question - it should have asked you during the second backup if you want to "Allow on every visit", but if you clicked "Allow this time", it will not ask you to allow it forever again. At least that's what I've seen in my tests.

@Goldendawn69
Copy link
Author

Goldendawn69 commented Sep 23, 2024 via email

@Juraj-Masiar
Copy link
Member

Oh, you are right, I've just tested it and Chrome indeed removed the handler.
They add all these security features but they don't mention them in the documentation :(

@Goldendawn69
Copy link
Author

Goldendawn69 commented Sep 25, 2024 via email

@Juraj-Masiar
Copy link
Member

Well, Firefox doesn't support the file system API (due to security concerns), but the Downloads API I've used there as a backup works pretty fine, it doesn't ask anything, the only issue is that now I have a "gsd" folder in my Downloads folder :).

Regarding the flag - that's something else, it's for an "unload" event that fires when you navigate away from a page, at least I think.
Also, note that you can't send images through the email reply into the Github issue. This email is just a github notification, if you want to upload image you need to do it on the web: #338

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Juraj-Masiar @Goldendawn69