This repository has been archived by the owner on Sep 11, 2020. It is now read-only.
Proposal to improve memory performance decoding packfiles. #832
Labels
Comments
distorhead
added a commit
to werf/werf
that referenced
this issue
Sep 6, 2018
Go-git experiment failed for now. There is blocking issue to continue use go-git: * Corrupt patch generation for files without newline-at-the-end: src-d/go-git#936 And some non-blocking, but harmful: * Memory usage is too big on big repos: src-d/go-git#832. * Corrupt patch generation (solved in fork): src-d/go-git#932. Libgit2 is too complex to install or statically build into dapp. Also libgit2 has memory leaks. The decision is to use raw git command in a clean and efficient way. Implemented Diff operation.
distorhead
added a commit
to werf/werf
that referenced
this issue
Sep 6, 2018
Go-git experiment failed for now. There is blocking issue to continue use go-git: * Corrupt patch generation for files without newline-at-the-end: src-d/go-git#936 And some non-blocking, but harmful: * Memory usage is too big on big repos: src-d/go-git#832 * Corrupt patch generation (solved in fork): src-d/go-git#932 Libgit2 is too complex to install or statically build into dapp. Also libgit2 has memory leaks. The decision is to use raw git command in a clean and efficient way. Implemented Diff operation.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Proposal
Problem
Analyzing the code, I saw the following on packfile decoder:
This is forcing the read of all the object contents, even if we are not gonna use them.
Proposed solution
Instead of copy all into memory and resolve all the object deltas, create a DeltaReader abstraction to encapsulate that logic:
All the actual logic of the Object Cache and delta resolution will be exposed as a Reader. This Reader will contain the same logic as the actual one that is in the decoder, but it will be only executed if the Object.Read() method is called.
This reader will use the ObjectStorer to resolve deltas. The deltaBaseCache that is on the decoder should be moved inside the ObjectStorer.
Applying this solution will decrease drastically the memory usage. Instead of having all the objects in memory + deltaBaseCache objects + delta base + delta target, we are going to have only deltaBaseCache objects + delta base and target.
Caveats @jfontan @smola @mcuadros ?
The text was updated successfully, but these errors were encountered: