-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adds progress Allocation to represent Decentralized Allocation Tree node. #1298
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
100 changes: 100 additions & 0 deletions
100
jib-core/src/main/java/com/google/cloud/tools/jib/event/progress/Allocation.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,100 @@ | ||
/* | ||
* Copyright 2018 Google LLC. | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); you may not | ||
* use this file except in compliance with the License. You may obtain a copy of | ||
* the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, WITHOUT | ||
* WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the | ||
* License for the specific language governing permissions and limitations under | ||
* the License. | ||
*/ | ||
|
||
package com.google.cloud.tools.jib.event.progress; | ||
|
||
import java.util.Optional; | ||
import javax.annotation.Nullable; | ||
|
||
/** | ||
* Represents a Decentralized Allocation Tree (DAT) node. | ||
* | ||
* <p>A DAT node is immutable and pointers only go in the direction from child to parent. Each node | ||
* has a set number of allocated units, the total of which represents a single allocation unit of | ||
* its parent. Each node is therefore a sub-allocation of its parent node. This allows the DAT to | ||
* sub-allocate progress in a decentralized, asynchronous manner. | ||
* | ||
* <p>For example, thread 1 creates node A as the root node with 2 allocation units. A subtask is | ||
* launched on thread 1 and creates node B with 3 allocation units as a child of node A. Thread 1 | ||
* then also launches a subtask on thread 2 that creates node C with 5 allocation units. Once the | ||
* first subtask finishes and reports its progress, that completion would entail completion of 3 | ||
* allocation units of node B and 1 allocation unit of node A. The second subtask finishes and | ||
* reports its progress as well, indicating completion of 5 units of node C and thus 1 unit of node | ||
* A. Allocation A is then deemed complete as well in terms of overall progress. | ||
* | ||
* <p>Note that it is up to the user of the class to ensure that the number of sub-allocations does | ||
* not exceed the number of allocation units. | ||
*/ | ||
class Allocation { | ||
|
||
/** | ||
* Creates a new root {@link Allocation}. | ||
* | ||
* @param description thuser-facing description of what the allocation represents | ||
* @param allocationUnits number of allocation units | ||
* @return a new {@link Allocation} | ||
*/ | ||
static Allocation newRoot(String description, long allocationUnits) { | ||
return new Allocation(description, allocationUnits, null); | ||
} | ||
|
||
/** The parent {@link Allocation}, or {@code null} to indicate a root node. */ | ||
@Nullable private final Allocation parent; | ||
|
||
/** User-facing description of what the allocation represents. */ | ||
private final String description; | ||
|
||
/** The number of allocation units this node holds. */ | ||
private final long allocationUnits; | ||
|
||
/** How much of the root allocation (1.0) this allocation accounts for. */ | ||
private final double fractionOfRoot; | ||
|
||
private Allocation(String description, long allocationUnits, @Nullable Allocation parent) { | ||
this.description = description; | ||
this.allocationUnits = allocationUnits; | ||
this.parent = parent; | ||
|
||
this.fractionOfRoot = parent == null ? 1.0 : parent.fractionOfRoot / parent.allocationUnits; | ||
} | ||
|
||
/** | ||
* Creates a new child {@link Allocation} (sub-allocation). | ||
* | ||
* @param description user-facing description of what the sub-allocation represents | ||
* @param allocationUnits number of allocation units the child holds | ||
* @return a new {@link Allocation} | ||
*/ | ||
Allocation newChild(String description, long allocationUnits) { | ||
return new Allocation(description, allocationUnits, this); | ||
} | ||
|
||
Optional<Allocation> getParent() { | ||
return Optional.ofNullable(parent); | ||
} | ||
|
||
String getDescription() { | ||
return description; | ||
} | ||
|
||
long getAllocationUnits() { | ||
return allocationUnits; | ||
} | ||
|
||
double getFractionOfRoot() { | ||
return fractionOfRoot; | ||
} | ||
} |
93 changes: 93 additions & 0 deletions
93
jib-core/src/test/java/com/google/cloud/tools/jib/event/progress/AllocationTest.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,93 @@ | ||
/* | ||
* Copyright 2018 Google LLC. | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); you may not | ||
* use this file except in compliance with the License. You may obtain a copy of | ||
* the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, WITHOUT | ||
* WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the | ||
* License for the specific language governing permissions and limitations under | ||
* the License. | ||
*/ | ||
|
||
package com.google.cloud.tools.jib.event.progress; | ||
|
||
import org.junit.Assert; | ||
import org.junit.Test; | ||
|
||
/** Tests for {@link Allocation}. */ | ||
public class AllocationTest { | ||
|
||
/** Error margin for checking equality of two doubles. */ | ||
private static final double DOUBLE_ERROR_MARGIN = 1e-10; | ||
|
||
@Test | ||
public void testSmoke_linear() { | ||
Allocation root = Allocation.newRoot("root", 1); | ||
Allocation node1 = root.newChild("node1", 2); | ||
Allocation node2 = node1.newChild("node2", 3); | ||
|
||
Assert.assertEquals("node2", node2.getDescription()); | ||
Assert.assertEquals(3, node2.getAllocationUnits()); | ||
Assert.assertEquals(1.0 / 2, node2.getFractionOfRoot(), DOUBLE_ERROR_MARGIN); | ||
Assert.assertTrue(node2.getParent().isPresent()); | ||
Assert.assertEquals(node1, node2.getParent().get()); | ||
|
||
Assert.assertEquals("node1", node1.getDescription()); | ||
Assert.assertEquals(2, node1.getAllocationUnits()); | ||
Assert.assertTrue(node1.getParent().isPresent()); | ||
Assert.assertEquals(root, node1.getParent().get()); | ||
Assert.assertEquals(1.0, node1.getFractionOfRoot(), DOUBLE_ERROR_MARGIN); | ||
|
||
Assert.assertEquals("root", root.getDescription()); | ||
Assert.assertEquals(1, root.getAllocationUnits()); | ||
Assert.assertFalse(root.getParent().isPresent()); | ||
Assert.assertEquals(1.0, root.getFractionOfRoot(), DOUBLE_ERROR_MARGIN); | ||
} | ||
|
||
@Test | ||
public void testFractionOfRoot_tree_partial() { | ||
Allocation root = Allocation.newRoot("ignored", 10); | ||
Allocation left = root.newChild("ignored", 2); | ||
Allocation right = root.newChild("ignored", 4); | ||
Allocation leftDown = left.newChild("ignored", 20); | ||
Allocation rightLeft = right.newChild("ignored", 20); | ||
Allocation rightRight = right.newChild("ignored", 100); | ||
Allocation rightRightDown = rightRight.newChild("ignored", 200); | ||
|
||
Assert.assertEquals(1.0, root.getFractionOfRoot(), DOUBLE_ERROR_MARGIN); | ||
Assert.assertEquals(1.0 / 10, left.getFractionOfRoot(), DOUBLE_ERROR_MARGIN); | ||
Assert.assertEquals(1.0 / 10, right.getFractionOfRoot(), DOUBLE_ERROR_MARGIN); | ||
Assert.assertEquals(1.0 / 10 / 2, leftDown.getFractionOfRoot(), DOUBLE_ERROR_MARGIN); | ||
Assert.assertEquals(1.0 / 10 / 4, rightLeft.getFractionOfRoot(), DOUBLE_ERROR_MARGIN); | ||
Assert.assertEquals(1.0 / 10 / 4, rightRight.getFractionOfRoot(), DOUBLE_ERROR_MARGIN); | ||
Assert.assertEquals( | ||
1.0 / 10 / 4 / 100, rightRightDown.getFractionOfRoot(), DOUBLE_ERROR_MARGIN); | ||
} | ||
|
||
@Test | ||
public void testFractionOfRoot_tree_complete() { | ||
Allocation root = Allocation.newRoot("ignored", 2); | ||
|
||
Allocation left = root.newChild("ignored", 3); | ||
Allocation leftLeft = left.newChild("ignored", 1); | ||
Allocation leftLeftDown = leftLeft.newChild("ignored", 100); | ||
Allocation leftMiddle = left.newChild("ignored", 100); | ||
Allocation leftRight = left.newChild("ignored", 100); | ||
|
||
Allocation right = root.newChild("ignored", 1); | ||
Allocation rightDown = right.newChild("ignored", 100); | ||
|
||
// Checks that the leaf allocations add up to a full 1.0. | ||
double total = | ||
leftLeftDown.getFractionOfRoot() | ||
+ leftMiddle.getFractionOfRoot() | ||
+ leftRight.getFractionOfRoot() | ||
+ rightDown.getFractionOfRoot(); | ||
Assert.assertEquals(1.0, total, DOUBLE_ERROR_MARGIN); | ||
} | ||
} |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So I see this example shows that the expect normal usage is to match the number of children and the parent allocation units. As I said in my previous comment, I am really curious how this can pan out or be enforced efficiently in practice. But I think we can move forward with this now, and I'd be able to see if the API will work out to minimize errors.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea, the user of the
Allocation
class will be responsible for sub-allocating the correct number of allocations. For example, for a task that knows it will launch 3 subtasks, it will first create its own allocation as 3 allocation units, and then launch the 3 subtasks, passing down its own allocation as the "parent" allocation, knowing that each of the subtasks will thus make a suballocation, resulting in 3 total suballocations.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, in that example, what if the writer of one of the subtasks (think of as separate methods) that get the "parent" allocation doesn't have anything to report progress or if the writer simply wants to mark completion before returning using the given parent allocation node? Should it emit an event with the parent node and 1 allocation done? But, in turn, what if the subtask called other methods and propagated down the parent node as-is? In that case, if the sub-subtasks did the same thing as what the subtask did (emitting the event with the parent node and 1 allocation done), the progress will go over the allocated amount.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, emitted an event with the parent node and 1 allocation done can work, or the subtask can create a suballocation and emit an event with all progress units in the suballocation. The allocation(s) are only to represent the subdivision of the overall progress and it will be up to the tasks to coordinate how to emit progress on those allocations (with the premise that parent tasks have power over child tasks) and up to the progress event listener to process the emitted progress in whichever way it chooses to do so (allow or disallow going over allocated amount, for eg.).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I think this premise is required for this work well. When the premise is applied recursively, I think it implies having power over all descendants. For example, if you are some subtask at some point, and you call several long-running external methods in sequence, you don't really know about whose internals. So, you just pass the allocation node you have, but you basically have no idea how those will make use up the node. They may or may not consume 1 allocation, or more. And those external methods may in the same situation recursively. So, basically, it looks to me that we should have the premise that parent tasks should have an idea about all descendants.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, in our use case in the builder steps, this premise will be met by a constraint that