Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds progress Allocation to represent Decentralized Allocation Tree node. #1298

Merged
merged 3 commits into from
Dec 5, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
/*
* Copyright 2018 Google LLC.
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not
* use this file except in compliance with the License. You may obtain a copy of
* the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
* WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
* License for the specific language governing permissions and limitations under
* the License.
*/

package com.google.cloud.tools.jib.event.progress;

import java.util.Optional;
import javax.annotation.Nullable;

/**
* Represents a Decentralized Allocation Tree (DAT) node.
*
* <p>A DAT node is immutable and pointers only go in the direction from child to parent. Each node
* has a set number of allocated units, the total of which represents a single allocation unit of
* its parent. Each node is therefore a sub-allocation of its parent node. This allows the DAT to
* sub-allocate progress in a decentralized, asynchronous manner.
*
* <p>For example, thread 1 creates node A as the root node with 2 allocation units. A subtask is
* launched on thread 1 and creates node B with 3 allocation units as a child of node A. Thread 1
* then also launches a subtask on thread 2 that creates node C with 5 allocation units. Once the
* first subtask finishes and reports its progress, that completion would entail completion of 3
* allocation units of node B and 1 allocation unit of node A. The second subtask finishes and
* reports its progress as well, indicating completion of 5 units of node C and thus 1 unit of node
* A. Allocation A is then deemed complete as well in terms of overall progress.
*
* <p>Note that it is up to the user of the class to ensure that the number of sub-allocations does
* not exceed the number of allocation units.
*/
class Allocation {

/**
* Creates a new root {@link Allocation}.
*
* @param description thuser-facing description of what the allocation represents
* @param allocationUnits number of allocation units
* @return a new {@link Allocation}
*/
static Allocation newRoot(String description, long allocationUnits) {
return new Allocation(description, allocationUnits, null);
}

/** The parent {@link Allocation}, or {@code null} to indicate a root node. */
@Nullable private final Allocation parent;

/** User-facing description of what the allocation represents. */
private final String description;

/** The number of allocation units this node holds. */
private final long allocationUnits;

/** How much of the root allocation (1.0) this allocation accounts for. */
private final double fractionOfRoot;

private Allocation(String description, long allocationUnits, @Nullable Allocation parent) {
this.description = description;
this.allocationUnits = allocationUnits;
this.parent = parent;

this.fractionOfRoot = parent == null ? 1.0 : parent.fractionOfRoot / parent.allocationUnits;
}

/**
* Creates a new child {@link Allocation} (sub-allocation).
*
* @param description user-facing description of what the sub-allocation represents
* @param allocationUnits number of allocation units the child holds
* @return a new {@link Allocation}
*/
Allocation newChild(String description, long allocationUnits) {
return new Allocation(description, allocationUnits, this);
}

Optional<Allocation> getParent() {
return Optional.ofNullable(parent);
}

String getDescription() {
return description;
}

long getAllocationUnits() {
return allocationUnits;
}

double getFractionOfRoot() {
return fractionOfRoot;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
/*
* Copyright 2018 Google LLC.
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not
* use this file except in compliance with the License. You may obtain a copy of
* the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
* WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
* License for the specific language governing permissions and limitations under
* the License.
*/

package com.google.cloud.tools.jib.event.progress;

import org.junit.Assert;
import org.junit.Test;

/** Tests for {@link Allocation}. */
public class AllocationTest {

/** Error margin for checking equality of two doubles. */
private static final double DOUBLE_ERROR_MARGIN = 1e-10;

@Test
public void testSmoke_linear() {
Allocation root = Allocation.newRoot("root", 1);
Allocation node1 = root.newChild("node1", 2);
Allocation node2 = node1.newChild("node2", 3);

Assert.assertEquals("node2", node2.getDescription());
Assert.assertEquals(3, node2.getAllocationUnits());
Assert.assertEquals(1.0 / 2, node2.getFractionOfRoot(), DOUBLE_ERROR_MARGIN);
Assert.assertTrue(node2.getParent().isPresent());
Assert.assertEquals(node1, node2.getParent().get());

Assert.assertEquals("node1", node1.getDescription());
Assert.assertEquals(2, node1.getAllocationUnits());
Assert.assertTrue(node1.getParent().isPresent());
Assert.assertEquals(root, node1.getParent().get());
Assert.assertEquals(1.0, node1.getFractionOfRoot(), DOUBLE_ERROR_MARGIN);

Assert.assertEquals("root", root.getDescription());
Assert.assertEquals(1, root.getAllocationUnits());
Assert.assertFalse(root.getParent().isPresent());
Assert.assertEquals(1.0, root.getFractionOfRoot(), DOUBLE_ERROR_MARGIN);
}

@Test
public void testFractionOfRoot_tree_partial() {
Allocation root = Allocation.newRoot("ignored", 10);
Allocation left = root.newChild("ignored", 2);
Allocation right = root.newChild("ignored", 4);
Allocation leftDown = left.newChild("ignored", 20);
Allocation rightLeft = right.newChild("ignored", 20);
Allocation rightRight = right.newChild("ignored", 100);
Allocation rightRightDown = rightRight.newChild("ignored", 200);

Assert.assertEquals(1.0, root.getFractionOfRoot(), DOUBLE_ERROR_MARGIN);
Assert.assertEquals(1.0 / 10, left.getFractionOfRoot(), DOUBLE_ERROR_MARGIN);
Assert.assertEquals(1.0 / 10, right.getFractionOfRoot(), DOUBLE_ERROR_MARGIN);
Assert.assertEquals(1.0 / 10 / 2, leftDown.getFractionOfRoot(), DOUBLE_ERROR_MARGIN);
Assert.assertEquals(1.0 / 10 / 4, rightLeft.getFractionOfRoot(), DOUBLE_ERROR_MARGIN);
Assert.assertEquals(1.0 / 10 / 4, rightRight.getFractionOfRoot(), DOUBLE_ERROR_MARGIN);
Assert.assertEquals(
1.0 / 10 / 4 / 100, rightRightDown.getFractionOfRoot(), DOUBLE_ERROR_MARGIN);
}

@Test
public void testFractionOfRoot_tree_complete() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I see this example shows that the expect normal usage is to match the number of children and the parent allocation units. As I said in my previous comment, I am really curious how this can pan out or be enforced efficiently in practice. But I think we can move forward with this now, and I'd be able to see if the API will work out to minimize errors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, the user of the Allocation class will be responsible for sub-allocating the correct number of allocations. For example, for a task that knows it will launch 3 subtasks, it will first create its own allocation as 3 allocation units, and then launch the 3 subtasks, passing down its own allocation as the "parent" allocation, knowing that each of the subtasks will thus make a suballocation, resulting in 3 total suballocations.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, in that example, what if the writer of one of the subtasks (think of as separate methods) that get the "parent" allocation doesn't have anything to report progress or if the writer simply wants to mark completion before returning using the given parent allocation node? Should it emit an event with the parent node and 1 allocation done? But, in turn, what if the subtask called other methods and propagated down the parent node as-is? In that case, if the sub-subtasks did the same thing as what the subtask did (emitting the event with the parent node and 1 allocation done), the progress will go over the allocated amount.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, emitted an event with the parent node and 1 allocation done can work, or the subtask can create a suballocation and emit an event with all progress units in the suballocation. The allocation(s) are only to represent the subdivision of the overall progress and it will be up to the tasks to coordinate how to emit progress on those allocations (with the premise that parent tasks have power over child tasks) and up to the progress event listener to process the emitted progress in whichever way it chooses to do so (allow or disallow going over allocated amount, for eg.).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with the premise that parent tasks have power over child tasks)

Yeah, I think this premise is required for this work well. When the premise is applied recursively, I think it implies having power over all descendants. For example, if you are some subtask at some point, and you call several long-running external methods in sequence, you don't really know about whose internals. So, you just pass the allocation node you have, but you basically have no idea how those will make use up the node. They may or may not consume 1 allocation, or more. And those external methods may in the same situation recursively. So, basically, it looks to me that we should have the premise that parent tasks should have an idea about all descendants.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, in our use case in the builder steps, this premise will be met by a constraint that

  1. A task may only create at most one suballocation of the parent allocation passed to it, and
  2. A task cannot pass the parent allocation on to subtasks

Allocation root = Allocation.newRoot("ignored", 2);

Allocation left = root.newChild("ignored", 3);
Allocation leftLeft = left.newChild("ignored", 1);
Allocation leftLeftDown = leftLeft.newChild("ignored", 100);
Allocation leftMiddle = left.newChild("ignored", 100);
Allocation leftRight = left.newChild("ignored", 100);

Allocation right = root.newChild("ignored", 1);
Allocation rightDown = right.newChild("ignored", 100);

// Checks that the leaf allocations add up to a full 1.0.
double total =
leftLeftDown.getFractionOfRoot()
+ leftMiddle.getFractionOfRoot()
+ leftRight.getFractionOfRoot()
+ rightDown.getFractionOfRoot();
Assert.assertEquals(1.0, total, DOUBLE_ERROR_MARGIN);
}
}