v0.2.0 RC1

Dockerfile.evaluator

@@ -3,6 +3,7 @@ FROM golang:1.10.3 as builder
 WORKDIR /go/src/github.com/GoogleCloudPlatform/open-match
 COPY examples/evaluators/golang/simple examples/evaluators/golang/simple 
 COPY config config
+COPY internal internal
 WORKDIR /go/src/github.com/GoogleCloudPlatform/open-match/examples/evaluators/golang/simple
 RUN go get -d -v
 RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo .

Dockerfile.mmf_python3_simple → Dockerfile.mmf_py3

@@ -1,9 +1,9 @@
 # Golang application builder steps
 FROM python:3.5.3 as builder
 WORKDIR /usr/src/open-match
-COPY examples/functions/python3/simple examples/functions/python3/simple
+COPY examples/functions/python3/mmlogic-simple examples/functions/python3/mmlogic-simple
 COPY config config
-WORKDIR /usr/src/open-match/examples/functions/python3/simple
+WORKDIR /usr/src/open-match/examples/functions/python3/mmlogic-simple
 RUN pip install --no-cache-dir -r requirements.txt
 
 CMD ["python", "./harness.py"]

README.md

@@ -12,6 +12,10 @@ This project attempts to solve the networking and plumbing problems, so game dev
 ## Disclaimer
 This software is currently alpha, and subject to change. **It is not yet ready to be used in production.**
 
+## Version
+The current stable version in master is 0.1.0.  
+The 0.2.0 RC1 is now available.
+
 # Core Concepts
 
 [Watch the introduction of Open Match at Unite Berlin 2018 on YouTube](https://youtu.be/qasAmy_ko2o)
@@ -24,7 +28,7 @@ Open Match is designed to support massively concurrent matchmaking, and to be sc
 * **Component** — One of the discrete processes in an Open Match deployment. Open Match is composed of multiple scalable microservices called 'components'.
 * **Roster** — A list of all the players in a match.
 * **Profile** — The json blob containing all the parameters used to select which players go into a roster.
-* **Match Object** — A json blob to contain the results of the matchmaking function. Sent with an empty roster section to the backend API from your game backend and then returned with the matchmaking results filled in.
+* **Match Object** — A protobuffer message format that contains the Profile and the results of the matchmaking function. Sent to the backend API from yoru game backend with an empty roster  and then returned from your MMF with the matchmaking results filled in.
 * **MMFOrc** — Matchmaker function orchestrator. This Open Match core component is in charge of kicking off custom matchmaking functions (MMFs) and evaluator processes.
 * **State Storage** — The storage software used by Open Match to hold all the matchmaking state. Open Match ships with [Redis](https://redis.io/) as the default state storage.
 * **Assignment** — Refers to assigning a player or group of players to a dedicated game server instance. Open Match offers a path to send dedicated game server connection details from your backend to your game clients after a match has been made.
@@ -41,11 +45,12 @@ Open Match is a set of processes designed to run on Kubernetes. It contains thes
 1. Frontend API
 1. Backend API
 1. Matchmaker Function Orchestrator (MMFOrc)
+1. Matchmaking Logic (MMLogic) API 
 
 It also explicitly depends on these two **customizable** components.
 
 1. Matchmaking "Function" (MMF)
-1. Evaluator
+1. Evaluator (may be deprecated in future versions)
 
 While **core** components are fully open source and *can* be modified, they are designed to support the majority of matchmaking scenarios *without need to change the source code*. The Open Match repository ships with simple **customizable** example MMF and Evaluator processes, but it is expected that most users will want full control over the logic in these, so they have been designed to be as easy to modify or replace as possible.
 
@@ -69,12 +74,27 @@ The Backend API is a server application that implements the [gRPC](https://grpc.
 
 Your game backend is expected to maintain a connection, waiting for 'filled' match objects containing a roster of players. The Backend API also provides a return path for your game backend to return dedicated game server connection details (an 'assignment') to the game client, and to delete these 'assignments'.
 
+
 ### Matchmaking Function Orchestrator (MMFOrc)
 
 The MMFOrc kicks off your custom matchmaking function (MMF) for every profile submitted to the Backend API. It also runs the Evaluator to resolve conflicts in case more than one of your profiles matched the same players.
 
 The MMFOrc exists to orchestrate/schedule your **custom components**, running them as often as required to meet the demands of your game. MMFOrc runs in an endless loop, submitting MMFs and Evaluator jobs to Kubernetes.
 
+### Matchmaking Logic (MMLogic) API
+
+The MMLogic API provides a series of gRPC functions that act as a Matchmaking Function SDK.  Much of the basic, boilerplate code for an MMF is the same regardless of what players you want to match together.  The MMLogic API offers a gRPC interface for many common MMF tasks, such as:
+
+1. Reading a profile from state storage.
+1. Running filters on players in state strorage.
+1. Removing chosen players from consideration by other MMFs (by adding them to an ignore list).
+1. Writing the matchmaking results to state storage.
+1. (Optional, NYI) Exporting MMF stats for metrics collection.
+
+More details about the available gRPC calls can be found in the [API Specification](api/protobuf-spec/messages.proto).
+
+**Note**: using the MMLogic API is **optional**.  It tries to simplify the development of MMFs, but if you want to take care of these tasks on your own, you can make few or no calls to the MMLogic API as long as your MMF still completes all the required tasks.  Read the [Matchmaking Functions section](https://github.com/GoogleCloudPlatform/open-match#matchmaking-functions-mmfs) for more details of what work an MMF must do.
+
 ### Evaluator
 
 The Evaluator resolves conflicts when multiple matches want to include the same player(s).
@@ -187,28 +207,37 @@ Open Match is in active development - we would love your help in shaping its fut
 
 Apache 2.0
 
-# Missing functionality
-
-* Player/Group records generated when a client enters the matchmaking pool need to be removed after a certain amount of time with no activity. When using Redis, this will be implemented as a expiration on the player record.
-* Instrumentation of MMFs is in the planning stages.  Since MMFs are by design meant to be completely customizable (to the point of allowing any process that can be packaged in a Docker container), metrics/stats will need to have an expected format and formalized outgoing pathway.  Currently the thought is that it might be that the metrics should be written to a particular key in statestorage in a format compatible with opencensus, and will be collected, aggreggated, and exported to Prometheus using another process.
-* The Kubernetes service account used by the MMFOrc should be updated to have min required permissions.
-* Autoscaling isn't turned on for the Frontend or Backend API Kubernetes deployments by default.
-* Match profiles should be able to define multiple MMF container images to run, but this is not currently supported. This enables A/B testing and several other scenarios.
-* Out-of-the-box, the Redis deployment should be a HA configuration using [Redis Sentinel](https://redis.io/topics/sentinel).
-* Redis watch should be unified to watch a hash and stream updates.  The code for this is written and validated but not committed yet. We don't want to support two redis watcher code paths, so the backend watch of the match object should be switched to unify the way the frontend and backend watch keys.  Unfortunately this change touches the whole chain of components that touch backend match objects (mmf, evaluator, backendapi) and so needs additional work and testing before it is integrated.
 
 # Planned improvements
 
-* “Writing your first matchmaker” getting started guide will be included in an upcoming version.
-* Documentation for using the example customizable components and the `backendstub` and `frontendstub` applications to do an end-to-end (e2e) test will be written. This all works now, but needs to be written up.
-* A [Helm](https://helm.sh/) chart to stand up Open Match will be provided in an upcoming version.
-* We plan to host 'official' docker images for all release versions of the core components in publicly available docker registries soon.
-* CI/CD for this repo and the associated status tags are planned.
-* Documentation on release process and release calendar.
-* [OpenCensus tracing](https://opencensus.io/core-concepts/tracing/) will be implemented in an upcoming version.
-* Read logrus logging configuration from matchmaker_config.json.
-* Golang unit tests will be shipped in an upcoming version.
-* A full load-testing and e2e testing suite will be included in an upcoming version.
-* All state storage operations should be isolated from core components into the `statestorage/` modules.  This is necessary precursor work to enabling Open Match state storage to use software other than Redis.
-* The MMFOrc component name will be updated in a future version to something easier to understand.  Suggestions welcome!
-* The MMFOrc component currently requires a default service account with permission to kick of k8s jobs, but the revision today makes the service account have full permissions.  This needs to be reworked to have min required RBAC permissions before it is used in production, but is fine for closed testing and development.
+## Documentation 
+- [ ] “Writing your first matchmaker” getting started guide will be included in an upcoming version.
+- [ ] Documentation for using the example customizable components and the `backendstub` and `frontendstub` applications to do an end-to-end (e2e) test will be written. This all works now, but needs to be written up.
+- [ ] Documentation on release process and release calendar.
+
+## State storage
+- [ ] All state storage operations should be isolated from core components into the `statestorage/` modules.  This is necessary precursor work to enabling Open Match state storage to use software other than Redis.
+- [ ] The Redis deployment should have an example HA configuration using HAProxy 
+- [ ] Redis watch should be unified to watch a hash and stream updates.  The code for this is written and validated but not committed yet. We don't want to support two redis watcher code paths, so the backend watch of the match object should be switched to unify the way the frontend and backend watch keys.  The backend part of this is in but the frontend part is in another branch and will be committed later. 
+
+## Instrumentation / Metrics / Analytics
+- [ ] Instrumentation of MMFs is in the planning stages.  Since MMFs are by design meant to be completely customizable (to the point of allowing any process that can be packaged in a Docker container), metrics/stats will need to have an expected format and formalized outgoing pathway.  Currently the thought is that it might be that the metrics should be written to a particular key in statestorage in a format compatible with opencensus, and will be collected, aggreggated, and exported to Prometheus using another process.
+- [ ] [OpenCensus tracing](https://opencensus.io/core-concepts/tracing/) will be implemented in an upcoming version.
+- [ ] Read logrus logging configuration from matchmaker_config.json.
+
+## Security
+- [ ] The Kubernetes service account used by the MMFOrc should be updated to have min required permissions.
+
+## Kubernetes
+- [ ] Autoscaling isn't turned on for the Frontend or Backend API Kubernetes deployments by default.
+- [ ] A [Helm](https://helm.sh/) chart to stand up Open Match will be provided in an upcoming version. For now just use the [installation YAMLs](./install/yaml).
+
+- [ ] Player/Group records generated when a client enters the matchmaking pool need to be removed after a certain amount of time with no activity. When using Redis, this will be implemented as a expiration on the player record.
+## CI / CD / Build
+- [ ] We plan to host 'official' docker images for all release versions of the core components in publicly available docker registries soon.
+- [ ] CI/CD for this repo and the associated status tags are planned.
+- [ ] Golang unit tests will be shipped in an upcoming version.
+- [ ] A full load-testing and e2e testing suite will be included in an upcoming version.
+
+## Will not Implement 
+- [X] Match profiles should be able to define multiple MMF container images to run, but this is not currently supported. This enables A/B testing and several other scenarios.

RELEASE.md

@@ -0,0 +1,30 @@
+# Release history
+
+##v0.2.0 RC1 (alpha)
+  This is a pretty large update.  Custom MMFs or evaluators from 0.1.0 may need some tweaking to work with this version. Some Backend API function arguments have changed. Please join the [Slack channel](https://open-match.slack.com/) if you need help ([Signup link](https://join.slack.com/t/open-match/shared_invite/enQtNDM1NjcxNTY4MTgzLWQzMzE1MGY5YmYyYWY3ZjE2MjNjZTdmYmQ1ZTQzMmNiNGViYmQyN2M4ZmVkMDY2YzZlOTUwMTYwMzI1Y2I2MjU))!
+
+  v0.2.0 focused on adding additional functionality to Backend API calls and on **reducing the amount of boilerplate code required to make a custom Matchmaking Function**.  For this, a new internal API for use by MMFs called the [Matchmaking Logic API (MMLogic API)](README.md#matchmaking-logic-mmlogic-api) has been added.  Many of the core components and examples had to be updated to use the new Backend API arguments and the modules to support them, so we recommend you rebuild and redeploy all the components to use v0.2.0. 
+
+### Release notes
+ - MMLogic API is now available.  Deploy it to kubernetes using the [appropriate json file]() and check out the [gRPC API specification](api/protobuf-spec/mmlogic.proto) to see how to use it.  To write a client against this API, you'll need to compile the protobuf files to your language of choice. There is an associated cloudbuild.yaml file and Dockerfile for it in the root directory.
+  - When using the MMLogic API to filter players into pools, it will attempt to report back the number of players that matched the filters and how long the filters took to query state storage.
+  - An [example MMF](examples/functions/python3/mmlogic-simple/harness.py) using it has been written in Python3. There is an associated cloudbuild.yaml file and Dockerfile for it in the root directory. By default the [example backend client](examples/backendclient/main.go) is now configured to use this MMF, so make sure you have it avaiable before you try to run the latest backend client.
+  - The API specs have been split into separate files per API and the protobuf messages are in a separate file.  Things were renamed slightly as a result, and you will need to update your API clients. The Frontend API hasn't had it's messages moved to the shared messages file yet, but this will happen in an upcoming version.
+ - The message model for using the Backend API has changed slightly - for calls that make MatchObjects, the expectation is that you will provide a MatchObject with a few fields populated, and it will then be shuttled along through state storage to your MMF and back out again, with various processes 'filling in the blanks' of your MatchObject, which is then returned to your code calling the Backend API.  Read the[gRPC API specification](api/protobuf-spec/backend.proto) for more information.
+  - As part of this, compiled protobuf golang modules now live in the [internal/pb](internal/pb) directory.  There's a handy [bash script](api/protoc-go.sh) for compiling them in your local golang environment if you need it.  
+  - As part of this Backend API message shift and the advent of the MMLogic API, 'player pools' and 'rosters' are now first-class data structures in MatchObjects for those who wish to use them. You can ignore them if you like, but if you want to use some of the MMLogic API calls to automate tasks for you - things like filtering a pool of players according attributes or adding all the players in your rosters to the ignorelist so other MMFs don't try to grab them - you'll need to put your data into the [protobuf messages](api/protobuf-spec/messages.proto) so Open Match knows how to read them.  The sample backend client [test profile JSON](examples/backendclient/profiles/testprofile.json)has been updated to use this format if you want to see an example.
+- Rosters were formerly space-delimited lists of player IDs.  They are now first-class repeated protobuf message fields in the [Roster message format](api/protobuf-spec/messages.proto). That means that in most languages, you can access the roster as a list of players using your native language data structures (more info can be found in the [guide for using protocol buffers in your langauge of choice](https://developers.google.com/protocol-buffers/docs/reference/overview)).  If you don't care about the new fields or the new functionality, you can just leave all the other fields but the player ID unset.
+ - Open Match is transitioning to using [protocol buffer messages](https://developers.google.com/protocol-buffers/) as its internal data format. There is now a Redis state storage [golang module](internal/statestorage/redis/redispb/) for marshaling and unmarshaling MatchObject messages to and from Redis. It isn't very clean code right now but will get worked on for the next couple releases.
+ - Ignorelists now exist, and have a Redis state storage [golang module](internal/statestorage/redis/ignorelist/) for CRUD access.  Currently three ignorelists are defined in the [config file](config/matchmaker_config.json) with their respective parameters. These are implemented as [Sorted Sets in Redis](https://redis.io/commands#sorted_set).
+ - For those who only want to stand up Open Match and aren't interested in individually tweaking the required kubernetes resources, there are now [three YAML files](install/yaml) that can be used to install Redis, install Open Match, and (optionally) install Prometheus. You'll still need the `sed` [instructions from the Developer Guide](docs/development.md#running-open-match-in-a-development-environment) to substitute in the name of your Docker container registry.
+
+
+### Roadmap 
+ - It has become clear from talking to multiple users that the software they write to talk to the Backend API needs a name.  'Backend API Client' is technically correct, but given how many APIs are in Open Match and the overwhelming use of 'Client' to refer to a Game Client in the industry, we're currently calling this a 'Director', as its primary purpose is to 'direct' which profiles are sent to the backend, and 'direct' the resulting MatchObjects to game servers.  Further discussion / suggestions are welcome.
+ - We'll be entering the design stage on longer-running MMFs before the end of the year. We'll get a proposal together and on the github repo as a request for comments, so please keep your eye out for that.
+ - Match profiles providing multiple MMFs to run isn't planned anymore.  Just send multiple copies of the profile with different MMFs specified via the backendapi. 
+ - Evaluators will be examined for removal in an upcoming version. There's a good change this logic should live in the Directory
+ - Redis Sentinel will likely not be supported.  Instead, replicated instances and HAProxy may be the HA solution of choice.
+
+## v0.1.0 (alpha)
+ Initial release.

api/protoc_go.sh

@@ -1,3 +1,20 @@
+#!/bin/bash
+# Script to compile golang versions of the OM proto files
+#
+# Copyright 2018 Google LLC
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# 
+#     http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 cd $GOPATH/src
 protoc \
 ${GOPATH}/src/github.com/GoogleCloudPlatform/open-match/api/protobuf-spec/backend.proto \