Skip to content

Commit

Permalink
ADO/GH guide updates (Github Issues - 123,124) (Azure#125)
Browse files Browse the repository at this point in the history
* Update README.md

Added instructions on free/trial subscriptions for users to be more cautious and carefully read steps to avoid failures. GitHub issue 123 and 124.

* Adding screenshot being referenced in instructions

* Update deployguide_ado.md

ADO deployment guide update. GitHUb issue 123 and 124.

* Update deployguide_ado.md

ADO deployment guide updates. Github issue 123 and 124.

* Update deployguide_gha.md

ADO deployment guide updates. Github issue 123 and 124.
  • Loading branch information
strugdt committed Apr 17, 2024
1 parent 7d9e098 commit 36a625f
Show file tree
Hide file tree
Showing 4 changed files with 37 additions and 2 deletions.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ It accomplishes these goals with a template-based approach for end-to-end data s
## Prerequisites

1. An Azure subscription. If you don't have an Azure subscription, [create a free account](https://azure.microsoft.com/en-us/free/machine-learning/search/?OCID=AIDcmm5edswduu_SEM_822a7351b5b21e0f1ffe102c9ca9e99f:G:s&ef_id=822a7351b5b21e0f1ffe102c9ca9e99f:G:s&msclkid=822a7351b5b21e0f1ffe102c9ca9e99f) before you begin.
* **Important** - If you use either a Free/Trial, or similar learning purpose subscriptions like Visual Studio Premium with MSDN, some provisioning tasks might not run as expected due to limitations imposed on 'Usage + quotas' on your subscription. To help you succeed, we have provided specific instructions before provisioning throughout the guide, and you are highly advised to read those instructions carefully
2. For Azure DevOps-based deployments and projects:
* [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) with `azure-devops` extension.
* [Terraform extension for Azure DevOps](https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks) if you are using Terraform to spin up infrastructure
Expand Down
25 changes: 23 additions & 2 deletions documentation/deployguides/deployguide_ado.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ This document will guide you through using the MLOps V2 project generator to dep

**Prerequisites:**
- One or more Azure subscription(s) based on whether you are deploying Prod only or Prod and Dev environments
- **Important:** - As mentioned in the **Prerequisites** at the beginning [here](https://github.com/Azure/mlops-v2?tab=readme-ov-file#prerequisites), if you plan to use either a Free/Trial or similar learning purpose subscriptions, they might pose 'Usage + quotas' limitations in the default Azure region being used for deployment. Please read provided instructions carefully to succeessfully execute this deployment.
- An Azure DevOps organization
- Ability to create Azure service principals to access / create Azure resources from Azure DevOps
- If using Terraform to create and manage infrastructure from Azure DevOps, install the [Terraform extension for Azure DevOps](https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks).
Expand Down Expand Up @@ -387,6 +388,17 @@ To do this, go back to **Repos** and your ML project repo, in this example, `tax
<img src="./images/ado-new-mlrepo.png" alt="Complete ML repo" />
</p>
>**Important:**
>> Note that `config-infra-prod.yml` and `config-infra-dev.yml` files use default region as **eastus** to deploy resource group and Azure ML Workspace. If you are using Free/Trial or similar learning purpose subscriptions, you must do one of the below -
> 1. If you decide to use **eastus** region, ensure that your subscription(s) have a quota/limit of up to 20 vCPUs for **Standard DSv2 Family vCPUs**. Visit Subscription page in Azure Portal as show below to validate this.
![alt text](images/susbcriptionQuota.png)
> 2. If not, you should change it to a region where **Standard DSv2 Family vCPUs** has a quota/limit of up to 20 vCPUs.
> 3. You may also choose to change the region and compute type being used for deployment. To do this you have to change region in these two files, and additionally search for **STANDARD_DS3_V2** in below listed DevOps pipeline files and change this with a compute type that would work for your setup.
> * `mlops-templates/aml-cli-v2/mlops/devops-pipelines/deploy-model-training-pipeline.yml`
> * `mlops-project-template/classical/aml-cli-v2/mlops/devops-pipelines/deploy-batch-endpoint-pipeline.yml`
> * `/mlops-project-template/classical/aml-cli-v2/mlops/azureml/deploy/online/online-deployment.yml`
> 4. Note in the path above that you need to navigate to the right repository (e.g. **mlops-templates**), and the right ML interface (e.g. **aml-cli-v2**).
Making sure you are in the **main** branch, click on `config-infra-prod.yml` to open it.
Under the Global section, you will see properties for `namespace`, `postfix`, and `location`.
Expand Down Expand Up @@ -474,7 +486,16 @@ Next you can see the pipeline details.
<img src="./images/ado-training-pipeline-details.png" alt="Training pipeline details"/>
</p>
Click **Run** to execute the pipeline. This will take several minutes to finish. When complete, you can view the pipeline jobs and tasks by selecting **Pipelines** then **taxi-fare-regression** under **Recently run pipelines**. The pipeline run will be tagged `#deploy-model-training-pipeline`. Drill down into the pipeline run to see the **DeployTrainingPipeline** job. Click on the job to see pipeline run details.
Click **Run** to execute the pipeline. This will take several minutes to finish.
**Important:**
>>
>* It is noted that, with Free/Trial and other learning purpose susbscriptions pipeline execution might take up to 90 minutes.
>* In case, your model training pipeline execution in DevOps fails after running for a long period of time, but you have validated that a model is registred in Model Registry, it confirms that your model training is Azure ML was successful.
>* This issue with longer running pipelines in Azure DevOps has been reported and is being tracked by product teams.
When complete, you can view the pipeline jobs and tasks by selecting **Pipelines** then **taxi-fare-regression** under **Recently run pipelines**. The pipeline run will be tagged `#deploy-model-training-pipeline`. Drill down into the pipeline run to see the **DeployTrainingPipeline** job. Click on the job to see pipeline run details.
<p align="center">
<img src="./images/ado-training-pipeline-run.png" alt="Training pipeline run"/>
Expand Down Expand Up @@ -596,4 +617,4 @@ This guide illustrated using Azure DevOps pipelines and Azure Machine Learning p
* Integrated feature store
* Deployment of secured Azure ML environments in a vnet
For questions, problems, or feature requests, please [submit an issue](https://github.com/Azure/mlops-v2/issues) or reach out to the development team at Microsoft.
For questions, problems, or feature requests, please [submit an issue](https://github.com/Azure/mlops-v2/issues) or reach out to the development team at Microsoft.
13 changes: 13 additions & 0 deletions documentation/deployguides/deployguide_gha.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
- [GitHub client](https://cli.github.com/)
- [Azure CLI ](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli)
- The [Terraform extension for Azure DevOps](https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks) if you are using Azure DevOps + Terraform to spin up infrastructure
- One or more Azure subscription(s) based on whether you are deploying Prod only or Prod and Dev environments
- **Important** - As mentioned in the **Prerequisites** at the beginning [here](https://github.com/Azure/mlops-v2?tab=readme-ov-file#prerequisites), if you plan to use either a Free/Trial or similar learning purpose subscriptions, they might pose 'Usage + quotas' limitations in the default Azure region being used for deployment. Please read provided instructions carefully to succeessfully execute this deployment.
- Azure service principals to access / create Azure resources from Azure DevOps or Github Actions (or the ability to create them)
- Git bash, [WSL](https://learn.microsoft.com/en-us/windows/wsl/install) or another shell script runner on your local machine
- When using WSL,
Expand Down Expand Up @@ -170,6 +172,17 @@

It is recommended to first create a dev branch from main and deploy this environment first.

>**Important:**
>> Note that `config-infra-prod.yml` and `config-infra-dev.yml` files use default region as **eastus** to deploy resource group and Azure ML Workspace. If you are using Free/Trial or similar learning purpose subscriptions, you must do one of the below -
> 1. If you decide to use **eastus** region, ensure that your subscription(s) have a quota/limit of up to 20 vCPUs for **Standard DSv2 Family vCPUs**. Visit Subscription page in Azure Portal as show below to validate this.
![alt text](images/susbcriptionQuota.png)
> 2. If not, you should change it to a region where **Standard DSv2 Family vCPUs** has a quota/limit of up to 20 vCPUs.
> 3. You may also choose to change the region and compute type being used for deployment. To do this you have to change region in these two files, and additionally search for **STANDARD_DS3_V2** in below listed DevOps pipeline files and change this with a compute type that would work for your setup.
> * `mlops-templates/aml-cli-v2/mlops/devops-pipelines/deploy-model-training-pipeline.yml`
> * `mlops-project-template/classical/aml-cli-v2/mlops/devops-pipelines/deploy-batch-endpoint-pipeline.yml`
> * `/mlops-project-template/classical/aml-cli-v2/mlops/azureml/deploy/online/online-deployment.yml`
> 4. Note in the path above that you need to navigate to the right repository (e.g. **mlops-templates**), and the right ML interface (e.g. **aml-cli-v2**).

Edit each file to configure a namespace, postfix string, Azure location, and environment for deploying your Dev and Prod Azure ML environments. Default values and settings in the files are show below:

> ```bash
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 36a625f

Please sign in to comment.