{"id":96,"date":"2022-11-10T10:32:17","date_gmt":"2022-11-10T10:32:17","guid":{"rendered":"https:\/\/demo.mekshq.com\/awsm-wp\/?p=96"},"modified":"2024-08-01T08:05:37","modified_gmt":"2024-08-01T08:05:37","slug":"ci-cd-in-azure-databricks-using-azure-devops-part-1","status":"publish","type":"post","link":"https:\/\/dipankar.me\/yogesh\/ci-cd-in-azure-databricks-using-azure-devops-part-1\/","title":{"rendered":"CI \/ CD in Azure Databricks using Azure DevOps – Part 1"},"content":{"rendered":"\n
\n \n \n \n

\n CICD in Azure Databricks using Azure DevOps is a big topic so I have broken this article into 2 parts as below <\/span>\n <\/h2>\n\n \n\n\n\n\n\n\n \n \n

\n Part 1:<\/strong> <\/span>Version Control Databricks Notebooks<\/strong>, here we will cover the steps to move workspace notebook to repo, and push it to GitHub\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

\n Part 2:<\/strong> <\/span>Azure DevOpsBuild to build CI\/CD pipeline to deploy the notebooks <\/span><\/strong>, in this part 2 article we will cover the end to end deployment pipeline <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

\n Before I deep dive into the topic lets look at what is Databricks, it is a web based managed Apache Spark Service that is hosted on all popular clouds like Azure, AWS and recently GCP <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

\n Databricks is a web based managed Apache Spark Service that is hosted on all popular clouds like Azure, AWS and recently GCP\n <\/blockquote>\n\n \n\n\n\n\n\n\n \n \n

\n <\/span>Databricks was founded by the <\/span>original creators of Apache spark<\/strong> <\/span>it is developed as a web-based platform for implementing Apache spark, it helps to implement complex data analytics code that needs cluster computing <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

\n It provides automated cluster management and IPython styled notebooks so the best thing about databricks is that the cluster management is completely automated making it a great Platform As A Service, we absolutely don’t need to manage any aspect of the cluster manually, all of its automated and top of that we get IPython style notebooks. <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

\n We get all these things pre-installed with databricks so we have a cluster ( multi-node \/ single node cluster ), On this cluster we can launch notebooks written in ( SQL, pySpark and Spark Scala ). The code in notebooks will utilize the full functionality of the cluster, the best thing about the databricks is Automated Cluster Management and Coding notebooks in multiple languages like ( SQL, pySpark, Spark Scala, Java ) <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

\n Clusters: – Clusters are nothing but a set of virtual machines that are used to do work <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

\n Databricks Provides\n <\/h3>\n\n \n\n\n\n\n\n\n \n \n

\n

  1. Flexibility ( pick any language of your choice : SQL, Scala, Python, Java ) <\/span><\/li>
  2. Workflow Optimisation ( intuitive orchestration service )<\/li>
  3. Collaborative environment for data engineering, data scientist and data analyst <\/span><\/li><\/ol>\n\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

    \n By now it is clear the Databricks is managed spark as PaaS service for cluster computing, so lets start with the main topic of this article, i.e <\/span>Version control of the notebooks<\/strong>. The databricks part of the workspace as just available in the platform for development, testing, execution etc, but these notebooks under workspace are not version controlled, so if someone accidentally deletes the notebook its gone, Also the changes are not tracked if its just in the workspace, so its very important for code management and release management to move the codebase to a safe location which is version controlled <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

    \n Detail steps of Moving notebooks to Repos\n <\/h3>\n\n \n\n\n\n\n\n\n \n \n

    \n Repos<\/strong> <\/span>feature is a recently added feature, it is integration with a remote code repository, this will make it easier to version control the development and follow the best practices, Databricks supports integration with GitHub, Azure DevOps Repos <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

    \n Step 1: -Add Repo to the Repos <\/span><\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

    \n Select new and click on repo to create new repo , after clicking a new pop-up will appear of add repo in that we have to provide Git\/AzureRepo Integration URL based on the GIT provider <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Landing Page of Databricks with new feature “Repos”\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n\n \n
    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Add Repo\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n In case of Azure DevOPs to get that URL go to AzureDevops Select the folder which we want to add in to Repository, select clone and copy the URL and paste it in the <\/span>AzureDatabricks <<Add Repo<\/strong>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n copy the URL\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n Step 2 : Move the notebooks to the recently added repo as below<\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

    \n Go to the workspace containing the notebook you wish to move. Click the downward arrow next to the notebook you wish to move. Select ‘Move<\/strong>‘\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n <\/span>Here select the notebooks, folders, etc you want to move from the workspace to the repo\n\nSelect Workspace\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n\n \n
    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Move the Workspace <\/span>\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n Step 3: -In the directory, select ‘Repos’ then select the repo you are interested in. Click Select.<\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Moving location for the Workspace notebook\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n Step 4: – Once the notebook has been moved, Click the downward arrow beside the repo name. Click “Git…”<\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

    \n Notebooks can be committed into a Git repository either by linking a Git repository to the notebook in the Databricks Workspace or by manually exporting the notebook as a <\/span>Source File<\/em>.\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Git Integration <\/span>\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n Step 5: – Review changes made, and add a commit message (required). Click “Commit & Push”<\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Commit and Push\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n Step 6: – The notebook should now be visible in GitHub.<\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Notebooks committed to Rep\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n Before the advent of databricks workflows notebooks were mostly orchestrated or scheduled using <\/span>Azure Data Factory<\/strong>, Or if we want to execute\/orchestrate through ADF follow these steps <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

    \n Step 7: -Now go back to our Azure portal and open Azure Data Factory<\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Azure portal\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n Step 8: -Select Launch Studio<\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Launch studio <\/span>\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n\n \n
    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n <\/span>Open the required Azure Data Factory <\/span>\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n Step 9: – Select Linked Service pointing to that databricks notebook <\/span><\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Open the Linked Service\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n Step 10: – Add the Access Token and link the service<\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Copy Paste the Access Token\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n Step 11: -Generate the access token<\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

    \n <\/span><\/strong>Go to Azure Databricks and select User Settings in that select Access Token and Generate new token and copy the Access Token.\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n User Settings\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n\n \n
    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Generate the access token\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n\n \n
    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n copy the access token <\/span>\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n Step 12: – Now paste the copied access token and select Apply, and select Test Connection.<\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Paste the access token <\/span>\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n Step 13: – Test connection Successful.<\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Test Connection Successful\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n Step 14: – Select Factory Resources in that select Pipelines and then select New Pipeline.<\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Pipeline in the Factory Resources\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n Step 15: – In that select Databricks and select Notebook.<\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Select Databricks and Notebooks\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n S<\/strong>tep 16: – Select the databricks which we linked before.<\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Browse notebooks\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n Step 17: – In the Settings select Browse and Select the notebook which we want to test.<\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Browse the notebooks\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n\n \n
    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n <\/figure>\n <\/div>\n \n\n\n\n\n\n\n \n
    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n <\/figure>\n <\/div>\n \n\n\n\n\n\n\n \n
    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n <\/figure>\n <\/div>\n \n\n\n\n\n\n\n \n
    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Select the notebook <\/span>\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n Step 18: – Now after selecting the notebook Test the connection and we can see that the connection is successful.<\/strong><\/span>\n <\/p>\n\n \n\n\n\n\n\n\n\n \n

    \n
    \n \n
    \n \n
    \n\n \"No\n <\/div>\n \n <\/div>\n \n\n
    \n Test Connection Successful\n <\/figcaption>\n <\/figure>\n <\/div>\n \n\n\n\n\n\n \n \n

    \n At this point, the Databricks orchestration through ADF is done. Now the code is in Repo and thoroughly tested manually and through ADF orchestration\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

    \n At the end I would say never leave the notebooks in workspace, move them to repo and version control <\/span>\n <\/blockquote>\n\n \n\n\n\n\n\n\n \n \n

    \n \n
    \n <\/p>\n\n \n\n\n\n\n\n\n \n \n

    \n Let me know your thoughts, this article or a detail tutorial is designed to help you move codebase in repo and test it manually and through orchestration in next article we are going to explore more on end to end CICD deployment process using Azure DevOps <\/span>\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

    \n Cheers\n <\/p>\n\n \n\n\n\n\n\n\n \n \n

    \n Yogesh\n <\/p>\n\n \n\n\n\n\n\n\n \n <\/div>\n","protected":false},"excerpt":{"rendered":"

    CICD in Azure Databricks using Azure DevOps is a big topic so I have broken this article into 2 parts as below Part 1: Version Control Databricks Notebooks, here we will cover the steps to move workspace notebook to…<\/p>\n","protected":false},"author":1,"featured_media":603,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[27],"tags":[32],"class_list":["post-96","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cummins-inc","tag-sep-2014-feb-2022"],"_links":{"self":[{"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/posts\/96"}],"collection":[{"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/comments?post=96"}],"version-history":[{"count":2,"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/posts\/96\/revisions"}],"predecessor-version":[{"id":555,"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/posts\/96\/revisions\/555"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/media\/603"}],"wp:attachment":[{"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/media?parent=96"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/categories?post=96"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dipankar.me\/yogesh\/wp-json\/wp\/v2\/tags?post=96"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}