Implementing CI/CD for Azure Data Factory

Continuous Integration (CI) is a practice which allows developers to seamlessly merge code in a common repository whereas Continuous Delivery (CD) is a practice that adds a layer to CI by providing multi-stage infrastructure provisioning and deployment which helps in automating the entire software release process.

For Azure data pipelines, CI/CD means nothing but moving data factory pipelines from one environment to another.

In this blog, I will walk you through the process of implementing continuous integration and continuous delivery for pipelines created in Azure data factory.

To begin with, let us first take a look at the high-level architecture:

From this, we can concur that the Azure Data factory is used to perform ETL/ELT operations on data. To implement CI/CD in ADF, Azure DevOps repository needs to be configured first. As shown in above flow, once the pipeline is developed and published, changes can be pushed in the master branch through Pull Request following which a Release pipeline is created to deploy these changes on Test and Production environments. This is done by configuring environment specific connections, which is elaborated later.

Before we move any further, let me tell you that an Azure DevOps account is a mandatory stepping stone if you want to successfully implement CI/CD in ADF.

Now, since implementing CI/CD is the main target of this blog, we will create and use a simple ADF pipeline.

Here are the steps for it:

1. Configure Azure DevOps repository in ADF:

First and foremost, we will create an ADF pipeline in Dev environment and then will deploy it in a QA environment through CI/CD process.

i. Create resource group for Dev environment:

ii. Create ADF instance under Dev resource group: 

iii. Create resource group for QA environment

iv. Create an ADF instance on QA environment: 

v. Now, create new project in Azure DevOps as shown in the diagram below

vi. Configure code repository in ADF:

vii. Configure repository as shown in the diagram below 

viii. Select a working branch:

ix. Add a pipeline to the ADF. These changes will be saved in the above branch. 

x. Create a Pull request to merge the above branch in master branch:

 xi. Once this Pull Request is approved, changes will be merged into master branch: 

xii. ADF pipeline changes are seen under master branch: 

Step 2: Creating Release Pipeline in Azure DevOps:

i.                    Go to Pipeline section in Azure DevOps: 

ii.                   Fill in the details as shown below: 

      iii.          Add and configure steps in Agent Job: 

iii.                 Configure the pipeline as mentioned in below diagram: 

iv.                 For configuring templates, click on the eclipse button and select ArmTemplateforFactory json file.

v.                   For configuring template parameter section, click on the eclipse button and select ArmParameterTemplateforFactory json file. 

vi.               Parameter override section:

          Provide the name required in QA environment ADF instance.

viii. In order to enable continuous integration, enable below option in trigger section. 

ix. Once completed, click on Save button:

 x. Click on run to build artefacts of the pipeline 

Step 3: Creating the final release:

i.   Click on new Pipeline: 

ii. Once clicked on new Pipeline, configure the Release pipeline as shown below:

iii. In stage section add QA instance name:

 iv. Once configured, click on add as well as on continuous deployment trigger symbol to enable.

v. Now click on configure Stage section -> Job link:

 Once this is created, it will add release in queue and will deploy the changes on ADF QA environment.

You can now make changes in Dev environment ADF instance and the same will be released on QA environment ADF instance.

There you have it. Now, that wasn’t so difficult was it?

Reach out to us at Nitor Infotech if you want to learn more about our DevOps offering and how we successfully implemented DevOps for a leading product development company to reduce their defect creation time by 14%.

7 Things You Probably Didn’t Know About AI Engineering

 

As we all know, the business world is changing at a rate faster than you can imagine. To cope up with it, we need to adopt dynamic technologies and engineering practices so our businesses keep running seamlessly. Out of the numerous emerging technologies, one of the most important and significant one is Artificial Intelligence Engineering or AI Engineering. If you are new to this field, do not worry, as everything that you need to know about AI Engineering will be discussed in this blog.

Let us start with the basics: What is AI Engineering?

AI entails behavior and mannerisms are similar to that of a human being performed by a machine. AI Engineering is simply the way it works and functions.

Machine learning mechanism is a deep-rooted technology that has emerged in today’s world like any ordinary technology while Deep Learning is a subcategory of machine learning and is one of the fastest-growing applications of AI used to solve problems that were previously too complex to be solved.

Read the full blog here 

Our Three Ingredient Recipe to Equip Your Business with Hyper Automation

 Adding an extra layer of intelligence to your business processes

What are the challenges with manual processing?

In 2021, most of us are familiar with the common drawbacks of manual processing- increased errors, greater time consumption, plentiful bottlenecks, to name a few. Not too long ago, some guy in a lab put on his thinking cap and his drive to overcome these challenges led to the evolution of automation. Simply put, automation entails the use of technology to bring human interference to a minimum. When we talk about computing, automation is generally performed by a program, script or batch processing that optimizes processes by reducing errors and increasing productivity.

Now you may think that automation is all-encompassing. However, allow me to use this blog to introduce you to a cutting-edge phenomenon that is Hyper automation.

While the idea of Hyper automation is not new as such, the term was coined only last year in Gartner’s Strategic Technology Trends for 2020.  

Essentially, hyper automation is making use of advanced technologies such as artificial intelligence, machine learning, advanced analytics, etc. and inculcating them within your preexisting automation infrastructure to further accelerate processes and augment humans. Think of it as automation on steroids!

And what can one do with this hyper automation, you may ask. The answer is simple- with hyper automation, you can essentially create a digital twin for your organization. Not only that, but you can also whip up this powerful concoction with the help of a three-ingredient secret sauce that goes like this:

1.      A solid mix of RPA and AI

Robotic Process Automation (RPA) is a business process automation technology that partially or fully automates complex, repetitive processes with the help of advanced technologies such as NLP, deep learning, speech, vision, predictive analysis, etc. making it easier to minimize human interference in business processes.

I think I can go out on a limb here and say that at its core, hyper automation is the amalgamation of RPA and AI wherein the RPA mimics human actions through bots and AI simulates human intelligence, and together they facilitate stellar end-to-end processes.

We see the RPA+AI marriage swiftly making its way through a plethora of industries. Let me elaborate with a use case from the Banking, Financial Services, and Insurance Industry.  The BFSI industry benefits amply from hyper automation. RPA and AI can be deployed not only for reporting, marketing, back office operations, etc. but also to proactively monitor transactions and identify fraudulent activities. Additionally, they can also be utilized in claims processing which is possibly one of the most critical aspects of insurance.

2.      A pinch of Business Process Management

No hyper automation process can be successfully implemented without Business Process Management (BPM). As we all know, introducing new digital tools into a business can often be difficult, especially when the business is new to automation. It is therefore supremely important for businesses to create seamless workflows and test them in advance to avoid any breakdowns.

BPM as a tool can be used to effectively manage strategies and initiatives in hyper automation. It carries additional weightage when it comes to industries such as Healthcare which is extremely regulated and effective process automation can make or break operational success.

3.      Finish off with a dash of Advanced Analytics

One of the drawbacks of using RPA in its vanilla format is that it is limited to structured data only. With hyper automation, it is possible to handle both- structured and unstructured data efficiently. It is also possible to convert unstructured data into structured data for use as aid to RPA technologies. With the help of these advance analytics, businesses can access important insights that were otherwise inaccessible and allow disparate digital tools to work together seamlessly.

Handling massive amounts of data is crucial while considering digitization in sectors such as retail. Hyper automation fueled by advanced analytics can help retailers by effectively automating business processes such as order processing, payments, logistics, warehousing, risk management, etc.

And there you have it! You are now acquainted with the revolutionary tool of Hyper automation that you can implement to transform your business processes.

To summarize in layman terms, hyper automation is the difference between beating eggs with a regular whisk as opposed to an electric whisk- reduced effort, time, and guaranteed better results. Once you automate it, there’s no going back. And for a business that means benefits such as:

-        Flexibility

-        Improved productivity

-        Seamless Integration

-        Increased ROI

Finally, at this juncture, I think it is important to establish that for businesses, hyper automation is not only a potential opportunity but an inevitable paradigm shift which augments human capabilities without replacing them. So, before you go ahead and villainize these robots, let me leave you with the assurance that they’re not here to take your job, but to give you a promotion.

If you want to know more about the electric-whisk-equivalent Hyper automation technology, reach out to us at Nitor Infotech and read to know more about our AI-ML capabilities.