At Microsoft, with the announcement of v2 of the Azure Data Factory service (ADF) preview service, we've invested in expanding the data integration service in Azure to enable a series of new use cases that we found to be very popular and very common in cloud-first ETL and data integration scenarios. The previous ADF service is a very time-window and ELT-focused service that enabled many Big Data Analytics projects centered on data movement, data lakes and machine learning. With ADF V2, enabling an expanded set of capabilities within control flow orchestration, flexible time & trigger based scheduling, on-demand spark execution and enabling SSIS services in the cloud, has enabled customers to build architectures using ADF that center around Modernizing the Data Warehouse, data-driven custom SaaS application and SSIS ETL lift & shift. Here is the overview to ADF V2, which builds on top of the existing ADF V1 capabilities:
We're going to publish a separate detail blog about SSIS in the cloud with ADF, so let me focus on a couple of features that are important to each of those 3 scenarios and we'll dive deeper into SSIS in a later blog post.
1. Modern Data Warehouse
With ADF V2, you can schedule your pipelines to run on a wall-clock schedule or trigger from an external event. A focus on control flow for your orchestrations that includes branching, conditional execution, parameters and looping enable incremental data loads and complex workflow. A new lookup activity is also very important in these scenarios. Click here for more on scheduling, control flow, lookup activity and incremental data load.
2. Data-Driven SaaS Application
In this scenario, we found that customers building their own cloud-based applications with data intelligence, analytics, ML, etc., required support for SDKs like Python and REST as well as support for connecting to hybrid data sources in the cloud, on-prem and in VNETs. Here is more info on the Python SDK for ADF and here you will find details on the ADF Integration Runtime with VNET support.
3. Lift & Shift SSIS ETL Packages to the Cloud
Running SSIS packages as a managed service in Azure provides the benefit of migrating to a cloud-first data integration architecture with minimal changes required from your data integration code and workflows. It also provides a means to leverage ADF with its extensive monitoring and management facilities to provision SSIS servers without needing to self-manage, install, maintain, patch, etc. SSIS servers. Details on running SSIS as a cloud service inside ADF can be found here.
