Azure数据工厂管道 ml

Question

我正在尝试在Azure Data Factory V1中进行管道，该V1将在文件上执行Azure批处理。我使用BLOB存储作为输入和输出实现了它，并且它起作用。但是，我并不是要将输入和输出更改为数据湖商店中的文件夹。当我尝试部署它时，它会给我以下错误：

Entity provisioning failed: AzureML Activity 'MLActivity' specifies 'DatalakeInput' in a property that requires an Azure Blob Dataset reference.  

我如何将输入和输出作为datalakestore而不是斑点？

管道：

{
        "name": "MLPipeline",
        "properties": {
            "description": "use AzureML model",
            "activities": [
                {
                    "type": "AzureMLBatchExecution",
                    "typeProperties": {
                        "webServiceInput": "DatalakeInput",
                        "webServiceOutputs": {
                            "output1": "DatalakeOutput"
                        },
                        "webServiceInputs": {},
                        "globalParameters": {}
                    },
                    "inputs": [
                        {
                            "name": "DatalakeInput"
                        }
                    ],
                    "outputs": [
                        {
                            "name": "DatalakeOutput"
                        }
                    ],
                    "policy": {
                        "timeout": "02:00:00",
                        "concurrency": 3,
                        "executionPriorityOrder": "NewestFirst",
                        "retry": 1
                    },
                    "scheduler": {
                        "frequency": "Hour",
                        "interval": 1
                    },
                    "name": "MLActivity",
                    "description": "description",
                    "linkedServiceName": "MyAzureMLLinkedService"
                }
            ],
            "start": "2016-02-08T00:00:00Z",
            "end": "2016-02-08T00:00:00Z",
            "isPaused": false,
            "hubName": "hubname",
            "pipelineMode": "Scheduled"
        }
    }

输出数据集：

  {
        "name": "DatalakeOutput",
        "properties": {
            "published": false,
            "type": "AzureDataLakeStore",
            "linkedServiceName": "AzureDataLakeStoreLinkedService",
            "typeProperties": {
                "folderPath": "/DATA_MANAGEMENT/"
            },
            "availability": {
                "frequency": "Hour",
                "interval": 1
            }
        }
    }

输入数据集：

 {
        "name": "DatalakeInput",
        "properties": {
            "published": false,
            "type": "AzureDataLakeStore",
            "linkedServiceName": "AzureDataLakeStoreLinkedService",
            "typeProperties": {
                "fileName": "data.csv",
                "folderPath": "/RAW/",
                "format": {
                    "type": "TextFormat",
                    "columnDelimiter": ","
                }
            },
            "availability": {
                "frequency": "Hour",
                "interval": 1
            }
        }
    }

AzureTatalakestorelinkedService：

{
    "name": "AzureDataLakeStoreLinkedService",
    "properties": {
        "description": "",
        "hubName": "xyzdatafactoryv1_hub",
        "type": "AzureDataLakeStore",
        "typeProperties": {
            "dataLakeStoreUri": "https://xyzdatastore.azuredatalakestore.net/webhdfs/v1",
            "authorization": "**********",
            "sessionId": "**********",
            "subscriptionId": "*****",
            "resourceGroupName": "xyzresourcegroup"
        }
    }
}

根据数据工厂V1，链接的服务是在本教程之后完成的。

Answer 1

我认为AzureTatalakestorelinkedService存在一些问题。请验证。

根据用于访问数据存储的身份验证，您的AzureAtAtalakestorelinkedService JSON必须看起来像以下内容 -

使用服务主体身份验证

{
    "name": "AzureDataLakeStoreLinkedService",
    "properties": {
        "type": "AzureDataLakeStore",
        "typeProperties": {
            "dataLakeStoreUri": "https://<accountname>.azuredatalakestore.net/webhdfs/v1",
            "servicePrincipalId": "<service principal id>",
            "servicePrincipalKey": {
                "type": "SecureString",
                "value": "<service principal key>"
            },
            "tenant": "<tenant info, e.g. microsoft.onmicrosoft.com>",
            "subscriptionId": "<subscription of ADLS>",
            "resourceGroupName": "<resource group of ADLS>"
        },
        "connectVia": {
            "referenceName": "<name of Integration Runtime>",
            "type": "IntegrationRuntimeReference"
        }
    }
}

使用托管服务身份身份验证

{
    "name": "AzureDataLakeStoreLinkedService",
    "properties": {
        "type": "AzureDataLakeStore",
        "typeProperties": {
            "dataLakeStoreUri": "https://<accountname>.azuredatalakestore.net/webhdfs/v1",
            "tenant": "<tenant info, e.g. microsoft.onmicrosoft.com>",
            "subscriptionId": "<subscription of ADLS>",
            "resourceGroupName": "<resource group of ADLS>"
        },
        "connectVia": {
            "referenceName": "<name of Integration Runtime>",
            "type": "IntegrationRuntimeReference"
        }
    }
}

这是用于参考的Microsoft文档 - 使用Azure Data Factory