Azure Files Streaming Configuration

Prerequisites

Existing Azure Blob connection: An Azure Files scan configuration must already exist.

If an Azure Files scan does not already exist, follow this guide to Azure Files scan and ensure the necessary credentials are set up.

Select an Existing Scan Configuration

1. Go to the Scan configurations page in the product UI.
2. Find your existing Azure Blob scan configuration and select Edit Configuration from the options menu.
   Note: The configured path (folder) and save it, as it will be used in step 9 to replace {FolderPath}.

Enable Data Streaming

1. Within the Edit Azure Blob Scan Configuration page, toggle Data Streaming to ON.
2. Copy the Webhook URL provided, as you will use it later in the Azure Portal.

   Save this Webhook URL, as it will be used in step 9 to replace {WebhookUrl}.

3. Click Save & Close button to save configuration.

Create Azure Event Hub

1. Navigate to Azure Portal Event hubs and click Create.

   

2. In Create Namespace Window fill in the details
   • Give it a Name.
   • Select your subscription and resource group.
   • Select location.
   • Pricing tier - standard.
   • Throughput Units - 1.

     

3. Click on Review + Create and then Create after validation.

   

4. After namespace is created, click on + Event Hub button.

   

5. In Create Event Hub Window fill in name and click Create + Review and Create after validation. Save the name of the Event Hub you created in this step, as it will be used later in step 9 to replace {eventHubName}.

   

6. Configure access policy
   1. In the event hubs namespace window click on Settings/Shared access policies and then +Add button.

      

   2. Fill in the details in the new tab, set LogicAppsListenerPolicy as name, select Listen policy, and click Save.
   3. Click on the newly created policy, then copy and save the Connection string–primary key. This will be needed later in step 8b.

Configure Azure Storage Diagnostic settings

1. Navigate to Azure Portal and open your Storage Account.

   

2. Select needed account from the Storage Accounts.

   

3. In the left-hand menu, select Monitoring/Diagnostic settings and click file.

   

4. In Diagnostic settings Window click on "+ Add diagnostic setting" button.

   

5. In Create Diagnostic setting Window fill in the details:
   1. Give it a Name.
   2. Select Category groups allLogs.
   3. Select Destination details Stream to an event hub and select newly created Event Hub Namespace and Event Hub.

      

   4. Click Save.

Configure Azure Logic Apps

1. Go to Azure logic apps and click Add button.

   

2. In Create Logic App Window select Workflow Service Plan.
3. In Create Logic App (Workflow Service Plan) Window fill in the details and click Create + Review:
   1. Select your subscription and resource group.
   2. Give logic app name
   3. Select region
   4. Pricing plan should be WS1
   5. In the monitoring tab select No for the application insights.

      

   6. Click Review + create button.
4. Click Create after validation.
5. In newly created logic app click on Workflows/Workflows and then +Add button.
6. In new workflow tab fill in name, select State type: Stateful and click Create.

   

7. In created workflow go to Developer/Designer and click on Add a trigger, then in search type Event hub and select When events are available in Event Hub.

   

8. Configure API connection
   1. Click on the trigger, set "Temp" for Event Hub Name and then click on Change connection.

      

   2. Then click Add New and fill in the details. Enter any name for the connection name and use the connection string {Connection string–primary key} from step 3.6.c.
   3. On the Change Connection tab, click Details and copy the Name from the connection details. Save this Name, as it will be used later in step 9 to replace {connectionName}.

      

   4. Click save on workflow designer window.

      

9. In workflow navigation tab go to Developer/Code and set the provided code, then click save:
   1. Replace {FolderPath} with a path to the streaming folder. For example, you want to get events from the folder "StreamingFolder" which is located in file share with the name "DocumentsShare" and in the folder with the name "Personal". In this case, the path should be "DocumentsShare/Personal/StreamingFolder"
   2. Replace {WebhookUrl} with webhook URL provided in the application in the scan configuration window.
   3. Replace {eventHubName} with azure event hub name that was created previously.
   4. Replace {connectionName} with connection name from previouse step.

      {
          "definition": {
              "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
              "actions": {
                  "Filter_Records": {
                      "type": "Query",
                      "inputs": {
                          "from": "@triggerBody()?['ContentData']?['records']",
                          "where": "@and(not(empty(item()?['uri'])),or(contains(item()?['uri'], '{FolderPath}/'),contains(item()?['uri'], '{FolderPath}?')))"
                      },
                      "runAfter": {}
                  },
                  "Condition": {
                      "type": "If",
                      "expression": "@greater(length(body('Filter_Records')), 0)",
                      "actions": {
                          "HTTP-copy": {
                              "type": "Http",
                              "inputs": {
                                  "uri": "{WebhookUrl}",
                                  "method": "POST",
                                  "headers": {
                                      "Content-Type": "application/json"
                                  },
                                  "body": {
                                      "event": "@setProperty(triggerBody(),'ContentData',setProperty(triggerBody()?['ContentData'],'records',body('Filter_Records')))"
                                  }
                              },
                              "runAfter": {}
                          }
                      },
                      "else": {},
                      "runAfter": {
                          "Filter_Records": [
                              "Succeeded"
                          ]
                      }
                  }
              },
              "contentVersion": "1.0.0.0",
              "outputs": {},
              "triggers": {
                  "When_events_are_available_in_Event_Hub": {
                      "type": "ApiConnection",
                      "inputs": {
                          "host": {
                              "connection": {
                                  "referenceName": "{connectionName}"
                              }
                          },
                          "method": "get",
                          "path": "/@{encodeURIComponent('{eventHubName}')}/events/batch/head",
                          "queries": {
                              "contentType": "application/json",
                              "consumerGroupName": "$Default",
                              "maximumEventsCount": 50
                          }
                      },
                      "recurrence": {
                          "interval": 30,
                          "frequency": "Second"
                      },
                      "splitOn": "@triggerBody()"
                  }
              }
          },
          "kind": "Stateful"
      }
      

Next Steps

Troubleshooting