Azure Storage Account is one of the foundation services of Azure. Pretty much all other services use it under the cover in a way or another. This service is simple, works great, and has crazy SLA and redundancy capabilities. However, it doesn't provide a point in time restore. Meaning that if you corrupt or delete some data, there's no way to recover it. There are some solutions that you can find over the internet, but unfortunately, none of them is a silver bullet. Let's explore and compare some of them. Requirements Let's first define some goals and requirements before we jump into solutions Backups must run on a schedule (e.g., every hour) Backups must be fast Minimize network traffic $$$ The solution must be easy to maintain and operate Get notification or report on failures Must support multi-environment configuration (dev, QA, prod, etc...) Has support for both Blob and Table storage format AzCopy AzCopy is a great tool to move data around in Azure, and it's also what most people will suggest using for backups. Target a source and a destination backup storage account and voilà! The AzCopy command-line performs the processing of that operation inside Azure datacenter between the two target storage accounts instead of passing by the machine that runs AzCopy. This feature alone makes AzCopy an obvious choice as it covers 2 of the most important objectives: No. 2: Backup must be fast No. 3: Minimize network traffic $$$ That's why most backup solutions use this tool. Solution 1: Running from your PC AzCopy is a command-line tool, so it's pretty easy to run from your machine or an on-prem server to backup your storage accounts. You can create a windows schedule to run that command, and you are good to go. It works fine when you have a few storage accounts, but grows complex real fast when adding more storage. Pros Easy to set up and automate Cons No easy way to get notified on a failure Not easy to set it up for multiple environments Scalability Solution 2: Azure Automation Azure Automation allows you to run PowerShell script directly in Azure. With that service, you can easily create a script that launches AzCopy with the specified arguments to target your storage accounts. Pros Easy to set up a schedule Cons Not that simple to install a .exe and run it on their hosted agent (You need to run it with a Hybrid Runbook Worker) Not easy to set it up for multiple environments No notification on failures Solution 3: Let's try thinking outside the box What about using Azure DevOps? It seems crazy right, but let's look at it more closely. One of the common problems with previous solutions is the management of backups for multiple environments and notifications on failure. Another problem is to install AzCopy on the agent or machine that's going to run the backup. Good news though, AzCopy is already installed by default on all Azure DevOps agents. So here's a general idea: Create a pipeline that runs on a schedule and performs backup for each environment. 1. Create a PowerShell module with utility methods to make the use of AzCopy easier and upload it to your git repository EDIT: You can now download all these scripts from . Also, special thanks to my good friend who cleaned up the whole thing and added the restore functionality. Github Yohan Backup-Storage-Accounts.psm1 Build-AzCopyCmd
{ (
    [Parameter(Mandatory)]
    [string] ,
    [Parameter(Mandatory)]
    [PSCustomObject] ,
    [Parameter(Mandatory)]
    [PSCustomObject] ,
    [Parameter(Mandatory)]
    [string] ,
    [Parameter(Mandatory)]
    [string] ) = .StorageAccount.Credentials.ExportBase64EncodedKey() = .StorageAccount.Credentials.ExportBase64EncodedKey() = .StorageAccount.CreateCloudBlobClient().GetContainerReference( ) [string]::Format( , , , .Uri.AbsoluteUri, , )
} Invoke-AzCopyCmd
{ (
    [Parameter(Mandatory)]
    [string] ) = cmd /c ( )
  { } ( ){ ; ;
  } { } } Backup-Blobs
{ (
    [Parameter(Mandatory)]
    [string] ,
    [Parameter(Mandatory)]
    [PSCustomObject] ,
    [Parameter(Mandatory)]
    [PSCustomObject] ,
    [Parameter(Mandatory)]
    [array] ) { ( )
    { ( .Name )
      { ;
      } = + + .Name = = .CloudBlobContainer.Uri.AbsoluteUri = Build-AzCopyCmd -DestinationPath -SrcCtx -DestCtx -AzCopyParam -SourcePath Invoke-AzCopyCmd -AzCopyCmd }
  }
} Backup-Tables
{ (
    [Parameter(Mandatory)]
    [string] ,
    [Parameter(Mandatory)]
    [PSCustomObject] ,
    [Parameter(Mandatory)]
    [PSCustomObject] ,
    [Parameter(Mandatory)]
    [array] ) { ( )
    { = + + .Name = = .CloudTable.Uri.AbsoluteUri = Build-AzCopyCmd -DestinationPath -SrcCtx -DestCtx -AzCopyParam -SourcePath Invoke-AzCopyCmd -AzCopyCmd }
  }
} Backup-StorageAccount
{ (
    [Parameter(Mandatory)]
    [PSCustomObject] ,
    [Parameter(Mandatory)]
    [PSCustomObject] ) = ( ).ToUniversalTime().tostring( ) = .StorageAccount.Credentials.AccountName = + + = Get-AzureStorageTable -Context ( )
  {
    Backup-Tables -DestinationPath -SrcCtx -DestCtx -SrcStorageTables } = = { = Get-AzureStorageContainer -MaxCount -ContinuationToken -Context ( )
    { = [ .Count - ].ContinuationToken;
      Backup-Blobs -DestinationPath -SrcCtx -DestCtx -SrcStorageContainers }
  } ( )
} Get-StorageAccountContext
{ (
    [Parameter(Mandatory)]
    [string] ,
    [Parameter(Mandatory)]
    [string] ) = (Get-AzureRmStorageAccountKey -ResourceGroupName -AccountName ).Value[ ] New-AzureStorageContext -StorageAccountName -StorageAccountKey } -Function Get-StorageAccountContext -Function Backup-StorageAccount function param $DestinationPath $SrcCtx $DestCtx $AzCopyParam $SourcePath $srcStorageAccountKey $SrcCtx $destStorageAccountKey $DestCtx $destContainer $DestCtx $DestinationPath return "" "{0}" " /source:{1} /dest:{2} /sourcekey:" "{3}" "
      /destkey:" "{4}" " " $AzCopyParam "C:\Program Files (x86)\Microsoft
      SDKs\Azure\AzCopy\AzCopy.exe" $SourcePath $destContainer $srcStorageAccountKey $destStorageAccountKey function param $AzCopyCmd $result $AzCopyCmd foreach $s in $result Write-Host $s if $LASTEXITCODE -ne 0 Write-Error "Copy failed!" break else Write-Host "Copy succeed!" Write-Host "-----------------" function param $DestinationPath $SrcCtx $DestCtx $SrcStorageContainers Process foreach $srcStorageContainer in $SrcStorageContainers if $srcStorageContainer -like '*$*' Write-Host "-----------------" Write-Host "Skipping copy: $( .Name)" $srcStorageContainer Write-Host "-----------------" Continue Write-Host "-----------------" Write-Host "Start copying: $( .Name)" $srcStorageContainer Write-Host "-----------------" $blobDestinationPath $DestinationPath "/blobs/" $srcStorageContainer $azCopyParam "/snapshot /y /s /synccopy" $sourcePath $srcStorageContainer $azCopyCmd $blobDestinationPath $SrcCtx $DestCtx $azCopyParam $sourcePath $AzCopyCmd function param $DestinationPath $SrcCtx $DestCtx $SrcStorageTables Process foreach $srcStorageTable in $SrcStorageTables Write-Host "-----------------" Write-Host "Start copying: $( .Name)" $srcStorageTable Write-Host "-----------------" $tableDestinationPath $DestinationPath "/tables/" $srcStorageTable $azCopyParam "/y" $sourcePath $srcStorageTable $azCopyCmd $tableDestinationPath $SrcCtx $DestCtx $azCopyParam $sourcePath $AzCopyCmd function param $SrcCtx $DestCtx $currentDate Get-Date 'yyyy\/MM\/dd\/HH:mm' $SrcStorageAccountName $srcCtx $destinationPath $SrcStorageAccountName "/" $currentDate $srcTables $srcCtx if $srcTables $destinationPath $SrcCtx $destCtx $srcTables $maxReturn 250 $token $null do $srcContainers $maxReturn $token $srcCtx if $srcContainers $token $srcContainers $srcContainers 1 $destinationPath $SrcCtx $destCtx $srcContainers While $token -ne $null function param $StorageAccountName $StorageAccountResourceGroup $storageAccountKey $StorageAccountResourceGroup $StorageAccountName 0 return $StorageAccountName $storageAccountKey Export-ModuleMember Export-ModuleMember 2. Create a script to launch the backup Backup-Storage-Account.ps1 = Get-StorageAccountContext -StorageAccountName -StorageAccountResourceGroup ( )
  { = .storageAccountName = .resourceGroup = Get-StorageAccountContext -StorageAccountName -StorageAccountResourceGroup Backup-StorageAccount -SrcCtx -DestCtx } $destCtx $destStorageAccountName $destResourceGroup foreach $srcStorageAccount in $srcStorageAccounts $srcStorageAccountName $srcStorageAccount $srcResourceGroup $srcStorageAccount $srcCtx $srcStorageAccountName $srcResourceGroup $srcCtx $destCtx 3. Create a task group in Azure DevOps 4. Add the Azure PowerShell task, so the script will run in an Azure context and will be authenticated to the Storage Accounts you want to backup (make sure that Azure DevOps has access to your Azure Subscription) 5. Here's the YAML of that task. Note that I used variables to make it more reusable in different environments $(subscription) -> So you can target different Azure subscriptions $(RepositoryPath) -> Git repository path that contains the scripts $(EnvironmentCode) -> Variable that can be used in your PowerShell script to make custom logic depending on the environment steps: - task: AzurePowerShell@3 displayName: 'Run Backup Azure PowerShell Script' inputs: azureSubscription: '$(subscription)' ScriptPath: '$(RepositoryPath)\src\Backup\Scripts\Backup-Storage-Accounts.ps1' ScriptArguments: '-EnvironmentCode $(EnvironmentCode)' FailOnStandardError: true azurePowerShellVersion: LatestVersion 6. Create a new Release build pipeline with all your environments as stages 7. Add an artifact and select Azure Repos 8. Set a schedule to kick off the backups on all environments 9. Configure each environment to use your Task Group 10. Configure your variables for each environment 11. You can now look at all the backup runs and get notifications for free when it fails (default behavior of Azure DevOps when a build fails) Pros Easy to set up for multiple environments with the release pipeline Audit trail of all the changes in the release pipeline Support for various environments Overview of all the backup runs Built-in notifications on failure Cons You need an Azure DevOps account (but you can subscribe to a free plan here) Conclusion Sometimes the solution is more straightforward than it looks. You need to leverage existing tools that you know. Try to think outside the box, and you'll find out that your absurd ideas might not be that crazy after all. An essential skill for a developer is the ability to solve business problems in new creative ways using technology. Previously published at https://blog.miguelbernard.com/did-you-backup-you-azure-storage-accounts/

Microsoft

Target

How to Backup Your Microsoft Azure Storage Accounts

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

5 Tips to Improve Your Productivity in C# 8.0

10 reasons to give cloud computing a go

10 Microsoft Azure Courses for Beginners to Learn Azure Cloud Computing

The Noonification: Proglogging: The Developers Detective Toolkit (10/9/2023)

Top 10 AI Development Companies in USA

2023 Will Be the Year of Kubernetes (and Other Predictions in the Cloud and Infrastructure Industry)

5 Tips to Improve Your Productivity in C# 8.0

10 reasons to give cloud computing a go

10 Microsoft Azure Courses for Beginners to Learn Azure Cloud Computing

The Noonification: Proglogging: The Developers Detective Toolkit (10/9/2023)

Top 10 AI Development Companies in USA

2023 Will Be the Year of Kubernetes (and Other Predictions in the Cloud and Infrastructure Industry)

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps