Avamar: Searching across clients and backup labels

Typically when I run into a scenario with a customer who wants to search across clients and backup labels for files the answer is DPSearch... drop the mike.

• Dell Data Protection Search 19.6.5 Release Notes

About Search

Search is a virtual appliance that supports indexing and searching across one or more backup servers. In Search, a backup server is known as a source.

With Search you can perform the following actions:

• Index and search for files by name, location, size, owner, file type, and date.
• Perform a targeted Full Content Index on search results to view a preview of the content and search for keywords and content inside.
• Restore files directly from search results.
• Perform advanced search queries including symbols, wildcards, filters, and operators.
• Preview and download search results.

This would be a very short article if I said "DPSearch" then dropped the mike. Also quite frankly it would have zero to do with automation so let's look at a scenario where DPSearch isn't being, or couldn't be used for whatever reason.

Scenario

Let's say you're leveraging Avamar to backup a sensitive network segment like PCI, or HIPPA and your security & compliance team dictates that ONLY the backup software, clients and your monitoring & alerting toolset can reside within this network segment.

In this scenario we are 100% compliant. So without DPSearch how could we potentially search across clients and backup labels? I would love to say the answer is the REST API but in this case its not so.

The answer is actually the Avamar, avtar, CLI... specifically the following command:

"avtar -v --list --noinformational --path=$($Client.fqdn) --labelnum=$($Label)"        

This command will return a report containing every file path backed up for the specified client and label. Let's take a look at the output for the command.

This avtar command contains EXACTLY the information we need... just not in a format that is useful, for the operations we are interested in...

Search across multiple clients and backup labels leveraging pattern matching (regular expressions) to find specific files in specific folders contained within the backup labels.

So now lets make it useful...

Thinking though the process we'll need to:

• Establish a programmatic SSH session with Avamar
• Pass in the avtar command above for each client and backup label we want to search across
• Capture the output and pattern match against the full file path to achieve the desired results

Going back to our scenario... the security and compliance teams wants to verify that the recoverme.txt file resides within the web1 and web3 folders in backup labels 1 and 2 for clients win-fsa-01 and win-fsa-02.

Picky, Picky...

Below is the path they are interested in on each client but we aren't going to specify the full path as part of our search because that would be cheating AND not showcase the power of regular expressions (pattern matching) for our use case.

Let's do this...

In pattern matching against the file path we came up with the following basic regular expression:

/web[1|3]/recoverme.txt$

This loosely translates to match at ANY file path that has a web1, or web3 folder and ends with recoverme.txt.

Results... (imagine doing this manually lol)

FULL concrete code example below. It requires PowerShell7 and POSH-SSH

[CmdletBinding()]
param
(
    [parameter(Mandatory=$true)]
    [string]$Pattern

)

$Search = @(
    # THIS IS A SEARCH REQUEST
    @{
        # LOOKING ON THIS AVAMAR NODE
        avamar = 'ave-01.vcorp.local'
        # FOR THE FILE MATCHING THIS REGULAR EXPRESSION
        match = $Pattern
        # ON THESE CLIENTS
        clients = @(
            # CLIENT
            @{
                fqdn = '/clients/win-fsa-01'
                labels = @(1,2)
            } # END CLIENT
            @{
                fqdn = '/clients/win-fsa-02'
                labels = @(1,2)
            } # END CLIENT
        ) # END CLIENTS
    } # END SEARCH
)
# REMOVE HOST KEYS
Get-SSHTrustedHost | Remove-SSHTrustedHost | Out-Null

$Total = @{
    files = 0
    matches = 0
}
$Report = @()
$Search | ForEach-Object {
    # CHECK FOR SSH CREDENTIALS
    $exists = Test-Path -Path ".\$($_.avamar).xml" -PathType Leaf
    if($exists) {
        $Credential = Import-CliXml ".\$($_.avamar).xml"
    } else {
        $Credential = Get-Credential `
        -Message "Please specify your credentials for ssh access to Avamar."
        $Credential | Export-CliXml ".\$($_.avamar).xml"
    }

    # TIMER
    $StopWatch = New-Object -TypeName System.Diagnostics.Stopwatch
    $StopWatch.start()

    # SSH SESSION
    try {
        $SSH = New-SSHSession `
        -ComputerName $_.avamar `
        -Credential $Credential `
        -AcceptKey `
        -KeepAliveInterval 60 `
        -ConnectionTimeout 60

    } catch {
        $_.message
    }
    
    
    foreach($Client in $_.clients) {
        $x = @()
        Write-Host "`n[Processing]: $($Client.fqdn)"
        foreach($Label in $Client.labels) {
            Write-Host "[Searching]: Label $($Label) for $($_.match)" -ForegroundColor Yellow
            # INVOKE THE SSH COMMAND
            $Cmd = Invoke-SSHCommand `
            -Command "avtar -v --list --noinformational --path=$($Client.fqdn) --labelnum=$($Label)" `
            -SessionId $SSH.SessionId `
            -Timeout 1800

            # GET THE COUNTS
            $x = $Cmd.Output
            $y = $Cmd.Output -match $_.match
            
            # AGGERATE THE FILE COUNTS
            $Total.files = $Total.files + $x.length
            $Total.matches = $Total.matches + $y.length
            
            $color = "Yellow"
            if($y.length -gt 0){
                $color = "Green"
            }
            Write-Host "[Results]: $( $y.length ) matches out of $( $x.length ) files`n" -ForegroundColor $color
            foreach($match in $y) {
                $temp = $match -split '\s'
                $object = [ordered]@{
                   client = $Client.fqdn
                   permissions = $temp[0]
                   account = $temp[1]
                   path = $temp[$temp.length-1]
                }
                $Report += (New-Object -TypeName psobject -Property $object)
            }
        }
       
    }

}
# STOP THE TIMER
$StopWatch.stop()
Write-Host
Write-Host "[TOTALS]: $(($Total.matches).ToString("N0")) matches out of $(($Total.files).ToString("N0")) file found in h:$($StopWatch.Elapsed.Hours) m:$($StopWatch.Elapsed.Minutes) s:$($StopWatch.Elapsed.Seconds)" `
-ForegroundColor Blue

$Report | Format-Table -AutoSize        

And with that have a great week and my apologies for the lag time between automation articles!