Automating Microfocus Data Protector using AWS Lambda (Python) + DP APIs + Windows Scheduler + SES

The internet is insecure by default. SSL certificates add a level of safety.

Kuharan Bhowmik

Published in

Nerd For Tech

5 min readSep 17, 2020

What is Dataprotector?

Micro Focus Data Protector is an enterprise-grade backup and disaster recovery solution for large, complex, and heterogeneous IT environments. It is a backup solution that provides reliable data protection and high accessibility for your fast-growing business data and offers comprehensive backup and restores functionality specifically tailored for enterprise-wide and distributed environments.

DP — APIs

Fortunately, Data Protector uses Swagger as a framework for APIs. You can access Data Protector APIs from the Help > API Documentation menu option on the Data Protector GUI.

It has a lot of APIs but let us focus on the APIs that are needed for this solution. Basically, log in and filter.

The bottom line is that the login API will give us an access token for making subsequent requests.

Login Code —

url = "https://xxx.xxx.xxx.com:xxxx/auth/realms/DataProtector/protocol/openid-connect/token"payload = {"username":"user_name|*|xxx.xxx.xxx.com", "password":"password", "client_id":"dp-gui", "grant_type":"password"}response =  json.loads(session.post(url, data = payload, verify = r"path_to_cert\xxx.xxx.xxx.com_cacert.pem").text)

Now, this needs an SSL certificate for authentication and should be present in the server you are trying to make a request to. This is a self-signed certificate and it will not have a certificate chain or a key/cert file. This pem file will have only one certificate present inside the file. You need to directly download the certificate from the server (you are going to make a request to) to the project directory. Then, provide the path of the downloaded pem file in the verify param while making the request.

The SSL problem and the solution —

You may get errors like —

SSL verification failed — For this, check the path to the certificate and the verify parameter. Do not try to bypass the authentication.
Grant type — Do not try to base 64 encode the payload. Send it as-is i.e plain json object

The code that can automate it —

After login:

It tries to filter the session check_session based on session filters (provided in API docs).

url = "https://xxx.xxx.xxx.com:xxx/idb/sessions/filter/"payload = "{\r\n  \"filter\": {\r\n    \"status\": [0,1,12,15],\r\n    \"sessionType\": [0]\r\n  }\r\n}"
    headers = {
        'Authorization': 'Bearer {}'.format(access_token),
        'Content-Type': 'application/json'
        }response = json.loads(session.post(url, headers=headers, data = payload, verify = r"path_to_cert\xxx.xxx.xxx.com_cacert.pem").text)return response

Now, here it uses a status filter and a session type filter. Status input can be provided in one of two supported formats:

Example —

{
   "filter":{
      "status":36
   }
}

This filter will return the sessions of status “Completed” and “Failed”. The value 36 is formed by setting the bit positions 2 and 5. How? —

1*2⁵+0*2⁴+0*2³+1*2²+0*2¹+0*2⁰ = 36

The above one is similar to this —

{
   "filter":{
      "status":[2,5]
   }
}

The above filter will return the sessions of type “Completed” and “Failed”.

The table below describes the mapping of bit position for each session type —

Bit position Status Type
0 Running
1 Running with Errors
2 Completed
3 Completed with Errors
4 Aborted with Failure
5 Failed
6 Aborted
7 Aborted with Errors
8 Mounted
9 Mounted with Errors
10 Queueing
11 Queueing with Errors
12 Running with Failures
13 Completed with Failures
14 Mounted with Failures
15 Queueing with Failures
16 Completed Mirror with Failures
17 Session Status No

Similarly, session inputs can also have two types —

{
   "filter":{
      "sessionType":131073
   }
}

The above filter will return the sessions of types “BACKUP” and “CONSOLIDATION”. The value 131073 is formed by setting the bit positions 0 and 17.

{
   "filter":{
      "sessionType":[0,17]
   }
}

The above filter will return the sessions of type “BACKUP” and “CONSOLIDATION”.

The table below describes the mapping of bit position for each session type —

Bit position Session Type
0 Backup
1 Restore
2 Database Query
3 Media Management
4 Backup preview
5 Restore preview
6 Test
7 BAR session(backup)
8 BAR session(restore)
9 Media management daemon
10 Purge
11 Restart
12 Admin session
13 Admin UDP
14 Admin purge
15 Backup Diskonly
16 Copy
17 Consolidation
18 Key management daemon

2. Then filters the long-running jobs with a custom param ‘hours’ in check_duration

epoch_time = int(time.time())
    # print(f"Now - {epoch_time} Session Start Time - {session_start_time}")
    if (epoch_time - session_start_time) > hours * 3600:
        return True
    else:
        return False

3. It creates a small structure to be sent to API-gateway.

if long_run_status:
            output['Name'] = response['items'][item]['name']
            output['Datalist'] = response['items'][item]['datalist']
            output['SessionType'] = session_type_dict[str(response['items'][item]['sessionType'])]
            output['Status'] = status_dict[str(response['items'][item]['status'])]
            output['StartTime'] = convert_epoch_to_datetime(response['items'][item]['startTime'])
            output['Owner'] = response['items'][item]['owner']
            long_running_sessions.append(output)

You can choose to dump it —

with open ("longrunningjobs.json", 'w+') as fp:
    json.dump(long_running_sessions, fp)

4. Finally, it sends a request to the lambda API-gateway where the payload JSON is converted to CSV and emailed. (You can find the email code and CSV converter in my other aws lambda blogs)

url = "https://api-gateway.us-east-1.amazonaws.com/default/lambda_name"headers = {
        'Content-Type': 'application/json'
    }response = requests.get(url, headers=headers, data = json.dumps(long_running_sessions))

You can choose to write this function in lambda itself and integrate it with the email code. But then, the lambda should have the vpc attached for the particular server and the code should have the certificate with it in the deployment package.

The End to End Code —

EC2 with windows scheduler (An experiment) — VPC attached

In my case, I had an EC2 that has the VPC attached, so thought of using it. The next step is to schedule it.

Steps:

Create a .bat file with the following content

"path_to_python/python.exe" "path_to_project_folder/main.py"
pause

2. Create a basic task in the windows task scheduler

Choose a trigger, action as start a program, and browse the path of the bat file and finish.

To the one reaching this part of the page — I hope you enjoyed reading the article as much as I enjoyed writing it. Incase of blockers, don’t hesitate to reach me!

I can be found here — https://www.linkedin.com/in/kuharan/