Installing Python and Configuring Linux Diagnostic Extension on an Azure VM

In this post, we’ll walk through how to manage Azure Virtual Machines (VMs) using Python, including installing Python 2, checking Python 3, and configuring the Azure Linux Diagnostic Extension. This tutorial uses the azure-identity and azure-mgmt-compute Python packages to authenticate and interact with Azure VMs. We’ll perform basic setup, run commands remotely on the VM, and configure diagnostic monitoring for enhanced performance insights.

Prerequisites

Azure Account: Ensure you have an active subscription and the necessary permissions to manage Azure VMs.
Python Environment: Install Python 3 and the following Python packages: pip install azure-identity azure-mgmt-compute
Azure VM: This example assumes you have an Azure VM running in the specified resource group.

Code Walkthrough

Here’s a detailed breakdown of the code to run a shell command on an Azure VM and install and configure the Linux Diagnostic Extension.

Step 1: Set Up Environment Variables

Define environment variables for your subscription, resource group, VM name, location, and storage account. These values are used to specify the Azure resources you’re working with.

subscription_id = "your_subscription_id"
resource_group_name = "your_resource_group"
vm_name = "your_vm_name"
location = "your_location"
storageAccountName = "your_storage_account_name"

Step 2: Authenticate and Initialize the Compute Client

Use DefaultAzureCredential to authenticate with Azure and initialize the ComputeManagementClient.

from azure.identity import DefaultAzureCredential
from azure.mgmt.compute import ComputeManagementClient

credential = DefaultAzureCredential()
compute_client = ComputeManagementClient(credential, subscription_id)

Step 3: Run Shell Command on the VM

The RunCommandInput API allows us to run shell scripts on the VM. Here, we install Python 2 and check the installed Python 3 version.

from azure.mgmt.compute.models import RunCommandInput

command = RunCommandInput(
    command_id='RunShellScript',
    script=[
        'apt install python2 -y',
        'python3 --version'
    ]
)

install_python = compute_client.virtual_machines.begin_run_command(
    resource_group_name, vm_name, command
)
result = install_python.result()

Step 4: Format and Display Output

Process the output to make it readable by extracting the stdout and stderr results from the command response.

import json

output_messages = []
for message in result.value:
    stdout_start = message.message.find("[stdout]") + len("[stdout]\n")
    stderr_start = message.message.find("[stderr]") + len("[stderr]\n")

    stdout = message.message[stdout_start:stderr_start - len("[stderr]\n")].strip()
    stderr = message.message[stderr_start:].strip()

    output_message = {
        "code": message.code,
        "stdout": stdout,
        "stderr": stderr
    }
    output_messages.append(output_message)

print(json.dumps({"output": output_messages}, indent=4))

Step 5: Define and Configure the Linux Diagnostic Extension

The Linux Diagnostic Extension gathers performance and diagnostic data and stores it in your Azure storage account. Define settings for metrics, logs, and performance counters, and specify protected settings for secure storage account access.

from azure.mgmt.compute.models import VirtualMachineExtension

settings = {
    "StorageAccount": storageAccountName,
    "ladCfg": {
        "diagnosticMonitorConfiguration": {
            "eventVolume": "Medium",
            "metrics": { "metricAggregation": [{"scheduledTransferPeriod": "PT1M"}, {"scheduledTransferPeriod": "PT1H"}] },
            "syslogEvents": {
                "syslogEventConfiguration": {
                    "LOG_AUTH": "LOG_DEBUG",
                    # Add additional syslog configurations as needed
                }
            },
            "performanceCounters": {
                "performanceCounterConfiguration": [
                    {
                        "annotation": [{"displayName": "CPU IO wait time", "locale": "en-us"}],
                        "class": "processor",
                        "condition": "IsAggregate=TRUE",
                        "counter": "percentiowaittime",
                        "counterSpecifier": "/builtin/processor/percentiowaittime",
                        "type": "builtin",
                        "unit": "Percent",
                        "sampleRate": "PT15S"
                    },
                    # Add other counters as required
                ]
            }
        }
    }
}

protected_settings = {
    "storageAccountName": storageAccountName,
    "storageAccountSasToken": "your_storage_account_sas_token"
}

Step 6: Install the Linux Diagnostic Extension

Initialize the VirtualMachineExtension object and deploy it to your VM to enable monitoring.

vmExt = VirtualMachineExtension(
    location=location,
    publisher="Microsoft.Azure.Diagnostics",
    type_handler_version="3.0",
    type_properties_type="LinuxDiagnostic",
    auto_upgrade_minor_version=True,
    settings=settings,
    protected_settings=protected_settings
)

result = compute_client.virtual_machine_extensions.begin_create_or_update(
    resource_group_name=resource_group_name,
    vm_name=vm_name,
    vm_extension_name="LinuxDiagnostic",
    extension_parameters=vmExt,
    content_type="application/json"
).result()

print("Linux Diagnostic Extension installation completed.")

Step 7: List Installed Extensions

List and verify the extensions installed on your VM.

list_result = compute_client.virtual_machine_extensions.list(resource_group_name, vm_name)
for extension in list_result.value:
    print(f"Extension: {extension.name}, Publisher: {extension.publisher}, Type: {extension.type_properties_type}")

Full Code

from azure.identity import DefaultAzureCredential
from azure.mgmt.compute import ComputeManagementClient
from azure.mgmt.compute.models import RunCommandInput, VirtualMachineExtension
import json

# Set up environment-specific variables
subscription_id = "your_subscription_id"
resource_group_name = "your_resource_group"
vm_name = "your_vm_name"
location = "your_location"
storageAccountName = "your_storage_account_name"

# Authenticate and initialize the Compute Management client
credential = DefaultAzureCredential()
compute_client = ComputeManagementClient(credential, subscription_id)

# Run shell command on VM to install Python2 and check Python3 version
command = RunCommandInput(
    command_id='RunShellScript',
    script=[
        'apt install python2 -y',  # Install Python 2
        'python3 --version'        # Verify Python 3 version
    ]
)

# Execute the command on the specified VM and wait for completion
install_python = compute_client.virtual_machines.begin_run_command(
    resource_group_name, vm_name, command
)
result = install_python.result()

# Process and structure output messages for easy readability
output_messages = []
for message in result.value:
    stdout_start = message.message.find("[stdout]") + len("[stdout]\n")
    stderr_start = message.message.find("[stderr]") + len("[stderr]\n")

    stdout = message.message[stdout_start:stderr_start - len("[stderr]\n")].strip()
    stderr = message.message[stderr_start:].strip()

    output_message = {
        "code": message.code,
        "stdout": stdout,
        "stderr": stderr
    }
    output_messages.append(output_message)

# Print command execution result in JSON format
result_dict = {"output": output_messages}
print(json.dumps(result_dict, indent=4))

# Define Linux Diagnostic Extension settings for detailed monitoring
settings = {
  "StorageAccount": f"{storageAccountName}",
  "ladCfg": {
    "diagnosticMonitorConfiguration": {
      "eventVolume": "Medium",
      "metrics": {
        "metricAggregation": [
          { "scheduledTransferPeriod": "PT1M" },
          { "scheduledTransferPeriod": "PT1H" }
        ],
        "resourceId": f"/subscriptions/{subscription_id}/resourceGroups/{resource_group_name}/providers/Microsoft.Compute/virtualMachines/{vm_name}"
      },
      "syslogEvents": {
        "syslogEventConfiguration": {
          "LOG_AUTH": "LOG_DEBUG",
          "LOG_AUTHPRIV": "LOG_DEBUG",
          "LOG_CRON": "LOG_DEBUG",
          "LOG_DAEMON": "LOG_DEBUG",
          "LOG_FTP": "LOG_DEBUG",
          "LOG_KERN": "LOG_DEBUG",
          "LOG_LOCAL0": "LOG_DEBUG",
          "LOG_LOCAL1": "LOG_DEBUG",
          "LOG_LOCAL2": "LOG_DEBUG",
          "LOG_LOCAL3": "LOG_DEBUG",
          "LOG_LOCAL4": "LOG_DEBUG",
          "LOG_LOCAL5": "LOG_DEBUG",
          "LOG_LOCAL6": "LOG_DEBUG",
          "LOG_LOCAL7": "LOG_DEBUG",
          "LOG_LPR": "LOG_DEBUG",
          "LOG_MAIL": "LOG_DEBUG",
          "LOG_NEWS": "LOG_DEBUG",
          "LOG_SYSLOG": "LOG_DEBUG",
          "LOG_USER": "LOG_DEBUG",
          "LOG_UUCP": "LOG_DEBUG"
        }
      },
      "performanceCounters": {
        "performanceCounterConfiguration": [
          {
            "annotation": [{ "displayName": "CPU IO wait time", "locale": "en-us" }],
            "class": "processor",
            "condition": "IsAggregate=TRUE",
            "counter": "percentiowaittime",
            "counterSpecifier": "/builtin/processor/percentiowaittime",
            "type": "builtin",
            "unit": "Percent",
            "sampleRate": "PT15S"
          },
          {
            "annotation": [{ "displayName": "CPU user time", "locale": "en-us" }],
            "class": "processor",
            "condition": "IsAggregate=TRUE",
            "counter": "percentusertime",
            "counterSpecifier": "/builtin/processor/percentusertime",
            "type": "builtin",
            "unit": "Percent",
            "sampleRate": "PT15S"
          },
          {
            "annotation": [{ "displayName": "Memory available", "locale": "en-us" }],
            "class": "memory",
            "counter": "availablememory",
            "counterSpecifier": "/builtin/memory/availablememory",
            "type": "builtin",
            "unit": "Bytes",
            "sampleRate": "PT15S"
          },
          {
            "annotation": [{ "displayName": "Filesystem % used space", "locale": "en-us" }],
            "class": "filesystem",
            "condition": "IsAggregate=TRUE",
            "counter": "percentusedspace",
            "counterSpecifier": "/builtin/filesystem/percentusedspace",
            "type": "builtin",
            "unit": "Percent",
            "sampleRate": "PT15S"
          }
        ]
      }
    }
  }
}

# Protected settings include secure credentials for the storage account
protected_settings = {
    "storageAccountName": f"{storageAccountName}",
    "storageAccountSasToken": "sv=2022-11-02&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2025-01-10T19:32:12Z&st=2024-10-09T11:32:12Z&spr=https&sig=6Y3YoU8xnRoeHFAESvDroBw1q%2FGMORaXl9MSjCvwpkg%3D"
}

# Install and configure the Linux Diagnostic Extension on the VM
print("Installation for VirtualMachineExtension Started")
vmExt = VirtualMachineExtension(
    location=location,
    publisher="Microsoft.Azure.Diagnostics",
    type_handler_version="3.0",
    type_properties_type="LinuxDiagnostic",
    auto_upgrade_minor_version=True,
    settings=settings,
    protected_settings=protected_settings
)

result = compute_client.virtual_machine_extensions.begin_create_or_update(
    resource_group_name=resource_group_name,
    vm_name=vm_name,
    vm_extension_name="LinuxDiagnostic",
    extension_parameters=vmExt,
    content_type="application/json"
).result()

print(result)
print("Installation for VirtualMachineExtension Completed")

# List and display installed extensions for the VM
list_result = compute_client.virtual_machine_extensions.list(resource_group_name, vm_name)
for extension in list_result.value:
    print(f"Location: {extension.location}")
    print(f"Tags: {extension.tags}")
    print(f"Publisher: {extension.publisher}")
    print(f"Type: {extension.type_properties_type}")
    print(f"Version: {extension.type_handler_version}")
    print(f"Settings: {extension.settings}")
    print(f"Protected Settings: {extension.protected_settings}")
    print(f"Name: {extension.name}")

Conclusion

This tutorial covered running shell commands on an Azure VM and configuring the Linux Diagnostic Extension for detailed monitoring. These capabilities are essential for managing virtual machines in the cloud, providing real-time diagnostics and ensuring optimal performance.

By following this guide, you can integrate similar diagnostics into your own Azure environment, enabling better performance tracking and efficient VM management.