# How to set up AI-powered cloud cost monitoring: AWS, Azure, and GCP > AI Release · @ai_release1 · https://ai-release.org/guides/kak-nastroit-ii-monitoring-rashodov-na-oblako-aws-azure_en.html ## TL;DR - **Time to complete:** 45–60 minutes (three clouds, one budget per account, one daily digest). - **Difficulty:** Intermediate. - **You will need:** - AWS CLI v2, authenticated profile with billing access - Azure CLI (`az`) logged in to the target subscription - Google Cloud CLI (`gcloud`) authenticated - `jq` 1.6+ for JSON parsing - Admin rights on each billing account (AWS: `budgets:*` + `ce:*`; Azure: Cost Management Contributor; GCP: Billing Account Costs Manager) - A notification address: email alias or Slack/Telegram webhook - **Result:** monthly budgets, anomaly alerts, and a uniform daily cost report for AWS, Azure and GCP. Values in angle brackets (``, ``) are placeholders. Anything not stated here — API flag behaviour, quota limits, currency rules — **check with the provider documentation before rollout**. ## Requirements and setup ![Illustration: Requirements and setup](https://ai-release.org/guides/img/kak-nastroit-ii-monitoring-rashodov-na-oblako-aws-azure-en-1.jpg?v=1792778796) - [ ] Operating system: Linux or macOS shell (Windows: use WSL2 or PowerShell 7). The `date -d` syntax in Step 3 is GNU; on macOS install coreutils and use `gdate`. - [ ] Install the three CLIs and confirm they run. - [ ] Confirm you have billing read/write rights on each account, not just resource rights. - [ ] Decide one **tag key** used identically in all three clouds, e.g. `team`, so cost reports line up. AWS calls them tags, Azure tags, GCP labels — same idea: a key/value pair attached to a resource so you can group spend by owner or product. - [ ] Create a notification destination (email distribution list is the simplest). - [ ] Free disk space is irrelevant here; the only local artifact is a small JSON file for the AWS budget. **Example:** check identity and permissions before writing any budget. ```bash aws sts get-caller-identity az account show --query "{sub:name, id:id}" -o table gcloud config get-value project ``` Expected output: an AWS account number, a one-line Azure table with the subscription name, and a GCP project ID. If any command errors, stop and fix authentication first — a budget created under the wrong account is silently useless. **Why this works:** every later command reads credentials from the CLI's own credential chain; verifying identity up front turns a confusing `AccessDenied` later into a five-second fix now. ## Step 1. Define a shared tagging and budget schema **What to do:** agree on one tag key, one currency, and one budget period across all three clouds before creating anything. | Cloud | Where you set it | Example value | Grouping key used later | |---|---|---|---| | AWS | Cost allocation tags → activate `team` | `ml-platform` | `user:team$ml-platform` | | Azure | Resource tag `team` | `ml-platform` | tag filter in Cost analysis | | GCP | Resource label `team` | `ml-platform` | label in BigQuery export | Rules to pin down in writing: 1. Tag key: lowercase `team`, no spaces. 2. Budget period: `MONTHLY` everywhere — mixing weekly and monthly budgets makes the digest incomparable. 3. Budget amount per account: `` in the account's billing currency. 4. Alert thresholds: 50%, 80%, 100% of the budget. **Why:** cost reports only aggregate by tag *after* the tag is activated in billing. In AWS a tag on a resource is invisible to Cost Explorer until you activate it as a cost allocation tag in the Billing console (**Billing and Cost Management → Cost allocation tags**). Azure and GCP apply tags immediately, which makes AWS the usual source of "my report is empty" tickets. **How to check:** confirm at least one tagged resource appears in the cost grouping. ```bash aws ce get-cost-and-usage \ --time-period Start=$(date -u -d '7 days ago' +%Y-%m-%d),End=$(date -u +%Y-%m-%d) \ --granularity DAILY \ --metrics UnblendedCost \ --group-by Type=TAG,Key=team \ --output json | jq -r '.ResultsByTime[].Groups[] | "\(.Keys[0])\t\(.Metrics.UnblendedCost.Amount)"' ``` Expected: lines like `team$ml-platform 12.34`. An empty `Groups` array means the tag is not activated yet, or no spend occurred in that window. If your agents also call paid model APIs and you need hard caps per agent rather than per cloud account, the per-agent ceiling pattern is described in [How to Set Spending Limits and Budgets for AI Agents](https://ai-release.org/guides/kak-nastroit-limity-rashodov-i-byudzhety-dlya-ii-agento.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide) — run both layers, cloud budgets and agent budgets, because cloud budgets see infrastructure while agent budgets see tokens. ## Step 2. Deploy native budgets and anomaly detection in AWS, Azure and GCP ![Illustration: Step 2. Deploy native budgets and anomaly detection in AWS, Azure and GCP](https://ai-release.org/guides/img/kak-nastroit-ii-monitoring-rashodov-na-oblako-aws-azure-en-2.jpg?v=1792778802) **What to do:** create one budget per cloud with percentage thresholds, plus an anomaly detector where the provider offers one. AWS and Azure support anomaly detection natively; GCP budgets are threshold-based only. ### AWS: budget file plus Cost Anomaly Detection Create `budget.json`: ```json { "BudgetName": "ai-monthly-budget", "BudgetLimit": { "Amount": "", "Unit": "USD" }, "TimeUnit": "MONTHLY", "BudgetType": "COST", "CostFilters": { "TagKeyValue": ["user:team$ml-platform"] }, "CostTypes": { "IncludeCredit": false, "IncludeRefund": false } } ``` Create `notifications.json`: ```json [ { "Notification": { "NotificationType": "ACTUAL", "ComparisonOperator": "GREATER_THAN", "Threshold": 80, "ThresholdType": "PERCENTAGE" }, "Subscribers": [ { "SubscriptionType": "EMAIL", "Address": "finops@example.com" } ] } ] ``` Apply both: ```bash aws budgets create-budget \ --account-id \ --budget file://budget.json \ --notifications-with-subscribers file://notifications.json ``` Then add an anomaly monitor scoped to service level — this catches a single runaway service before the monthly budget threshold is hit: ```bash aws ce create-anomaly-monitor --anomaly-monitor \ '{"MonitorName":"ai-services","MonitorType":"DIMENSIONAL","MonitorDimension":"SERVICE"}' aws ce create-anomaly-subscription --anomaly-subscription \ '{"SubscriptionName":"ai-anomalies","MonitorArnList":[""],"Subscribers":[{"Type":"EMAIL","Address":"finops@example.com"}],"Threshold":100,"Frequency":"DAILY"}' ``` The `Threshold` here is an absolute currency amount, not a percentage — the monitor fires when impact exceeds it. ### Azure: subscription budget via CLI ```bash az consumption budget create \ --budget-name ai-monthly-budget \ --amount \ --category cost \ --time-grain Monthly \ --start-date \ --end-date \ --notifications '[{"enabled":true,"operator":"GreaterThan","threshold":80,"contactEmails":["finops@example.com"]}]' ``` If your CLI build rejects `--notifications`, create the budget with the command minus that flag, then add the alert in **Cost Management → Budgets → ai-monthly-budget → Alert conditions**. Flag availability varies by CLI release — **check with the documentation**. ### GCP: budget with a Pub/Sub topic for programmatic alerts ```bash gcloud pubsub topics create budget-alerts gcloud billing budgets create \ --billing-account= \ --display-name="ai-monthly-budget" \ --budget-amount= \ --threshold-rule=percent=0.5 \ --threshold-rule=percent=0.9 \ --threshold-rule=percent=1.0 \ --all-updates-rule-pubsub-topic=projects//topics/budget-alerts ``` **Why:** all three providers evaluate budgets against *actual* spend for the current period and can also warn on *forecast* spend. Routing GCP alerts to Pub/Sub instead of email lets you forward them into the same channel as the AWS and Azure alerts, so a single on-call rotation sees all three clouds. **How to check:** list what you created. ```bash aws budgets describe-budgets --account-id --output table az consumption budget list --query "[].{name:name, amount:amount}" -o table gcloud billing budgets list --billing-account= ``` Expected: `ai-monthly-budget` in each output with your amount. For AWS also run `aws ce get-anomaly-monitors` and confirm `ai-services` has `"Status": "ACTIVE"`; a monitor created in the last 24 hours shows limited history and will not alert until it has baseline data. ## Step 3. Verify alerts and ship a daily cost digest ![Illustration: Step 3. Verify alerts and ship a daily cost digest](https://ai-release.org/guides/img/kak-nastroit-ii-monitoring-rashodov-na-oblako-aws-azure-en-3.jpg?v=1792778806) **What to do:** prove the alert path end-to-end, then automate a daily top-services report. 1. Confirm the email subscriber in AWS is confirmed — an unconfirmed SNS email subscription swallows every alert. 2. Publish a test message to the GCP topic and confirm it reaches your forwarder. 3. Install the digest script below as a cron job (`0 7 * * *`). ```bash #!/usr/bin/env bash set -euo pipefail START=$(date -u -d 'yesterday' +%Y-%m-%d) END=$(date -u +%Y-%m-%d) aws ce get-cost-and-usage \ --time-period Start="$START",End="$END" \ --granularity DAILY \ --metrics UnblendedCost \ --group-by Type=DIMENSION,Key=SERVICE \ --output json \ | jq -r '.ResultsByTime[].Groups[] | "\(.Keys[0])\t\(.Metrics.UnblendedCost.Amount)"' \ | sort -k2 -nr | head -5 ``` **Example output** for a single day, sorted by spend: ``` Amazon SageMaker 41.87 Amazon Elastic Compute Cloud - Compute 22.10 Amazon Bedrock 9.42 Amazon Simple Storage Service 1.03 AWS Key Management Service 0.14 ``` The `Unit` field is returned per group; check it once with `--output json | jq '.ResultsByTime[0].Groups[0].Metrics'` so you are not comparing USD against another currency. **Why:** a monthly budget tells you that you overspent; a daily top-five list tells you *which service* started the overspend, usually a day earlier. Shipping that list to a chat channel closes the loop — the same pattern used for server monitoring in [How to Set Up VPS Monitoring via a Telegram Bot with Mini App](https://ai-release.org/guides/kak-nastroit-monitoring-vps-cherez-telegram-bota-s-mini.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide), where the bot just forwards a formatted daily summary and a small web UI for history. **How to check:** run the script manually and compare its total against the console. ```bash aws ce get-cost-and-usage \ --time-period Start="$START",End="$END" \ --granularity DAILY --metrics UnblendedCost \ --output json | jq '[.ResultsByTime[].Total.UnblendedCost.Amount | tonumber] | add' ``` Expected: a single number matching the sum of your top-five lines plus everything below the cutoff. If the console shows a different figure, the usual cause is a different date range or a currency setting — both are visible in the Cost Explorer header. ## Troubleshooting **Problem. AWS refuses to create the budget from the CLI.** - Error: `An error occurred (AccessDeniedException) when calling the CreateBudget operation: User: arn:aws:iam:::user/ is not authorized to perform: budgets:CreateBudget` - Cause: the IAM principal can read cost data but lacks the `budgets:CreateBudget` action, which lives in the AWS Budgets service, not in Cost Explorer. - Solution: attach a policy granting `budgets:CreateBudget`, `budgets:ModifyBudget` and `budgets:ViewBudget`, then re-run: ```bash aws iam attach-user-policy --user-name --policy-arn arn:aws:iam::aws:policy/AWSBudgetsActionsWithAWSResourceControlAccess ``` Verify with `aws budgets describe-budgets --account-id `. **Problem. Azure budget creation fails with an authorization error.** - Error: `(InsufficientPermissions) The client '' with object id '' does not have authorization to perform action 'Microsoft.Consumption/budgets/write' over scope '/subscriptions/'` - Cause: the service principal is a Reader on the subscription. Budgets are written through the Consumption resource provider, which needs a write role. - Solution: assign Cost Management Contributor at subscription scope and retry: ```bash az role assignment create \ --assignee \ --role "Cost Management Contributor" \ --scope /subscriptions/ ``` The role assignment can take a few minutes to propagate; `az consumption budget list` returning an empty list after a successful command usually means propagation is still in flight. **Problem. AWS budget alerts never arrive.** - Error: `SubscriptionARN ... is pending confirmation` in `aws sns list-subscriptions-by-topic --topic-arn ` output. - Cause: an email subscriber must click the confirmation link SNS sends; until then the subscription stays `PendingConfirmation` and every budget notification is dropped silently. - Solution: find the confirmation email from `no-reply@sns.amazonaws.com` and confirm it, or re-create the subscription with an address a human monitors: ```bash aws sns subscribe \ --topic-arn \ --protocol email \ --notification-endpoint finops@example.com ``` Re-check with `aws sns list-subscriptions-by-topic --topic-arn `; the status must read `SubscriptionArn` with a full ARN, not `PendingConfirmation`. ## Summary and next steps You now have a monthly budget with percentage thresholds in AWS, Azure and GCP, an AWS anomaly monitor for per-service spikes, and a daily top-five cost digest you can route into chat. Next, activate the `team` tag as a cost allocation tag in every account and add a second budget per team so the digest can be split by owner. ## FAQ **How do I monitor cloud costs per team instead of per account?** Attach the same tag key (`team`) to resources in all three clouds, activate it as a cost allocation tag in AWS Billing, then group by `Type=TAG,Key=team` in Cost Explorer, by tag in Azure Cost analysis, and by label in the GCP BigQuery billing export. **Why is my AWS budget always showing zero spend?** Usually the `CostFilters` tag key is not activated for cost allocation, or the tag value in `budget.json` does not match the resource exactly (`user:team$ml-platform` is case-sensitive). Verify with `aws ce get-cost-and-usage --group-by Type=TAG,Key=team`. **Do budget alerts cost anything?** AWS Budgets allows a limited number of budgets and alert evaluations per account at no charge, with additional ones billed — the current allowance and pricing **check with the AWS documentation**. Azure budgets and GCP budgets are configuration objects on the billing account; GCP charges for the Pub/Sub messages and BigQuery storage the alerts produce. **Can I get alerts in Slack or Telegram instead of email?** Yes for GCP: point `--all-updates-rule-pubsub-topic` at a topic with a push subscription to your handler. For AWS, add a Lambda subscriber to the budget's SNS topic. For Azure, forward action-group notifications to a webhook. The formatting and forwarding pattern is the same one used in the Telegram monitoring bot described above.