AI-ENG-AE — Sustainable AI - Energy, Infrastructure Efficiency & Lifecycle Impact

Conceptual Glossary

Sustainable AI architecture demands a rigorous, standardized lexicon to transition environmental metrics from abstract environmental, social, and governance (ESG) reporting into active runtime engineering variables. The following table establishes the operational definitions utilized throughout this architectural doctrine.

Sustainable AI Architecture Doctrine

The rapid expansion of generative AI workloads has introduced a fundamental systemic challenge: the rate of digital infrastructure expansion far outpaces the physical capacity of global energy and utility grids.27 Traditional application performance monitoring focuses on a binary status model, assuming that any transaction returning an HTTP 200 payload represents a healthy system.29 In high-dimensional, non-deterministic AI systems, this perspective is an architectural failure.29 A system can report a green operational dashboard while simultaneously executing multi-megawatt training runs that yield zero downstream business value, or executing highly verbose, repetitive reasoning steps that deplete limited water reserves and overload regional electrical substations.17
The governing doctrine of sustainable AI architecture asserts that sustainability is a first-class engineering constraint, co-equal with latency, quality, cost, security, and governance.29 A system that achieves optimal task accuracy at the cost of unconstrained compute consumption, excessive context bloat, and unchecked hardware degradation is a poorly designed architecture. Historically, model selection has been treated as a prestige marker, where teams deploy the largest available frontier models to solve basic classification, data transformation, or informational lookup tasks.32 Sustainable architecture rejects this paradigm. The correct architecture is the smallest, most efficient, sufficiently capable model and routing topology that satisfies the validated criteria of the task.
Maximizing efficiency requires analyzing the entire physical chain of computation. Every prompt executed on a client device initiates a physical cascade: a local processor packages the request; network switches route the packets across regional grids; a data center substation steps down high-voltage power to run specialized silicon; cooling systems evaporate water or run high-power compressors to extract heat; and global supply chains fabricate the accelerators that will eventually decay into e-waste.7 To optimize this system, architects must treat energy, carbon, and hardware degradation as hard constraints within which all software, routing, and quantization decisions must be negotiated. The greenest token is the one the system did not need to generate.

The Macroeconomic Resource Challenge

The expansion of generative AI workloads has transformed data centers from passive, predictable consumers of commodity electricity into volatile, resource-dense industrial nodes. To contextually frame this shift, the physical parameters of data center energy demand, grid integration, water consumption, and thermal dissipation must be quantified.

Global and Regional Grid Projections

Global data-center electricity demand is rising quickly, but projections vary by scope, methodology, hardware assumptions, utilization assumptions, and whether cryptocurrency or broader digital infrastructure is included. Sustainable AI architecture should therefore avoid treating a single forecast as destiny. The correct planning posture is to maintain a baseline, a near-term estimate, a 2030 base case, and high/sensitivity cases.

In the United States, Europe, China, Ireland, Singapore, Northern Virginia, Frankfurt, Dublin, and other dense data-center markets, the operational issue is not merely aggregate global electricity share. It is localized concentration. A workload that looks small in global percentage terms can still create major interconnection delays, utility-rate pressure, water-basin stress, or local reliability risk when clustered in constrained regions.

The architectural conclusion is straightforward: sustainable AI cannot depend on vague global averages. It must make deployment decisions using region-specific electricity, water, cooling, carbon, and capacity constraints.

Power Density and Dynamic Grid Interconnection

The physical characteristics of AI compute clusters impose severe stresses on grid infrastructure. Traditional server racks operate at predictable power densities of 5 kW to 15 kW. By contrast, advanced AI server racks packed with modern accelerator blocks feature power densities that increased 11-fold between 2020 and 2025, with an additional fourfold increase projected by 2027.30 A single advanced server rack can have a peak power demand equivalent to that of 65 average households.30
Furthermore, AI workloads exhibit extreme load variability. While traditional enterprise workloads present smooth, diurnal load profiles, AI data centers experience repeated swings of server load exceeding 50% of rated capacity within a single second.30 These rapid step-changes inject significant frequency instability into the local transmission grid. To mitigate the risk of voltage sag or grid collapse, operators are forced to either overbuild dedicated onsite gas-fired backup generation by 30% to 70% relative to average demand, or contract with private producers to install multiple, highly inefficient natural gas reciprocating engines directly at the data center boundary.28

Hydrological and Thermal Footprints

AI infrastructure has a water and thermal footprint as well as an electricity footprint. Water use varies sharply by region, cooling design, facility efficiency, weather, time of day, and electricity-generation mix. A single universal “water per prompt” number is therefore misleading. Published GPT-3-style estimates suggest roughly one 500 ml bottle of water for about 10 to 50 medium-length responses, depending on when and where inference is served. That figure should be treated as an order-of-magnitude estimate, not a universal constant.

Water consumption has three main channels:

Thermal stress is similarly local. Dense AI clusters convert nearly all consumed electrical energy into heat that must be removed from racks, rooms, and facilities. High-density accelerator racks may require liquid cooling, rear-door heat exchangers, or facility-level cooling redesign. In hot or water-stressed regions, the environmental tradeoff can become acute: reducing electricity use, reducing water use, and maintaining hardware reliability may require different operating points.

The practical engineering rule is that water and thermal metrics must be attached to deployment decisions:

Sustainable AI architecture should therefore report water estimates with scope, region, method, and confidence level. “Water per response” is useful only when it is tied to a specific model class, facility type, electricity mix, cooling design, request size, and measurement method.

Energy, Carbon, and Water Cost Model

Sustainable AI requires unit-explicit accounting. Energy, carbon, water, and embodied hardware impact should be modeled as related but distinct quantities. A useful model must separate measured energy from allocated estimates, operational emissions from embodied emissions, and successful work from failed or reworked computation.

Unit-Explicit Operational Energy

For a workload executed over time window T, the system energy is:

E_system_kWh =
  sum over t in T [
    (P_compute_kW(t)
   + P_memory_kW(t)
   + P_storage_kW(t)
   + P_network_kW(t)
   + P_cooling_allocated_kW(t)) * delta_t_hours
  ]

Where:

For most production systems, not every component is directly measured per request. The architecture should label each value as measured, estimated, allocated, or vendor-reported.

Operational Carbon

Operational carbon is calculated by multiplying electricity consumption by grid carbon intensity:

CO2e_operational_kg =
  sum over t in T [
    E_system_kWh(t) * CI_grid_kgCO2e_per_kWh(g, t)
  ]

For scheduling and workload-shifting decisions, marginal carbon intensity is usually the more appropriate control signal when available. Average grid intensity may be useful for reporting but can understate or misrepresent the impact of new incremental load.

Embodied Carbon Allocation

Embodied carbon should not be added wholesale to every request. It must be allocated across the hardware’s useful life according to a declared allocation method:

CO2e_embodied_allocated_kg =
  CO2e_hardware_lifecycle_kg
  * workload_allocation_share
  * accounting_window_share

Possible allocation bases include:

Allocation method must be documented. Otherwise embodied-carbon accounting becomes spreadsheet alchemy with a leaf icon.

Verified-Task Efficiency

A successful AI task is not merely a completed model call. It is an accepted, policy-compliant, verified outcome. Instead of dividing by a binary success variable and creating infinite dashboard values, the system tracks verified work and waste separately.

E_per_verified_task_kWh =
  E_total_accepted_tasks_kWh / count(verified_successful_tasks)

CO2e_per_verified_task_kg =
  (CO2e_operational_accepted_kg + CO2e_embodied_allocated_kg)
  / count(verified_successful_tasks)

Failed or rejected work is tracked as waste:

E_waste_kWh =
  E_failed_generations
+ E_retries
+ E_policy_rejections
+ E_unverified_tool_attempts
+ E_rework

Waste_ratio =
  E_waste_kWh / E_total_kWh

This makes brittle systems visible. A tiny model that fails repeatedly, triggers retries, or requires human cleanup may be less sustainable than a larger model that solves the task correctly once.

Water Accounting

Water use should be modeled as direct facility water plus indirect electricity water plus allocated supply-chain water where available:

Water_direct_liters =
  E_IT_kWh * WUE_facility_liters_per_kWh

Water_indirect_liters =
  E_grid_kWh * WI_grid_liters_per_kWh(g, t)

Water_task_liters =
  Water_direct_allocated
+ Water_indirect_allocated
+ Water_supply_chain_allocated

Where:

Because water estimates vary dramatically by region and cooling design, water metrics must include location, measurement method, and confidence level.

Metric Families

Model Sizing Matrix

Model size is an architectural decision, not a prestige marker. The correct route is the smallest sufficiently capable execution path that satisfies quality, safety, latency, privacy, and governance constraints. The following matrix is indicative rather than universal: energy use varies by hardware, batching, context length, output length, quantization, cache state, compiler backend, and utilization.

The matrix should be implemented as a routing policy, not a static spreadsheet shrine. Each route must be validated on production-like traffic and periodically reviewed as models, hardware, costs, and environmental conditions change.

Energy-Aware Routing Policy

A static, single-model API gateway is an architectural anti-pattern for sustainable AI. The routing layer should match each request to the least resource-intensive path that still satisfies quality, safety, privacy, latency, and governance requirements. Sustainability never overrides hard safety or access-control boundaries; it chooses among safe and valid execution paths.

ENERGY-AWARE ROUTING FLOW

[ Incoming Request ]
        |
        v
[ Classify Task ]
  deterministic | lookup | extraction | RAG | reasoning | high-impact
        |
        v
[ Check Hard Floors ]
  safety | privacy | tenant scope | policy | data residency | tool authority
        |
        v
[ Estimate Route Cost ]
  energy | carbon | water | latency | money | rework risk
        |
        v
[ Select Smallest Sufficient Route ]
        |
        +--> deterministic route
        +--> small-model route
        +--> specialist route
        +--> cascade route
        +--> large-model route
        +--> local route
        +--> cloud route
        +--> batch/offline route
        +--> fail-closed/degraded route
        |
        v
[ Verify Outcome ]
        |
        v
[ Log Routing Decision According to Retention Policy ]

Route Modes

Routing decisions should record route ID, model/profile, task class, policy version, estimated energy/carbon/water method, and verification outcome according to retention and audit policy. Do not retain sustainability telemetry forever by default; eternal logs become their own zombie workload.

Efficient Inference Playbook

Efficient inference is achieved by combining architecture, compiler, memory, routing, cache, context, and output controls. No optimization is universally safe. Each must be evaluated against task quality, policy compliance, data isolation, and downstream failure rate.

The efficiency objective is not maximum tokens per second. It is minimum resource use per verified, policy-compliant task.

Context and Retrieval Efficiency Model

Retrieval-Augmented Generation (RAG) is a highly effective pattern for grounding model responses, but naive implementations can lead to significant resource waste. Sending large, unranked document chunks into a prompt to answer a question that requires only a single sentence is a primary source of context inflation and energy waste.42

The Context Bloat Cascade

When a system retrieves ten documents of 400 tokens each to answer a query, it forces the model to process 4,000 context tokens.42 If the actual answer is buried in a single sentence in the sixth document, several systemic failures occur:

• Prefill Energy Penalty: The model pays a major computational tax to execute self-attention over the entire 4,000-token input.41
• Decoding Memory Pressure: The size of the KV cache increases linearly with context length, restricting batch size on the accelerator and driving down hardware efficiency.13
• Attention Degradation: Language models demonstrate “lost in the middle” phenomena, where they reliably process information at the absolute beginning and end of the prompt but fail to track facts buried in the center of a long context window.42
• Financial Waste: The enterprise pays per input token, meaning that over-retrieval results in direct financial inflation on every single execution.42

Retrieval and Context Metrics

To enforce efficiency across retrieval architectures, teams must integrate five core metrics into their observability pipelines:

Retrieved-Token Usefulness Ratio (R_useful)

Measures the fraction of retrieved and prompt-inserted tokens that actually contribute to the validated output response, defined as:

R_useful = Tokens in Context Containing Gold Facts / Total Retrieved Tokens in Prompt

Low usefulness ratios (less than 0.1) indicate poorly sized document chunks or the absence of semantic rerankers.42

Duplicate Context Ratio (R_dup)

Quantifies the volume of redundant semantic information present within the retrieved context set:

R_dup = Redundant Sentences or Duplicate Semantic Chunks / Total Sentences in Context Set

High redundancy indicates a lack of deduplication filters or clustering steps in the retrieval pipeline.38

Stale Context Ratio (R_stale)

Tracks the portion of the retrieved document set that has been updated or deprecated in the source system of record but remains active in the vector database index:

R_stale = Stale or Deprecated Chunks Retrieved / Total Retrieved Chunks

This metric measures index maintenance efficiency, identifying systemic energy waste in continuous indexing pipelines.26

Grounding Gain per Token (G_token)

Calculates the marginal improvement in output correctness (measured by automated factual consistency scores) per additional token of context inserted into the prompt:

G_token = (Accuracy_grounded - Accuracy_zero_shot) / Context Tokens

This curve helps identify the point of diminishing returns, where adding more context documents fails to improve output quality while continuing to inflate energy consumption.

Context Tokens per Successful Answer (C_task_tokens)

The absolute number of prompt context tokens processed to deliver a verified, policy-compliant user response:

C_task_tokens = Total Context Tokens processed in Run / S_task

Hardware Utilization Model

Maximizing the energy efficiency of an AI deployment requires a deep understanding of accelerator physics. The primary source of resource waste in modern enterprise AI is not the power consumed by active computation, but the static energy wasted by underutilized, idle accelerators.10

The Accelerator Power Curve

Modern AI accelerators exhibit highly non-linear power curves. An advanced GPU or TPU baseboard draws a substantial amount of power when idling—frequently 30% to 50% of its maximum thermal design power (TDP)—even when executing zero floating-point operations.10 For example, a single high-performance NVIDIA HGX H100 GPU baseboard features a typical power consumption of 5,600 W.8 While a single H100 GPU has a TDP of approximately 700 W (compared to 400 W for the previous-generation A100), it continues to draw nearly 300 W of static power while idling.9
Consequently, a cluster of accelerators running at 15% average utilization (which represents the baseline average for most enterprise deployments 32) consumes nearly the same baseline operational energy as a cluster running at 70% utilization, while delivering only a fraction of the useful output.10 This represents an immense waste of both operational energy and amortized embodied carbon.

Instantaneous Power (Watts)  
  ▲  
  │                       ┌─────────────────────────  ◄── Maximum TDP (700 Watts)   
  │                     ┌─┘  
  │                   ┌─┘  
  │                 ┌─┘  
  │               ┌─┘  
  │             ┌─┘  
  │  ───────────┘  ◄──────────────────────────────────  Idle Draw (300 Watts)   
  │  │  
  └──┴──────────┬───────────────────────────┬────────►  
     0%        15% (Typical Enterprise)    70%+ (Target)  
                Accelerator Utilization (%) 

To eliminate this waste, platform teams must monitor and optimize several hardware-level telemetry points:

• Accelerator Memory and Bandwidth Utilization: Captures the fraction of high-bandwidth memory (HBM) capacity occupied by active model weights and KV cache, and tracks the active memory bus traffic.13
• Active Queue Depth and Batch Occupancy: Monitors the number of queued execution requests awaiting processing. If queue depth is consistently zero while accelerators are kept online, the cluster is over-provisioned.32
• Cold Replica Fragmentation: The presence of warm, idle model replicas running across multiple isolated server nodes, each drawing static idle power rather than consolidating traffic onto a single node.
• Accelerator Power Caps and Thermal Throttling: Restricts the maximum allowable power draw of individual GPUs (e.g., capping an H100 at 550 W instead of 700 W). This technique achieves a substantial reduction in energy-per-token with minimal impact on latency.
• Cooling Infrastructure Overhead (PUE/WUE): The localized data center power and water consumed to dissipate heat generated by the active accelerators.15 Less-efficient enterprise data centers feature cooling overheads exceeding 30% of total energy draw, compared to less than 7% in highly optimized hyperscale facilities.34

Energy-Aware Deployment Profiles

To operationalize sustainable architecture across diverse enterprise application scenarios, engineers must implement structured deployment profiles. These profiles define the exact configuration knobs, target architectures, and optimization strategies appropriate for specific operational tolerances.

Carbon-Aware Scheduling Model

Carbon-aware scheduling shifts flexible AI workloads to cleaner, cooler, cheaper, or less constrained execution windows. It is most useful for non-urgent jobs such as offline evaluations, batch embeddings, vector index rebuilds, document parsing, pretraining, fine-tuning, and synthetic data generation.

Live user interactions, security response, critical transaction routing, and legally time-bound workflows usually cannot be delayed for environmental optimization. Sustainability is a scheduling objective only inside the envelope permitted by safety, policy, deadline, and user commitments.

Scheduling Candidates and Exclusions

Scheduling Decision Flow

CARBON-AWARE SCHEDULING FLOW

[ Workload Submitted ]
        |
        v
[ Classify Flexibility ]
  immediate | deadline-bound | flexible | opportunistic
        |
        v
[ Check Hard Constraints ]
  data residency
  privacy / tenant scope
  legal deadline
  freshness SLO
  security urgency
  capacity reservation
        |
        v
[ Estimate Execution Windows ]
  grid carbon intensity
  marginal carbon signal
  regional water stress
  ambient temperature / cooling load
  electricity price
  queue backlog
  accelerator availability
        |
        v
[ Optimize ]
  minimize carbon + water + cost
  subject to deadline + policy + quality + fairness
        |
        +--> execute now
        +--> queue until cleaner window
        +--> shift to approved region
        +--> split job across windows
        +--> reject or escalate if constraints conflict

Scheduling Constraints

Scheduler Outputs

Carbon-aware scheduling should be evaluated by measured or estimated avoided emissions, missed-deadline rate, queue fairness, cost impact, and user/business outcome quality.

Local vs Cloud Decision Matrix

The local-versus-cloud decision is not a simple sustainability contest. Local execution can reduce network dependency and preserve privacy, but may run on inefficient, underutilized, poorly measured hardware. Cloud execution can improve utilization and observability, but may concentrate electricity, water, heat, and grid stress in constrained regions. Sustainable architecture requires a full lifecycle comparison.

The sustainable choice is the route with the best verified outcome per lifecycle impact under the system’s privacy, safety, latency, governance, and regional constraints.

Sustainability Telemetry Plane

Sustainability metrics must move from annual reports into operational telemetry, but they should be modeled correctly. W3C trace context should carry correlation identifiers, not every environmental measurement. Sustainability telemetry should join traces, span attributes, resource metrics, facility metrics, grid signals, and allocation models.

SUSTAINABILITY TELEMETRY ARCHITECTURE

[ Request Trace ]
  trace_id | span_id | route_id | tenant/workload class
        |
        v
[ AI Span Attributes ]
  model route | prompt tokens | output tokens | cache hits
  batch size | tool calls | verifier outcome | task status
        |
        v
[ Hardware Metrics Stream ]
  accelerator power | utilization | memory | temperature
  queue depth | batch occupancy | replica count
        |
        v
[ Facility / Region Metrics ]
  PUE | WUE | cooling mode | region | grid carbon intensity
  water stress | electricity price | capacity state
        |
        v
[ Allocation Layer ]
  joins request traces to metric windows
  estimates energy, carbon, water, embodied allocation
        |
        v
[ Control Plane ]
  routing | scheduling | autoscaling | scale-to-zero | procurement review

Trace and Span Schema

The request trace should record identifiers and AI execution metadata:

{
  "trace_id": "ot-9821a-bf091",
  "span_id": "span-0918a",
  "ai.route_id": "cascade.route.triage",
  "ai.model_profile": "small-specialist-fp8",
  "ai.hardware_class": "accelerator.shared.h100",
  "ai.region": "us-east-1",
  "ai.prompt_tokens": 1042,
  "ai.completion_tokens": 412,
  "ai.cached_prefix_tokens": 512,
  "ai.dynamic_batch_size": 64,
  "ai.verifier_status": "accepted",
  "ai.task_status": "verified_success"
}

Power, grid carbon, PUE, WUE, and water stress should generally be metric streams or resource attributes correlated by time, region, hardware pool, and route. They should not be stuffed into trace headers.

Metric Sources

Derived Sustainability Metrics

Production systems should use sampling, profiling, and allocation where direct measurement is too costly. Sustainability telemetry that costs more to collect than it saves is just another tiny ceremonial bonfire.

Lifecycle Impact Model

A sustainable AI strategy must account for hardware lifecycle impact, not merely operational electricity. Accelerators, high-bandwidth memory, networking, storage, cooling equipment, substations, concrete, steel, and eventual e-waste all contribute to the system footprint.

Lifecycle numbers should be labeled by evidence confidence:

Amortized Embodied Carbon of Accelerators

High-performance AI accelerators have substantial cradle-to-gate emissions from semiconductor fabrication, HBM, integrated circuits, substrates, power delivery, cooling assemblies, and assembly. NVIDIA’s published HGX H100 Product Carbon Footprint summary reports 1,312 kg CO2e cradle-to-gate for one HGX H100 GPU baseboard, with materials and components dominating the footprint.

HGX H100 Baseboard PCF Example

Cradle-to-gate footprint:
  1,312 kg CO2e per HGX H100 baseboard

Major contributors:
  materials and components: 91%
    high-bandwidth memory: 42%
    integrated circuits:   25%
    thermals/cooling:      18%
  assembly:                8.6%
  transport/logistics:     0.4%

Evidence class:
  vendor PCF summary

These figures should not be blindly copied into all deployments. They should be used as a documented input to workload allocation, procurement comparison, and hardware refresh analysis.

Generational Efficiency and Replacement Tradeoffs

Newer hardware may reduce energy per task or embodied carbon intensity, but replacing hardware too aggressively can increase e-waste and strand embodied emissions. A hardware upgrade is sustainable only when the lifecycle benefit exceeds the lifecycle cost under expected utilization.

Decommissioning, E-Waste, and Zombie Infrastructure

Sustainable AI must manage retirement as deliberately as deployment. Idle accelerators, obsolete vector indexes, persistent debug traces, duplicated datasets, abandoned notebooks, and unused model replicas consume storage, power, cooling, administrative attention, and sometimes license cost.

Lifecycle Review Triggers

Sustainability Governance Checklist

Sustainability governance should be attached to AI system inventory, procurement, release gates, and operational review. Controls should be risk-tiered. A low-volume internal assistant does not need the same evidence burden as a high-volume public model route or a dedicated accelerator fleet.

System Inventory Integration

Every AI system registration should include an environmental profile proportional to its expected scale and risk.

Procurement and Vendor Audit Controls

Vendor sustainability requirements should be tiered by volume, criticality, data sensitivity, and integration depth.

Release Gate Questions

Before production launch or major route change, the system owner should answer:

Sustainability Theater Diagnostic

Many enterprise Green AI programs suffer from “sustainability theater”—the practice of deploying decorative dashboards and publishing leaf-branded marketing summaries that fail to drive real environmental or architectural change. Platform teams must run this diagnostic to identify and eliminate standard theater anti-patterns.

Diagnostic Check 1: The Green Dashboard Fallacy

• The Theater Pattern: Visualizing tree-planting graphs or displaying average grid-carbon metrics on SRE dashboards, while continuing to route trivial lookup tasks to unquantized, 400B-parameter frontier models running on fossil-heavy grids.13
• Diagnostic Check: Ask: Does the telemetry dashboard programmatically change gateway routing tables, alter queue scheduling, or trigger auto-scaling scale-to-zero routines? If the answer is no, the dashboard is theater.

Diagnostic Check 2: The Offset Illusion

• The Theater Pattern: Claims of “carbon neutrality” or “zero-emissions execution” based on the purchase of annual Renewable Energy Certificates (RECs) or speculative carbon offset credits, while operating physical servers on grids dominated by coal and gas generation.1
• Diagnostic Check: Ask: Is the workload’s electrical consumption actively matched hour-by-hour with local, 24/7 carbon-free energy (CFE) on the specific grid where the computation occurs? 3 If the matching is calculated annually or globally, the claim is ungrounded.

Diagnostic Check 3: Generative and Traditional Blurring

• The Theater Pattern: Publishing reports that highlight massive emissions reductions achieved by deploying AI models (e.g., claiming that AI reduced data center cooling energy by 15%), while conflating highly efficient predictive/traditional ML with highly resource-intensive generative LLMs.22
• Diagnostic Check: Ask: Does corporate carbon reporting explicitly isolate generative AI training and inference volumes from traditional machine learning tasks? 22

Diagnostic Check 4: The Tiny Model Trap

• The Theater Pattern: Forcing the deployment of highly quantized, extremely small models (e.g., 1B parameters) on tasks that require complex, multi-step logical reasoning to meet “green computing” targets.
• Diagnostic Check: Ask: What is the downstream failure rate? If the tiny model repeatedly hallucinates, outputting syntactically invalid structured schemas or incorrect logic that forces the system to initiate multiple retry loops or forces human engineers to spend hours manually correcting errors, the tiny model is less sustainable than a larger model that solves the task correctly on the first forward pass.37

Diagnostic Check 5: The Zombie Workload Swarm

• The Theater Pattern: Hundreds of forgotten development notebooks, under-utilized model replicas, obsolete vector indices, and redundant dataset copies sitting warm on high-cost cloud resources, while the platform team focuses on micro-optimizing prompt lengths.26
• Diagnostic Check: Ask: Are there active, automated network and traffic sweeps to identify any server or container running with near-zero CPU or GPU utilization for over 48 hours, programmatically shutting down these “zombie” environments? 26

Sustainable AI Decision Record Template

Every major model selection, regional deployment, hardware procurement, or routing configuration should be documented using a Sustainable AI Architecture Decision Record. The goal is to make sustainability tradeoffs explicit before production deployment.

# Architectural Decision Record: SUST-AI-<ID>

## Title

## Status
Proposed | Accepted | Superseded | Retired

## Context and Problem Statement
- Business requirement:
- User population:
- Expected workload volume:
- Input profile:
- Output profile:
- Latency / availability SLO:
- Quality / safety / governance floor:
- Current baseline route or architecture:

## Scope Boundary
Mark each metric as included, excluded, or unknown.

| Boundary Item | Included? | Measurement Method | Confidence |
| :---- | :---- | :---- | :---- |
| Model inference energy | Yes / No / Unknown | measured / estimated / allocated / vendor-reported | High / Medium / Low |
| Retrieval and vector search | Yes / No / Unknown | measured / estimated / allocated / vendor-reported | High / Medium / Low |
| Tool calls and external APIs | Yes / No / Unknown | measured / estimated / allocated / vendor-reported | High / Medium / Low |
| Storage and logs | Yes / No / Unknown | measured / estimated / allocated / vendor-reported | High / Medium / Low |
| Network transfer | Yes / No / Unknown | measured / estimated / allocated / vendor-reported | High / Medium / Low |
| Facility PUE/WUE | Yes / No / Unknown | measured / estimated / allocated / vendor-reported | High / Medium / Low |
| Embodied hardware carbon | Yes / No / Unknown | measured / estimated / allocated / vendor-reported | High / Medium / Low |
| Human rework / retry cost | Yes / No / Unknown | measured / estimated / allocated / vendor-reported | High / Medium / Low |

## Architectural Design Constraints
- Maximum latency:
- Minimum quality / accuracy:
- Safety / policy constraints:
- Data residency constraints:
- Privacy constraints:
- Cost ceiling:
- Carbon objective:
- Water / region objective:
- Retention and deletion requirements:

## Evaluated Alternatives

### Option 1
- Model / route:
- Hardware / provider / region:
- Precision / quantization:
- Scheduling mode:
- Estimated or measured energy per request:
- Estimated or measured energy per verified task:
- Estimated or measured carbon per verified task:
- Estimated or measured water per verified task:
- Expected failure / retry / rework rate:
- Confidence level:
- Main risks:

### Option 2
- Model / route:
- Hardware / provider / region:
- Precision / quantization:
- Scheduling mode:
- Estimated or measured energy per request:
- Estimated or measured energy per verified task:
- Estimated or measured carbon per verified task:
- Estimated or measured water per verified task:
- Expected failure / retry / rework rate:
- Confidence level:
- Main risks:

## Decision Outcome
- Chosen option:
- Justification:
- Why smaller routes were rejected:
- Why larger routes were rejected:
- How the choice balances quality, latency, cost, carbon, water, privacy, and governance:

## Implementation Plan
- Routing rule:
- Quantization / precision:
- Cache policy:
- Context compression policy:
- Carbon-aware scheduling policy:
- Telemetry integration:
- Verification / eval gate:
- Rollback or degraded-mode path:

## Monitoring Plan
- Energy metric:
- Carbon metric:
- Water metric:
- Waste / retry metric:
- Utilization metric:
- Quality metric:
- Review cadence:

## Review Triggers
- Workload volume growth:
- Route/model/provider change:
- Hardware generation change:
- Regional grid/water change:
- Failure or retry spike:
- Cost spike:
- New regulatory/procurement requirement:

## Decommissioning Policy
- Retirement condition:
- Data/index/log cleanup:
- Hardware or reserved-capacity release:
- Vendor exit or route migration plan:

## Sign-off
- System Owner:
- Platform / Infrastructure Lead:
- FinOps Owner:
- Sustainability / ESG Owner:
- Governance / Compliance Owner:
- Date:

Cross-Canon Handoff Map

AI-ENG-AE defines sustainability as an architectural constraint across model selection, serving, routing, scheduling, telemetry, procurement, and lifecycle management. It consumes runtime, governance, evaluation, and operations evidence from surrounding canon layers and returns sustainability controls back into routing, procurement, and lifecycle review.

The core handoff principle is that sustainability cannot live in a separate reporting layer. It must be wired into the same routes, gates, inventories, evaluations, runbooks, and procurement controls that govern production AI behavior.

Works cited

1. Carbon-Aware Scheduling of AI Data Center Workloads Using Environmental and Energy Grid Forecasts - ESS Open Archive, accessed June 14, 2026, https://essopenarchive.org/doi/pdf/10.22541/essoar.177248651.16591616
2. Carbon-Aware AI Workload Scheduling With Renewable Energy Sources - Scribd, accessed June 14, 2026, https://www.scribd.com/document/1007342445/Carbon-Aware-AI-Workload-Scheduling-With-Renewable-Energy-Sources
3. Google data centers shift their computations to cleaner times and locations - Electricity Maps, accessed June 14, 2026, https://www.electricitymaps.com/resources/case-studies/google-data-centers-shift-their-computations-to-cleaner-times-and-locations
4. Carbon-Aware Compute–Power Scheduling for AI Data Centers with Microgrid Prosumer Operations - arXiv, accessed June 14, 2026, https://arxiv.org/html/2605.03751v1

5. Embodied carbon vs. operational carbon

   One Click LCA, accessed June 14, 2026, https://oneclicklca.com/en-us/resources/articles/embodied-carbon-vs-operational-carbon

6. Embodied Carbon and Its Role in Product Development - Ecochain, accessed June 14, 2026, https://ecochain.com/blog/embodied-carbon/
7. Understanding embodied and operational carbon in data centers: ESG considerations, accessed June 14, 2026, https://www.gresb.com/understanding-embodied-and-operational-carbon-in-data-centers-esg-considerations/
8. Product Carbon Footprint (PCF) Summary for NVIDIA HGX H100, accessed June 14, 2026, https://images.nvidia.com/aem-dam/Solutions/documents/HGX-H100-PCF-Summary.pdf
9. Carbon Emissions of NVIDIA A100 and H100 GPUs with 100% Renewable Energy, accessed June 14, 2026, https://massedcompute.com/faq-answers/?question=What%20are%20the%20estimated%20carbon%20emissions%20of%20NVIDIA%27s%20A100%20and%20H100%20GPUs%20in%20a%20data%20center%20with%20a%20100%%20renewable%20energy%20source?
10. Life-Cycle Emissions of AI Hardware: A Cradle-To-Grave Approach and Generational Trends - arXiv, accessed June 14, 2026, https://arxiv.org/html/2502.01671v1

11. Ironwood TPUs deliver 3.7x carbon efficiency gains

    Google Cloud Blog, accessed June 14, 2026, https://cloud.google.com/blog/topics/systems/ironwood-tpus-deliver-37x-carbon-efficiency-gains

12. FinOps in GCCs: Managing AI & Cloud Spend with Intelligence - Polestar Analytics, accessed June 14, 2026, https://www.polestaranalytics.com/blog/how-finops-in-gccs-is-transforming-finance-operations-with-ai-and-automation
13. Where Do the Joules Go? Diagnosing Inference Energy Consumption - arXiv, accessed June 14, 2026, https://arxiv.org/html/2601.22076v1
14. Babbling Suppression: Making LLMs Greener One Token at a Time - arXiv, accessed June 14, 2026, https://arxiv.org/html/2604.06755v1
15. How much heat does an AI data centre produce, and where are they located? - Al Jazeera, accessed June 14, 2026, https://www.aljazeera.com/news/2026/6/11/how-much-heat-does-an-ai-data-centre-produce-and-where-are-they-located
16. Whole Life Carbon Assessment vs. Embodied Carbon: Key Differences Explained - Cerclos, accessed June 14, 2026, https://cerclos.com/whole-life-carbon-assessment-vs-embodied-carbon-key-differences-explained/
17. Majority of US’s new AI datacenters to be built on drought-hit land, accessed June 14, 2026, https://www.theguardian.com/us-news/2026/jun/08/datacenter-ai-drought-water
18. [2508.12590] Energy-Efficient Wireless LLM Inference via Uncertainty and Importance-Aware Speculative Decoding - arXiv, accessed June 14, 2026, https://arxiv.org/abs/2508.12590

19. Local AI vs Cloud AI in 2026: When to Run Models on Your Own Hardware

    MindStudio, accessed June 14, 2026, https://www.mindstudio.ai/blog/local-ai-vs-cloud-ai-2026

20. Data Drain: The Land and Water Impacts of the AI Boom - Lincoln Institute of Land Policy, accessed June 14, 2026, https://www.lincolninst.edu/publications/land-lines-magazine/articles/land-water-impacts-data-centers/

21. Global energy demands within the AI regulatory landscape

    Brookings, accessed June 14, 2026, https://www.brookings.edu/articles/global-energy-demands-within-the-ai-regulatory-landscape/

22. Big Tech Accused of AI ‘Greenwashing’ - Resilience.org, accessed June 14, 2026, https://www.resilience.org/stories/2026-02-19/big-tech-accused-of-ai-greenwashing/

23. The Great Green AI Hoax Machine

    TechPolicy.Press, accessed June 14, 2026, https://www.techpolicy.press/the-great-green-ai-hoax-machine/

24. Report exposes Big Tech’s AI climate hoax: 74% of industry’s claims about AI’s climate benefits are unproven - Stand.earth, accessed June 14, 2026, https://stand.earth/press-releases/report-exposes-big-techs-ai-climate-hoax-74-of-industrys-claims-about-ais-climate-benefits-are-unproven/
25. Report exposes Big Tech’s AI climate hoax: 74% of industry’s claims about AI’s climate benefits are unproven - Green Web Foundation, accessed June 14, 2026, https://www.thegreenwebfoundation.org/news/report-exposes-big-techs-ai-climate-hoax-74-of-industrys-claims-about-ais-climate-benefits-are-unproven/

26. Zombie Workloads: The Walking Dead of Your Cloud Bill

    by FinOps Universe

    Medium, accessed June 14, 2026, https://finopsuniverse.medium.com/zombie-workloads-the-walking-dead-of-your-cloud-bill-585756ce44da

27. Gartner: Data Centres Electricity Consumption Up 26% in 2026, accessed June 14, 2026, https://energydigital.com/news/gartner-data-centres-electricity-consumption-up-26-in-2026
28. AI, Data Centers, and the U.S. Electric Grid: A Watershed Moment, accessed June 14, 2026, https://www.belfercenter.org/research-analysis/ai-data-centers-us-electric-grid
29. ai-eng-ad-governance-architecture.md
30. Executive summary – Key Questions on Energy and AI – Analysis - IEA, accessed June 14, 2026, https://www.iea.org/reports/key-questions-on-energy-and-ai/executive-summary
31. AI’s secret water crisis: How data centres are draining freshwater reserves across the world, accessed June 14, 2026, https://timesofindia.indiatimes.com/science/ais-secret-water-crisis-how-data-centres-are-draining-freshwater-reserves-across-the-world/articleshow/131590203.cms

32. AI Cost Optimization 2026: GPU, LLM & Infrastructure Spend Guide

    Opslyft, accessed June 14, 2026, https://www.opslyft.com/guides/ai-cost-optimization

33. The AI Boom Has a Water Bill, accessed June 14, 2026, https://medium.com/activated-thinker/the-ai-boom-has-a-water-bill-a77fae04505a
34. AI Energy Consumption Statistics - AIMultiple, accessed June 14, 2026, https://aimultiple.com/ai-energy-consumption

35. TPUs improved carbon-efficiency of AI workloads by 3x

    Google Cloud Blog, accessed June 14, 2026, https://cloud.google.com/blog/topics/sustainability/tpus-improved-carbon-efficiency-of-ai-workloads-by-3x

36. Towards Green AI: Decoding the Energy of LLM Inference in Software Development - arXiv, accessed June 14, 2026, https://arxiv.org/html/2602.05712v1
37. Tokens per Joule: How to Quantify and Reduce the Energy Footprint of Clinical LLM Inference - John Snow Labs, accessed June 14, 2026, https://www.johnsnowlabs.com/tokens-per-joule-how-to-quantify-and-reduce-the-energy-footprint-of-clinical-llm-inference/
38. How to Build Context Compression - OneUptime, accessed June 14, 2026, https://oneuptime.com/blog/post/2026-01-30-context-compression/view
39. Carbon-Aware Model Training: Scheduling GPU Workloads Around Electricity Carbon Intensity - DEV Community, accessed June 14, 2026, https://dev.to/nilofer_tweets/carbon-aware-model-training-scheduling-gpu-workloads-around-electricity-carbon-intensity-b4b
40. (PDF) Towards Green AI: Decoding the Energy of LLM Inference in Software Development, accessed June 14, 2026, https://www.researchgate.net/publication/400506146_Towards_Green_AI_Decoding_the_Energy_of_LLM_Inference_in_Software_Development
41. LLM Inference Energy Use - Emergent Mind, accessed June 14, 2026, https://www.emergentmind.com/topics/llm-inference-energy-consumption
42. Context Compression Before the LLM: Cutting Tokens Without Cutting Recall - DEV Community, accessed June 14, 2026, https://dev.to/gabrielanhaia/context-compression-before-the-llm-cutting-tokens-without-cutting-recall-9hh
43. Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling - Shashikant Ilager, accessed June 14, 2026, https://shashikantilager.com/assets/pdf/publications/LLM_charecterization_ccgrid_2026.pdf

44. An Introduction to Speculative Decoding for Reducing Latency in AI Inference

    NVIDIA Technical Blog, accessed June 14, 2026, https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/

45. Babbling Suppression: Making LLMs Greener One Token at a Time - arXiv, accessed June 14, 2026, https://arxiv.org/pdf/2604.06755
46. TokenPowerBench: Benchmarking the Power Consumption of LLM Inference - AAAI Publications, accessed June 14, 2026, https://ojs.aaai.org/index.php/AAAI/article/view/40535/44496
47. Should I Use Cloud or Local AI Models for My Project? - Zen van Riel, accessed June 14, 2026, https://zenvanriel.com/ai-engineer-blog/should-i-use-cloud-or-local-ai-models-complete-guide/

48. NVIDIA HGX B200 Reduces Embodied Carbon Emissions Intensity

    NVIDIA Technical Blog, accessed June 14, 2026, https://developer.nvidia.com/blog/nvidia-hgx-b200-reduces-embodied-carbon-emissions-intensity/

49. Context Compression Techniques (2026) - SurePrompts, accessed June 14, 2026, https://sureprompts.com/blog/context-compression-techniques

50. Context compression finally works in production: new research cuts LLM input 16x without the accuracy hit

    VentureBeat, accessed June 14, 2026, https://venturebeat.com/data/context-compression-finally-works-in-production-new-research-cuts-llm-input-16x-without-the-accuracy-hit

51. GreenOps & FinOps – WisdomAI: Why zombie data is the hidden cost of cloud, accessed June 14, 2026, https://www.computerweekly.com/blog/CW-Developer-Network/GreenOps-FinOps-WisdomAI-Why-zombie-data-is-the-hidden-cost-of-cloud
52. Appendix A. - Greenpeace, accessed June 14, 2026, https://www.greenpeace.org/static/planet4-eastasia-stateless/2025/04/c97a710a-energy_consumption_of_ai_appendix.pdf
53. How Shadow AI Creates Zombie Infrastructure - Netscout, accessed June 14, 2026, https://www.netscout.com/blog/how-shadow-ai-creates-zombie-infrastructure
54. Claims that AI can help fix climate dismissed as greenwashing - The Guardian, accessed June 14, 2026, https://www.theguardian.com/technology/2026/feb/17/tech-companies-traditional-ai-generative-climate-breakdown-report
55. Datacenter & AI water use is overblown, accessed June 14, 2026, https://www.reddit.com/r/artificial/comments/1u4128s/datacenter_ai_water_use_is_overblown/

Attribution

Part of Stunspot’s Guide to AI Systems — The AI Engineering Systems Canon.

Created by Sam “stunspot” Walker / Collaborative Dynamics.

Repository: https://github.com/Stunspot/stunspots-guide-to-ai-systems
Stunspot: https://stunspot.com
Collaborative Dynamics: https://www.collaborative-dynamics.com
Discord: https://discord.gg/stunspot

Licensed under CC BY-NC-SA 4.0 unless otherwise stated.
Commercial use, resale, paid redistribution, inclusion in commercial training products, or incorporation into paid knowledge-base products requires prior written permission.

← Back to Canon Map