Calling a large language model is relatively simple. Building a dependable application around that model is considerably more difficult.
Production AI applications rarely consist of a single prompt followed by a single response. They may need to retrieve supporting information, construct prompts, select a model, invoke multiple models, validate responses, call external tools, route low-confidence answers for review, and store the final result.
That is where LLM orchestration becomes important.
AWS Step Functions provides a visual, state-based way to coordinate these operations. Instead of placing an entire generative AI process inside one Lambda function or application service, developers can represent each operation as an explicit step within a state machine.
The result is an LLM workflow that is easier to understand, debug, monitor, and modify.
What Is LLM Orchestration?
LLM orchestration is the coordination of models, data sources, tools, validation services, and business rules required to complete an AI-driven process.
A basic orchestration workflow might look like this:
- Receive and validate a user request.
- Determine what type of request was submitted.
- Retrieve relevant documents or application data.
- Construct a model-specific prompt.
- Invoke an LLM.
- evaluate the response for quality or safety.
- Retry, revise, approve, or reject the response.
- Store the result and notify another system.
Each of these operations has its own inputs, outputs, failure conditions, and retry requirements. Combining them inside a single function can produce code that is difficult to maintain and nearly impossible to visualize.
Step Functions separates the process into states connected by defined transitions. AWS supports task states for invoking services and flow states such as Choice, Parallel, Map, Wait, Pass, Succeed, and Fail.
Why Use AWS Step Functions for LLM Workflows?
AWS Step Functions was designed to coordinate distributed applications and services. Many of the same capabilities are especially valuable for generative AI.
Visual workflow definitions
An AI workflow can be viewed as a graph rather than reconstructed from application code and log entries. Developers can see where prompts are generated, models are invoked, decisions are made, and failures occur.
This becomes particularly useful when an application contains several model calls or conditional paths.
Built-in retries and error handling
LLM endpoints may experience throttling, network failures, timeouts, capacity limitations, or temporary service errors.
Step Functions supports Retry and Catch configurations that allow individual tasks to retry with exponential backoff or transition to a fallback state. Without a matching error handler, a failed task normally causes the workflow execution to fail.
Start free. No AWS account needed.
ZERO AWS costs.
Download Thrubit and run your first state machine locally in under five minutes. No cloud setup, no IAM policies, no waiting.
Conditional routing
A Choice state can direct an execution to different branches based on information in the workflow state.
For example, a workflow might:
- Send a simple classification request to a smaller model.
- Send a complex analysis request to a more capable model.
- Escalate low-confidence results to a human reviewer.
- Reject content that fails a policy check.
- Repeat generation when an evaluation score is below a threshold.
Step Functions evaluates the rules in a Choice state and moves the execution to the appropriate next state.
Parallel model execution
A Parallel state runs multiple branches concurrently and waits for the branches to finish before continuing.
This makes it possible to send the same request to multiple models or run separate AI tasks at the same time. One branch could summarize a document while another extracts entities and a third evaluates its sentiment.
Batch processing with Map states
A Map state applies the same workflow to multiple items. This can be used to process collections of documents, support tickets, product descriptions, transcripts, or database records.
Distributed Map can execute each item as a child workflow and supports high levels of concurrency for large datasets.
Human approval
Some AI outputs should not be published or acted upon automatically.
Step Functions callback patterns allow a workflow to pause until an external system returns a task token. This can be used for legal review, editorial approval, financial authorization, or other human-in-the-loop processes.
Connecting Step Functions to an LLM
There are several ways to invoke a model from a Step Functions workflow.
1. Invoke Amazon Bedrock Directly
Step Functions includes an optimized Amazon Bedrock integration for invoking a model with InvokeModel. It can also coordinate model customization jobs.
A simplified Amazon States Language task might look like this:
{
"InvokeModel": {
"Type": "Task",
"Resource": "arn:aws:states:::bedrock:invokeModel",
"Parameters": {
"ModelId": "<MODEL_ID>",
"Body": {
"prompt.$": "$.prompt",
"max_tokens": 800
},
"ContentType": "application/json",
"Accept": "application/json"
},
"ResultPath": "$.modelResponse",
"Next": "EvaluateResponse"
}
}The contents of Body depend on the selected model. Step Functions sends the model-specific body to Amazon Bedrock but does not validate the model parameters contained within it.
For especially large prompts, Step Functions can provide Bedrock input from Amazon S3. The optimized integration supports an inline Body of up to 256 KiB and provides Input and Output fields for using S3 locations instead.
AWS also provides an official Step Functions prompt-chaining sample that demonstrates coordinating multiple Bedrock invocations in a state machine.
2. Use Bedrock APIs Through AWS SDK Integrations
Step Functions AWS SDK integrations can call API actions across hundreds of AWS services. This provides access to Bedrock capabilities that may not have dedicated optimized Step Functions integrations.
Amazon Bedrock recommends its Converse API for conversational applications because it provides a consistent request structure across supported message-based models.
AWS SDK integration availability can lag behind newly released service APIs, however, so developers should confirm that the required Bedrock Runtime operation is currently supported before designing around a direct SDK task.
3. Call External LLM APIs with an HTTP Task
Step Functions can invoke HTTPS endpoints through the arn:aws:states:::http:invoke resource.
Authentication and network configuration are managed through an Amazon EventBridge connection, which helps prevent API credentials from being hard-coded in the state machine definition.
This approach can be used to coordinate models and AI services hosted outside Amazon Bedrock.
One limitation is that HTTP Task requests time out after 60 seconds. Longer-running model calls may require a Lambda adapter, asynchronous API pattern, callback workflow, or another integration architecture.
4. Invoke Models Through AWS Lambda
A Lambda function can provide a common adapter between Step Functions and one or more model providers.
The function can:
- Add provider-specific authentication.
- Normalize request formats.
- Parse model-specific responses.
- Implement streaming or asynchronous behavior.
- Apply application-specific validation.
- Return a consistent response structure to Step Functions.
Lambda is also useful when the model invocation requires custom preprocessing that cannot be expressed conveniently through Amazon States Language.
The tradeoff is that the Lambda function adds another layer of code. When a native Step Functions integration can accomplish the same task, direct service integration usually produces a simpler workflow.
LLM Orchestration Patterns with Step Functions
Once model access has been established, Step Functions can support several common generative AI patterns.
Prompt Chaining
Prompt chaining divides a complex request into a series of smaller model calls.
For example:
- Classify the user’s intent.
- Extract the required facts.
- Produce an outline.
- Generate a draft.
- Review the draft.
- Revise the result.
The output of each model becomes part of the input for the next model.
Prompt chaining can produce more controlled results than asking one model to complete an entire complex process in a single request. It also makes each stage independently observable.
If the outline is incorrect, developers can inspect the outline state instead of troubleshooting one enormous prompt and response.
Model Routing
Not every request requires the same model.
A routing workflow can begin with a classifier that identifies the request’s complexity, language, subject, or risk level. A Choice state can then select the appropriate model or prompt.
For example:
- Route routine classifications to a smaller, faster model.
- Route complex analysis to a more capable model.
- Route requests containing sensitive information through additional safeguards.
- Route multilingual requests to a model with stronger support for that language.
- Route an unavailable model to a fallback provider.
Amazon Bedrock also offers intelligent prompt routing within supported model families, but Step Functions remains useful when routing decisions include broader business rules, multiple providers, retrieval processes, or human review.
Parallel Generation and Model Ensembles
A Parallel state can send a prompt to multiple models concurrently.
The responses can then be passed to an evaluator that selects the strongest result or combines portions of several responses.
Possible uses include:
- Comparing answers from different models.
- Generating several creative variations.
- Running generation and fact extraction simultaneously.
- Using one model to answer and another to critique.
- Comparing a low-cost model against a premium model.
- Running several safety or quality checks in parallel.
This approach can increase confidence, although it also increases token usage and model costs. Workflows should only invoke multiple models when the additional result quality justifies the expense.
Retrieval-Augmented Generation
Retrieval-augmented generation, commonly called RAG, adds relevant data to a model’s context before generation.
A Step Functions RAG workflow could:
- Validate the incoming question.
- Generate or retrieve an embedding.
- Search a vector database or knowledge base.
- Filter and rank the retrieved documents.
- Construct a grounded prompt.
- Invoke the model.
- Check whether the response is supported by the retrieved context.
- Store the response and its source references.
Amazon Bedrock Knowledge Bases can retrieve information from configured data sources and use it to enrich generated responses.
Step Functions can coordinate the surrounding application logic, including authorization, retrieval fallbacks, response evaluation, notifications, and persistence.
Automated Response Evaluation
An LLM response does not have to be accepted immediately.
After generation, a separate evaluation step can check for:
- Relevance
- Completeness
- Format compliance
- Unsupported claims
- Missing citations
- Policy violations
- Required keywords or fields
- Confidence thresholds
- Tone or brand alignment
The evaluator might be another LLM, a Lambda function, a Bedrock Guardrail, a rules engine, or a combination of these methods.
A Choice state can then decide whether to accept the result, regenerate it, use another model, or request human review.
The workflow should include a maximum attempt count so a low-quality response does not create an unlimited regeneration loop.
Tool Use and Agentic Workflows
Some LLMs can determine that they need an external tool, such as a search service, database query, calculator, CRM action, or internal API.
Step Functions can act as the controlled execution layer around those tools:
- Invoke the model.
- Parse its requested tool action.
- Validate the requested action.
- Route to an approved integration.
- Execute the tool.
- Return the tool result to the model.
- Continue until a final response is produced.
Amazon Bedrock also supports model tool use, allowing supported models to request tools that the application can execute.
For more autonomous agent implementations, Step Functions can invoke an Amazon Bedrock AgentCore harness. AgentCore harnesses can coordinate inference, tools, memory, and multi-turn conversations, while Step Functions controls the larger business process surrounding the agent.
Step Functions is especially valuable when an agent’s proposed action must pass through deterministic authorization or approval rules before it can affect another system.
Guardrails and Content Safety
Generative AI workflows should evaluate both model inputs and outputs.
Amazon Bedrock Guardrails can detect or filter undesirable content, sensitive information, denied topics, and prompt attacks. Guardrails can be associated with supported model and agent operations or applied separately as part of a workflow.
A safety-focused workflow might:
- Inspect the user input.
- Reject or redact sensitive content.
- Invoke the model.
- Inspect the model response.
- Route blocked results to a safe response.
- Send uncertain results for review.
- Record the policy decision for auditing.
Guardrails should be one part of a layered security strategy. Applications should also restrict IAM permissions, validate tool arguments, control accessible data, and prevent model-generated values from being executed without verification.
Human-in-the-Loop LLM Workflows
Human review is appropriate when a model response could create legal, financial, safety, or reputational consequences.
A Step Functions workflow can pause after generation and send the response to an approval system. The execution continues only after an authorized reviewer approves, rejects, or edits the result.
Examples include:
- Approving contracts or legal summaries.
- Reviewing medical or financial communications.
- Authorizing high-value account actions.
- Publishing customer-facing content.
- Confirming changes proposed by an AI agent.
- Reviewing low-confidence data extraction.
Standard Workflows are usually more appropriate for these processes because they are durable, auditable, and can run for up to one year. Express Workflows are better suited to high-volume, short-duration workloads. The workflow type cannot be changed after the state machine is created.
A Simplified LLM Evaluation Workflow
The following example shows the overall structure of a generation and evaluation workflow:
{
"Comment": "Generate and evaluate an LLM response",
"StartAt": "PreparePrompt",
"States": {
"PreparePrompt": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "prepare-prompt",
"Payload.$": "$"
},
"ResultPath": "$.prepared",
"Next": "InvokeModel"
},
"InvokeModel": {
"Type": "Task",
"Resource": "arn:aws:states:::bedrock:invokeModel",
"Parameters": {
"ModelId": "<MODEL_ID>",
"Body": {
"prompt.$": "$.prepared.Payload.prompt",
"max_tokens": 800
},
"ContentType": "application/json",
"Accept": "application/json"
},
"ResultPath": "$.generation",
"Retry": [
{
"ErrorEquals": [
"States.Timeout",
"States.TaskFailed"
],
"IntervalSeconds": 2,
"MaxAttempts": 3,
"BackoffRate": 2
}
],
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"ResultPath": "$.generationError",
"Next": "GenerationFailed"
}
],
"Next": "EvaluateResponse"
},
"EvaluateResponse": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "evaluate-response",
"Payload.$": "$"
},
"ResultPath": "$.evaluation",
"Next": "CheckScore"
},
"CheckScore": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.evaluation.Payload.score",
"NumericGreaterThanEquals": 0.85,
"Next": "SaveResult"
}
],
"Default": "RequestReview"
},
"RequestReview": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "request-human-review",
"Payload.$": "$"
},
"Next": "SaveResult"
},
"SaveResult": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:putItem",
"Parameters": {
"TableName": "GeneratedResponses",
"Item": {
"ExecutionId": {
"S.$": "$$.Execution.Id"
},
"Status": {
"S": "COMPLETED"
}
}
},
"End": true
},
"GenerationFailed": {
"Type": "Fail",
"Error": "ModelGenerationFailed",
"Cause": "The model could not produce a response."
}
}
}This is intentionally simplified. A production workflow should use provider-specific request schemas, narrowly targeted retry errors, least-privilege IAM policies, payload size controls, idempotency safeguards, and explicit review outcomes.
LLM Workflow Observability
LLM applications need more than infrastructure monitoring. Teams should record enough information to understand how each AI decision was produced.
Useful execution data includes:
- State machine execution ID
- Prompt template version
- Model or inference profile
- Retrieval query
- Retrieved document identifiers
- Input and output token usage
- Model latency
- Evaluation score
- Retry count
- Guardrail outcome
- Human approval result
- Estimated model cost
Standard Step Functions executions provide a state-by-state execution history, making it possible to inspect the input and output associated with each operation. Express Workflows can send execution history to CloudWatch Logs when logging is enabled.
Avoid logging raw prompts or responses when they may contain credentials, personal information, confidential documents, or regulated data.
Cost and Reliability Controls
LLM workflows can become unexpectedly expensive when retries, parallel branches, or Map executions are not controlled.
Consider implementing:
- Maximum generation attempts.
- Maximum prompt and response sizes.
- Token limits for each model call.
- Map concurrency limits.
- Per-request cost ceilings.
- Idempotency keys for external actions.
- Timeouts for every task.
- Fallback models for temporary failures.
- Dead-letter or failure-processing workflows.
- Alerts for repeated evaluation failures.
Retries should focus on transient failures. A malformed prompt or invalid model identifier will not become valid simply because the request is repeated.
For batch operations, start with conservative concurrency settings and increase them after measuring provider quotas, latency, and cost.
When Step Functions Is a Good Fit
AWS Step Functions is a strong choice when an LLM workflow:
- Contains several distinct processing stages.
- Must coordinate multiple AWS services.
- Requires retries, branching, or fallback logic.
- Invokes several models or tools.
- Processes large collections of records.
- Requires human approval.
- Needs a visual and auditable execution history.
- Performs actions with business or operational consequences.
It may be unnecessary for a simple application that makes one model request and immediately returns the response.
Step Functions may also be a poor fit for token-by-token streaming or extremely latency-sensitive conversational loops. In those cases, an application service can manage the live model interaction while Step Functions coordinates longer-running work before or after the conversation.
Developing LLM Step Functions Workflows Locally
LLM orchestration can be difficult to test when every change requires a cloud deployment and every execution makes paid model requests.
Thrubit allows developers to run and debug AWS Step Functions workflows locally. State machines can be tested with local Lambda handlers, mock Amazon Bedrock responses, execution inputs, payload transformations, and structured logs before they are deployed to AWS.
This is particularly useful for prompt chains and branching workflows because developers can test:
- How one model response is passed into another prompt.
- Whether
Choicerules select the correct model. - How retries and errors affect an execution.
- What happens when a guardrail rejects a response.
- Whether Parallel and Map states combine results correctly.
- How the workflow behaves when a model is unavailable.
Local execution does not eliminate the need for cloud integration testing, but it can reduce the number of paid, deployment-dependent iterations required to reach a stable workflow.
Building Reliable AI Systems with Step Functions
The model is only one component of an AI application.
The reliability of the overall system depends on how the model is connected to data, tools, policies, evaluation methods, human decisions, and downstream services.
AWS Step Functions gives developers a structured way to make those connections visible and controllable. Prompt chains can be represented as individual states. Model selection can be handled through explicit rules. Parallel calls can be coordinated without custom concurrency code. Failures can be retried or routed to fallback paths. Sensitive actions can pause for human approval.
By moving orchestration out of a single application function and into a state machine, teams can build LLM workflows that are easier to inspect, test, govern, and improve.
Frequently Asked Questions
Yes. Step Functions provides an optimized Amazon Bedrock integration for invoking supported foundation models with InvokeModel. The state machine must have permission to invoke the selected model.
Yes. External LLM APIs can be invoked through an HTTP Task, API Gateway integration, or Lambda function. HTTP Tasks use EventBridge connections to manage API authentication and have a 60-second request timeout.
Standard Workflows are generally better for durable, auditable processes, long-running operations, and human approval. Express Workflows are better for high-volume, short-duration executions.
Step Functions does not directly prevent hallucinations. It can coordinate retrieval, evaluation, guardrails, validation, regeneration, and human review steps that reduce the likelihood of unsupported responses reaching users.
Not necessarily. An agent framework can manage model reasoning, tools, and conversations, while Step Functions manages the larger deterministic business process. Step Functions can also invoke Amazon Bedrock AgentCore harnesses as part of a workflow.
Yes. Local development tools such as Thrubit can execute state machines, local Lambda handlers, and mocked Bedrock interactions without requiring every development test to run as a deployed AWS execution.