Tool choice with Amazon Nova models

In many generative AI applications, a large language model (LLM) like Amazon Nova is used to respond to a user query based on the model’s own knowledge or context that it is provided. However, as use cases have matured, the ability for a model to have access to tools or structures that would be inherently outside of the model’s frame of reference has become paramount. This could be APIs, code functions, or schemas and structures required by your end application. This capability has developed into what is referred to as tool use or function calling.

To add fine-grained control to how tools are used, we have released a feature for tool choice for Amazon Nova models. Instead of relying on prompt engineering, tool choice forces the model to adhere to the settings in place.

In this post, we discuss tool use and the new tool choice feature, with example use cases.

Tool use with Amazon Nova

To illustrate the concept of tool use, we can imagine a situation where we provide Amazon Nova access to a few different tools, such as a calculator or a weather API. Based on the user’s query, Amazon Nova will select the appropriate tool and tell you how to use it. For example, if a user asks “What is the weather in Seattle?” Amazon Nova will use the weather tool.

The following diagram illustrates an example workflow between an Amazon Nova model, its available tools, and related external resources.

Tool use at the core is the selection of the tool and its parameters. The responsibility to execute the external functionality is left to application or developer. After the tool is executed by the application, you can return the results to the model for the generation of the final response.

Let’s explore some examples in more detail. The following diagram illustrates the workflow of an Amazon Nova model using a function call to access a weather API, and returning the response to the user.

The following diagram illustrates the workflow of an Amazon Nova model using a function call to access a calculator tool.

Tool choice with Amazon Nova

The toolChoice API parameter allows you to control when a tool is called. There are three supported options for this parameter:

Any – With tool choice Any, the model will select at least one of the available tools each time:

{
“toolChoice”: {
“any”: {}
}
}

Tool – With tool choice Tool, the model will always use the requested tool:

{
“toolChoice”: {
“tool”: {
“name”: “name_of_tool”
}
}
}

Auto – Tool choice Auto is the default behavior and will leave the tool selection completely up to the model:

{
“toolChoice”: {
“auto”: {}
}
}

A popular tactic to improve the reasoning capabilities of a model is to use chain of thought. When using the tool choice of auto, Amazon Nova will use chain of thought and the response of the model will include both the reasoning and the tool that was selected.

This behavior will differ depending on the use case. When tool or any are selected as the tool choice, Amazon Nova will output only the tools and not output chain of thought.

Use cases

In this section, we explore different use cases for tool choice.

Structured output/JSON mode

In certain scenarios, you might want Amazon Nova to use a specific tool to answer the user’s question, even if Amazon Nova believes it can provide a response without the use of a tool. A common use case for this approach is enforcing structured output/JSON mode. It’s often critical to have LLMs return structured output, because this enables downstream use cases to more effectively consume and process the generated outputs. In these instances, the tools employed don’t necessarily need to be client-side functions—they can be used whenever the model is required to return JSON output adhering to a predefined schema, thereby compelling Amazon Nova to use the specified tool.

When using tools for enforcing structured output, you provide a single tool with a descriptive JSON inputSchema. You specify the tool with {“tool” : {“name” : “Your tool name”}}. The model will pass the input to the tool, so the name of the tool and its description should be from the model’s perspective.

For example, consider a food website. When provided with a dish description, the website can extract the recipe details, such as cooking time, ingredients, dish name, and difficulty level, in order to facilitate user search and filtering capabilities. See the following example code:

import boto3
import json

tool_config = {
“toolChoice”: {
“name”: { “tool” : “extract_recipe”}
},
“tools”: [
{
“toolSpec”: {
“name”: “extract_recipe”,
“description”: “Extract recipe for cooking instructions”,
“inputSchema”: {
“json”: {
“type”: “object”,
“properties”: {
“recipe”: {
“type”: “object”,
“properties”: {
“name”: {
“type”: “string”,
“description”: “Name of the recipe”
},
“description”: {
“type”: “string”,
“description”: “Brief description of the dish”
},
“prep_time”: {
“type”: “integer”,
“description”: “Preparation time in minutes”
},
“cook_time”: {
“type”: “integer”,
“description”: “Cooking time in minutes”
},
“servings”: {
“type”: “integer”,
“description”: “Number of servings”
},
“difficulty”: {
“type”: “string”,
“enum”: [“easy”, “medium”, “hard”],
“description”: “Difficulty level of the recipe”
},
“ingredients”: {
“type”: “array”,
“items”: {
“type”: “object”,
“properties”: {
“item”: {
“type”: “string”,
“description”: “Name of ingredient”
},
“amount”: {
“type”: “number”,
“description”: “Quantity of ingredient”
},
“unit”: {
“type”: “string”,
“description”: “Unit of measurement”
}
},
“required”: [“item”, “amount”, “unit”]
}
},
“instructions”: {
“type”: “array”,
“items”: {
“type”: “string”,
“description”: “Step-by-step cooking instructions”
}
},
“tags”: {
“type”: “array”,
“items”: {
“type”: “string”,
“description”: “Categories or labels for the recipe”
}
}
},
“required”: [“name”, “ingredients”, “instructions”]
}
},
“required”: [“recipe”]
}
}
}
}
]
}

messages = [{
“role”: “user”,
“content”: [
{“text”: input_text},
]
}]

inf_params = {“topP”: 1, “temperature”: 1}

client = boto3.client(“bedrock-runtime”, region_name=”us-east-1″)

response = client.converse(
modelId=”us.amazon.nova-micro-v1:0″,
messages=messages,
toolConfig=tool_config,
inferenceConfig=inf_params,
additionalModelRequestFields= {“inferenceConfig”: { “topK”: 1 } }
)
print(json.dumps(response[‘output’][‘message’][‘content’][0][], indent=2))

We can provide a detailed description of a dish as text input:

Legend has it that this decadent chocolate lava cake was born out of a baking mistake in New York’s Any Kitchen back in 1987, when chef John Doe pulled a chocolate sponge cake out of the oven too early, only to discover that the dessert world would never be the same. Today I’m sharing my foolproof version, refined over countless dinner parties. Picture a delicate chocolate cake that, when pierced with a fork, releases a stream of warm, velvety chocolate sauce – it’s pure theater at the table. While it looks like a restaurant-worthy masterpiece, the beauty lies in its simplicity: just six ingredients (good quality dark chocolate, unsalted butter, eggs, sugar, flour, and a pinch of salt) transform into individual cakes in under 15 minutes. The secret? Precise timing is everything. Pull them from the oven a minute too late, and you’ll miss that magical molten center; too early, and they’ll be raw. But hit that sweet spot at exactly 12 minutes, when the edges are set but the center still wobbles slightly, and you’ve achieved dessert perfection. I love serving these straight from the oven, dusted with powdered sugar and topped with a small scoop of vanilla bean ice cream that slowly melts into the warm cake. The contrast of temperatures and textures – warm and cold, crisp and gooey – makes this simple dessert absolutely unforgettable.

We can force Amazon Nova to use the tool extract_recipe, which will generate a structured JSON output that adheres to the predefined schema provided as the tool input schema:

{
“toolUseId”: “tooluse_4YT_DYwGQlicsNYMbWFGPA”,
“name”: “extract_recipe”,
“input”: {
“recipe”: {
“name”: “Decadent Chocolate Lava Cake”,
“description”: “A delicate chocolate cake that releases a stream of warm, velvety chocolate sauce when pierced with a fork. It’s pure theater at the table.”,
“difficulty”: “medium”,
“ingredients”: [
{
“item”: “good quality dark chocolate”,
“amount”: 125,
“unit”: “g”
},
{
“item”: “unsalted butter”,
“amount”: 125,
“unit”: “g”
},
{
“item”: “eggs”,
“amount”: 4,
“unit”: “”
},
{
“item”: “sugar”,
“amount”: 100,
“unit”: “g”
},
{
“item”: “flour”,
“amount”: 50,
“unit”: “g”
},
{
“item”: “salt”,
“amount”: 0.5,
“unit”: “pinch”
}
],
“instructions”: [
“Preheat the oven to 200u00b0C (400u00b0F).”,
“Melt the chocolate and butter together in a heatproof bowl over a saucepan of simmering water.”,
“In a separate bowl, whisk the eggs and sugar until pale and creamy.”,
“Fold the melted chocolate mixture into the egg and sugar mixture.”,
“Sift the flour and salt into the mixture and gently fold until just combined.”,
“Divide the mixture among six ramekins and bake for 12 minutes.”,
“Serve straight from the oven, dusted with powdered sugar and topped with a small scoop of vanilla bean ice cream.”
],
“prep_time”: 10,
“cook_time”: 12,
“servings”: 6,
“tags”: [
“dessert”,
“chocolate”,
“cake”
]
}
}
}

API generation

Another common scenario is to require Amazon Nova to select a tool from the available options no matter the context of the user query. One example of this is with API endpoint selection. In this situation, we don’t know the specific tool to use, and we allow the model to choose between the ones available.

With the tool choice of any, you can make sure that the model will always use at least one of the available tools. Because of this, we provide a tool that can be used for when an API is not relevant. Another example would be to provide a tool that allows clarifying questions.

In this example, we provide the model with two different APIs, and an unsupported API tool that it will select based on the user query:

import boto3
import json

tool_config = {
“toolChoice”: {
“any”: {}
},
“tools”: [
{
“toolSpec”: {
“name”: “get_all_products”,
“description”: “API to retrieve multiple products with filtering and pagination options”,
“inputSchema”: {
“json”: {
“type”: “object”,
“properties”: {
“sort_by”: {
“type”: “string”,
“description”: “Field to sort results by. One of: price, name, created_date, popularity”,
“default”: “created_date”
},
“sort_order”: {
“type”: “string”,
“description”: “Order of sorting (ascending or descending). One of: asc, desc”,
“default”: “desc”
},
},
“required”: []
}
}
}
},
{
“toolSpec”: {
“name”: “get_products_by_id”,
“description”: “API to retrieve retail products based on search criteria”,
“inputSchema”: {
“json”: {
“type”: “object”,
“properties”: {
“product_id”: {
“type”: “string”,
“description”: “Unique identifier of the product”
},
},
“required”: [“product_id”]
}
}
}
},
{
“toolSpec”: {
“name”: “unsupported_api”,
“description”: “API to use when the user query does not relate to the other available APIs”,
“inputSchema”: {
“json”: {
“type”: “object”,
“properties”: {
“reasoning”: {
“type”: “string”,
“description”: “The reasoning for why the user query did not have a valid API available”
},
},
“required”: [“reasoning”]
}
}
}
}
]
}

messages = [{
“role”: “user”,
“content”: [
{“text”: input_text},
]
}]

inf_params = {“topP”: 1, “temperature”: 1}

client = boto3.client(“bedrock-runtime”, region_name=”us-east-1″)

print(json.dumps(response[‘output’][‘message’][‘content’][0], indent=2))

A user input of “Can you get all of the available products?” would output the following:

{
“toolUse”: {
“toolUseId”: “tooluse_YCNbT0GwSAyjIYOuWnDhkw”,
“name”: “get_all_products”,
“input”: {}
}
}

Whereas “Can you get my most recent orders?” would output the following:

{
“toolUse”: {
“toolUseId”: “tooluse_jpiZnrVcQDS1sAa-qPwIQw”,
“name”: “unsupported_api”,
“input”: {
“reasoning”: “The available tools do not support retrieving user orders. The user’s request is for personal order information, which is not covered by the provided APIs.”
}
}
}

Chat with search

The final option for tool choice is auto. This is the default behavior, so it is consistent with providing no tool choice at all.

Using this tool choice will allow the option of tool use or just text output. If the model selects a tool, there will be a tool block and text block. If the model responds with no tool, only a text block is returned. In the following example, we want to allow the model to respond to the user or call a tool if necessary:

import boto3
import json

tool_config = {
“toolChoice”: {
“auto”: {}
},
“tools”: [
{
“toolSpec”: {
“name”: “search”,
“description”: “API that provides access to the internet”,
“inputSchema”: {
“json”: {
“type”: “object”,
“properties”: {
“query”: {
“type”: “string”,
“description”: “Query to search by”,
},
},
“required”: [“query”]
}
}
}
}
]
}

messages = [{
“role”: “user”,
“content”: [
{“text”: input_text},
]
}]

system = [{
“text”: “ou are a helpful chatbot. You can use a tool if necessary or respond to the user query”
}]

inf_params = {“topP”: 1, “temperature”: 1}

client = boto3.client(“bedrock-runtime”, region_name=”us-east-1″)

if (response[“stopReason”] == “tool_use”):
tool_use = next(
block[“toolUse”]
for block in response[“output”][“message”][“content”]
if “toolUse” in block
)
print(json.dumps(tool_use, indent=2))
else:
pattern = r'<thinking>.*?</thinking>\n\n|<thinking>.*?</thinking>’
text_response = response[“output”][“message”][“content”][0][“text”]
stripped_text = re.sub(pattern, ”, text_response, flags=re.DOTALL)

print(stripped_text)

A user input of “What is the weather in San Francisco?” would result in a tool call:

{
“toolUseId”: “tooluse_IwtBnbuuSoynn1qFiGtmHA”,
“name”: “search”,
“input”: {
“query”: “what is the weather in san francisco”
}
}

Whereas asking the model a direct question like “How many months are in a year?” would respond with a text response to the user:

There are 12 months in a year.

Considerations

There are a few best practices that are required for tool calling with Nova models. The first is to use greedy decoding parameters. With Amazon Nova models, that requires setting a temperature, top p, and top k of 1. You can refer to the previous code examples for how to set these. Using greedy decoding parameters forces the models to produce deterministic responses and improves the success rate of tool calling.

The second consideration is the JSON schema you are using for the tool consideration. At the time of writing, Amazon Nova models support a limited subset of JSON schemas, so they might not be picked up as expected by the model. Common fields would be $def and $ref fields. Make sure that your schema has the following top-level fields set: type (must be object), properties, and required.

Lastly, for the most impact on the success of tool calling, you should optimize your tool configurations. Descriptions and names should be very clear. If there are nuances to when one tool should be called over the other, make sure to have that concisely included in the tool descriptions.

Conclusion

Using tool choice in tool calling workflows is a scalable way to control how a model invokes tools. Instead of relying on prompt engineering, tool choice forces the model to adhere to the settings in place. However, there are complexities to tool calling; for more information, refer to Tool use (function calling) with Amazon Nova, Tool calling systems, and Troubleshooting tool calls.

Explore how Amazon Nova models can enhance your generative AI use cases today.

About the Authors

Jean Farmer is a Generative AI Solutions Architect on the Amazon Artificial General Intelligence (AGI) team, specializing in agentic applications. Based in Seattle, Washington, she works at the intersection of autonomous AI systems and practical business solutions, helping to shape the future of AGI at Amazon.

Sharon Li is an AI/ML Specialist Solutions Architect at Amazon Web Services (AWS) based in Boston, Massachusetts. With a passion for leveraging cutting-edge technology, Sharon is at the forefront of developing and deploying innovative generative AI solutions on the AWS cloud platform.

Lulu Wong is an AI UX designer on the Amazon Artificial General Intelligence (AGI) team. With a background in computer science, learning design, and user experience, she bridges the technical and user experience domains by shaping how AI systems interact with humans, refining model input-output behaviors, and creating resources to make AI products more accessible to users.

Tool use with Amazon Nova

Tool choice with Amazon Nova

Use cases

Structured output/JSON mode

API generation

Chat with search

Considerations

Conclusion

About the Authors

Related Posts

Leave a Comment Cancel Reply