Build Multi-Agent Apps with OpenAI’s Agent SDK

of abstraction constructed on high of basically easy concepts, some agent framework devs appear to consider complexity is a advantage.

I are inclined to associate with Einstein’s maxim, “All the things needs to be made so simple as attainable, however not less complicated”. So, let me present you a framework that’s straightforward to make use of and straightforward to grasp.

OpenAI takes a refreshingly totally different strategy to different framework builders : they don’t attempt to be intelligent, they attempt to be clear.

On this article, I’ll present how one can construct multi-agent apps utilizing OpenAI’s open-source SDK.

We’ll see the way to assemble a easy single-agent app after which go on to discover multi-agent configurations. We’ll cowl tool-calling, linear and hierarchical configurations, handoffs from one agent to a different and utilizing brokers as instruments.

Particularly, we are going to see the next examples:

• A easy name to an agent
• A tool-using agent
• Handoffs from one agent to a different
• Handoffs to a number of brokers
• Utilizing brokers as instruments
• Hierarchical agent orchestration utilizing brokers as instruments

The agent SDK

The agent SDK relies on a handful of ideas important to agentic and multi-agent programs and builds a framework round them — it replaces Swarm, an academic framework developed by OpenAI, the place these ideas had been recognized and carried out. The Agent SDK builds upon and expands Swarm whereas sustaining its founding ideas of being light-weight and easy.

Easy it could be, however you’ll be able to assemble subtle agent-based programs with this framework the place brokers use instruments (which could be different brokers), hand off to different brokers, and could be orchestrated in any variety of intelligent methods.

Set up is by way of pip, or your most well-liked package deal administration device, and the package deal known as openai-agents. I favour UV, so to start out a brand new challenge, I might do one thing like the next.

uv init agentTest
cd agentTest
uv add openai-agents

A easy name to an agent

A easy agent name is proven within the diagram beneath.

It is a information circulate diagram that reveals the working agent as a course of with information flowing out and in. The circulate that begins the method is the consumer immediate; the agent makes a number of calls to the LLM and receives responses. When it has accomplished its job, it outputs the agent response.

Beneath we see the code for a primary program that makes use of the SDK to implement this circulate. It instantiates an agent, offers it a reputation and a few directions; it then runs it and prints the end result. It’s just like the primary instance from OpenAI’s documentation, however right here we are going to create a Streamlit app.

First, we import the libraries.

import streamlit as st
import asyncio from brokers 
import Agent, Runner 

We’d like the Streamlit package deal, in fact, and asyncio as a result of we are going to use its performance to attend for the agent to finish earlier than continuing. Subsequent, we import the minimal from the brokers package deal, Agent (to create an agent) and Runner (to run the agent).

Beneath, we outline the code to create and run the agent.

agent = Agent(identify="Assistant", directions="You're a useful assistant")

async def run_agent(input_string):
    end result = await Runner.run(agent, input_string)
    return end result.final_output 

This code makes use of the default mannequin from OpenAI (and assumes you’ve got a legitimate API key set as an surroundings variable – and you’ll, in fact, be charged , however for our functions right here, it gained’t be a lot.  I’ve solely spent a number of tens of cents on this).

First, we instantiate an agent referred to as “Assistant” with some easy directions, then we outline an asynchronous perform that can run it with a string (the question) offered by the consumer.

The run perform is asynchronous; we have to look ahead to the LLM to finish earlier than we proceed, and so we are going to run the perform utilizing asyncio.

We outline the consumer interface with Streamlit features.

st.title("Easy Agent SDK Question")

user_input = st.text_input("Enter a question and press 'Ship':")

st.write("Response:")
response_container = st.container(peak=300, border=True)

if st.button("Ship"):
    response = asyncio.run(run_agent(user_input))
    with response_container:
        st.markdown(response)

That is principally self-explanatory. The consumer is prompted to enter a question and press the ‘Ship’ button. When the button is pressed run_agent is run by way of a name to asyncio.run. The result’s displayed in a scrollable container. Beneath is a screenshot of a pattern run.

Your end result might differ (LLMs are famend for not giving the identical reply twice).

To outline an agent, give it a reputation and a few directions. Operating can also be easy; go within the agent and a question. Operating the agent begins a loop that completes when a remaining reply is reached. This instance is easy and doesn’t must run by way of the loop greater than as soon as, however an agent that calls instruments would possibly must undergo a number of iterations earlier than a solution is finalised.

The result’s simply displayed. As we are able to see, it’s the final_output attribute of the worth that’s returned from the Runner.

This program makes use of default values for a number of parameters that could possibly be set manually, such because the mannequin identify and the temperature setting for the LLM. The Agent SDK additionally makes use of the Responses API by default. That’s an OpenAI-only API (thus far, no less than), so if it’s essential use the SDK with one other LLM, you need to swap to the extra extensively supported Chat Completions API.

from brokers import set_default_openai_api
set_default_openai_api("chat_completions")

Initially, and for simplicity, we’ll use the default Response API.

A tool-using agent

Brokers can use instruments, and the agent, along with the LLM, decides which instruments, if any, it wants to make use of.

Here’s a information circulate diagram that reveals a tool-using agent.

It’s just like the straightforward agent, however we are able to see an extra course of, the device, that the agent utilises. When the agent makes a name to the LLM, the response will point out whether or not or not a device must be used. If it does, then the agent will make that decision and submit the end result again to the LLM. Once more, the response from the LLM will point out whether or not one other device name is critical. The agent will proceed this loop till the LLM now not requires the enter from a device. At this level, the agent can reply to the consumer.

Beneath is the code for a single agent utilizing a single device.

This system consists of 4 components:

• The imports from the Brokers library and wikipedia (which will likely be used as a device).
• The definition of a device — that is merely a perform with the @function_tool decorator.
• The definition of the agent that makes use of the device.
• Operating the agent and printing the lead to a Streamlit app, as earlier than.

import streamlit as st
import asyncio
from brokers import Agent, Runner, function_tool
import wikipedia

@function_tool
def wikipedia_lookup(q: str) -> str:
    """Lookup a question in Wikipedia and return the end result"""
    return wikipedia.web page(q).abstract

research_agent = Agent(
    identify="Analysis agent",
    directions="""You analysis subjects utilizing Wikipedia and report on 
                    the outcomes. """,
    mannequin="o4-mini",
    instruments=[wikipedia_lookup],
)

async def run_agent(input_string):
    end result = await Runner.run(research_agent, input_string)
    return end result.final_output

# Streamlit UI

st.title("Easy Software-using Agent")
st.write("This agent makes use of Wikipedia to lookup data.")

user_input = st.text_input("Enter a question and press 'Ship':")

st.write("Response:")
response_container = st.container(peak=300, border=True)

if st.button("Ship"):
    response = asyncio.run(run_agent(user_input))
    with response_container:
        st.markdown(response)

The device appears up a Wikipedia web page and returns a abstract by way of a typical name to a library perform. Word that we’ve used kind hints and a docstring to explain the perform so the agent can work out the way to use it.

Subsequent is the definition of the agent, and right here we see that there are extra parameters than earlier: we specify the mannequin that we wish to use and a listing of instruments (there’s just one on this checklist).

Operating and printing the result’s as earlier than, and it dutifully returns a solution (the peak of the Eiffel Tower).

That may be a easy take a look at of the tool-using agent, which solely requires a single lookup. A extra complicated question might use a device greater than as soon as to gather the data.

For instance, I requested, “Discover the identify of the well-known tower in Paris, discover its peak after which discover the date of delivery of its creator“. This required two device calls, one to get details about the Eiffel Tower and the second to seek out when Gustav Eiffel was born.

This course of shouldn’t be mirrored within the remaining output, however we are able to see the levels that the agent went by way of by viewing the uncooked messages within the agent’s end result. I printed end result.raw_messages for the question above, and the result’s proven beneath.

[
0:"ModelResponse(output=[ResponseReasoningItem(id='rs_6849968a438081a2b2fda44aa5bc775e073e3026529570c1', summary=[], kind='reasoning', standing=None), 
ResponseFunctionToolCall(arguments='{"q":"Eiffel Tower"}', call_id='call_w1iL6fHcVqbPFE1kAuCGPFok', identify='wikipedia_lookup', kind='function_call', id='fc_6849968c0c4481a29a1b6c0ad80fba54073e3026529570c1', standing='accomplished')], utilization=Utilization(requests=1, input_tokens=111, output_tokens=214, total_tokens=325), 
response_id='resp_68499689c60881a2af6411d137c13d82073e3026529570c1')"

1:"ModelResponse(output=[ResponseReasoningItem(id='rs_6849968e00ec81a280bf53dcd30842b1073e3026529570c1', summary=[], kind='reasoning', standing=None), 
ResponseFunctionToolCall(arguments='{"q":"Gustave Eiffel"}', call_id='call_DfYTuEjjBMulsRNeCZaqvV8w', identify='wikipedia_lookup', kind='function_call', id='fc_6849968e74ac81a298dc17d8be4012a7073e3026529570c1', standing='accomplished')], utilization=Utilization(requests=1, input_tokens=940, output_tokens=23, total_tokens=963), 
response_id='resp_6849968d7c3081a2acd7b837cfee5672073e3026529570c1')"

2:"ModelResponse(output=[ResponseReasoningItem(id='rs_68499690e33c81a2b0bda68a99380840073e3026529570c1', summary=[], kind='reasoning', standing=None), 
ResponseOutputMessage(id='msg_6849969221a081a28ede4c52ea34aa54073e3026529570c1', content material=[ResponseOutputText(annotations=[], textual content='The well-known tower in Paris is the Eiffel Tower.  n• Top: 330 metres (1,083 ft) tall  n• Creator: Alexandre Gustave Eiffel, born 15 December 1832', kind='output_text')], function='assistant', standing='accomplished', kind='message')], utilization=Utilization(requests=1, input_tokens=1190, output_tokens=178, total_tokens=1368), 
response_id='resp_6849968ff15481a292939a6eed683216073e3026529570c1')"
]

You’ll be able to see that there are three responses: the primary two are the results of the 2 device calls, and the final is the ultimate output, which is generated from the data derived from the device calls.

We’ll see instruments once more shortly after we use brokers as instruments, however now we’re going to take into account how we are able to use a number of brokers that cooperate.

A number of brokers

Many agent functions solely require a single agent, and these are already an extended step past easy chat completions that you simply discover within the LLM chat interfaces, akin to ChatGPT. Agents run in loops and might use instruments, making even a single agent fairly highly effective. Nevertheless, a number of brokers working collectively can obtain much more complicated behaviours.

In step with its easy philosophy, OpenAI doesn’t try to include agent orchestration abstractions like another frameworks. However regardless of its easy design, it helps the development of each easy and complicated configurations.

First, we’ll have a look at handoffs the place one agent passes management to a different. After that, we’ll see how brokers could be mixed hierarchically.

Handoffs

When an agent decides that it has accomplished its job and passes data to a different agent for additional work, that’s termed a handoff.

There are two basic methods of attaining a handoff: with an agentic handoff, your entire message historical past is handed from one agent to a different. It’s a bit like while you name the financial institution however the individual you first converse to doesn’t know your explicit circumstances, and so passes you on to somebody who does. The distinction is that, within the case of the AI agent, the brand new agent has a file of all that was mentioned to the earlier one.

The second technique is a programmatic handoff. That is the place solely the required data offered by one agent is handed to a different (by way of standard programming strategies).

Let’s have a look at programmatic handoffs first.

Programmatic handoffs

Typically the brand new agent doesn’t must know your entire historical past of a transaction; maybe solely the ultimate result’s required. On this case, as an alternative of a full handoff, you’ll be able to prepare a programmatic handoff the place solely the related information is handed to the second agent.

The diagram reveals a generic programmatic handoff between two brokers.

Beneath is an instance of this performance, the place one agent finds details about a subject and one other takes that data and writes an article that’s appropriate for youths.

To maintain issues easy, we gained’t use our Wikipedia device on this instance; as an alternative, we depend on the LLM’s data.

import streamlit as st
import asyncio
from brokers import Agent, Runner

writer_agent = Agent(
    identify="Author agent",
    directions=f"""Re-write the article in order that it's appropriate for youths
                     aged round 8. Be enthusiastic in regards to the subject -
                     all the pieces is an journey!""",
    mannequin="o4-mini",
)

researcher_agent = Agent(
    identify="Analysis agent",
    directions=f"""You analysis subjects and report on the outcomes.""",
    mannequin="o4-mini",
)

async def run_agent(input_string):
    end result = await Runner.run(researcher_agent, input_string)
    result2 = await Runner.run(writer_agent, end result.final_output)
    return result2

# Streamlit UI

st.title("Author Agent")
st.write("Write stuff for youths.")

user_input = st.text_input("Enter a question and press 'Ship':")

st.write("Response:")
response_container = st.container(peak=300, border=True)

if st.button("Ship"):
    response = asyncio.run(run_agent(user_input))
    with response_container:
        st.markdown(response.final_output)
    st.write(response)
    st.json(response.raw_responses)

Within the code above, we outline two brokers: one researches a subject and the opposite produces textual content appropriate for youths.

This system doesn’t depend on any particular SDK features; it merely runs one agent, will get the output in end result and makes use of it because the enter for the subsequent agent (output in result2). It’s identical to utilizing the output of 1 perform because the enter for the subsequent in standard programming. Certainly, that’s exactly what it’s.

Agentic handoffs

Nevertheless, generally an agent must know the historical past of what occurred beforehand. That’s the place the OpenAI Brokers Handoffs are available.

Beneath is the info circulate diagram that represents the Agentic Handoff. You will notice that it is rather just like the Programmatic Handoff; the distinction is the info being transferred to the second agent, and likewise, there’s a attainable output from the primary agent when the handoff shouldn’t be required.

The code can also be just like the earlier instance. I’ve tweaked the directions barely, however the primary distinction is the handoffs checklist in researcher_agent. This isn’t dissimilar to the way in which we declare instruments.

The Analysis Agent has been allowed at hand off to the Child’s Author Agent when it has accomplished its work. The impact of that is that the Child’s Author Agent not solely takes over management of the processing but in addition has data of what the Analysis Agent did, in addition to the unique immediate.

Nevertheless, there’s one other main distinction. It’s as much as the agent to find out whether or not the handoff takes place or not. Within the instance run beneath, I’ve instructed the agent to write down one thing appropriate for youths, and so it palms off to the Youngsters’ Author Agent. If I had not informed it to try this, it might have merely returned the unique textual content.

import streamlit as st
import asyncio
from brokers import Agent, Runner

kids_writer_agent = Agent(
    identify="Youngsters Author Agent",
    directions=f"""Re-write the article in order that it's appropriate for youths aged round 8. 
                     Be enthusiastic in regards to the subject - all the pieces is an journey!""",
    mannequin="o4-mini",
)

researcher_agent = Agent(
    identify="Analysis agent",
    directions=f"""Reply the question and report the outcomes.""",
    mannequin="o4-mini",
    handoffs = [kids_writer_agent]
)

async def run_agent(input_string):
    end result = await Runner.run(researcher_agent, input_string)
    return end result

# Streamlit UI

st.title("Author Agent2")
st.write("Write stuff for youths.")

user_input = st.text_input("Enter a question and press 'Ship':")

st.write("Response:")
response_container = st.container(peak=300, border=True)

if st.button("Ship"):
    response = asyncio.run(run_agent(user_input))
    with response_container:
        st.markdown(response.final_output)
    st.write(response)
    st.json(response.raw_responses)

It’s not within the screenshot, however I’ve added code to output the response and the raw_responses to be able to see the handoff in operation should you run the code your self.

Beneath is a screenshot of this agent.

An agent can have a listing of handoffs at its disposal, and it’ll intelligently select the proper agent (or none) at hand off to. You’ll be able to see how this could be helpful in a customer support state of affairs the place a troublesome buyer question could be escalated by way of a collection of extra knowledgeable brokers, every of whom wants to concentrate on the question historical past.

We’ll now have a look at how we are able to use handoffs that contain a number of brokers.

Handoffs to a number of brokers

We are going to now see a brand new model of the earlier program the place the Analysis Agent chooses at hand off to totally different brokers relying on the reader’s age.

The agent’s job is to supply textual content for 3 audiences: adults, youngsters and youngsters. The Analysis Agent will collect data after which hand it off to one in all three different brokers. Right here is the info circulate (notice that I’ve excluded the hyperlinks to an LLM for readability – every agent communicates with an LLM, however we are able to take into account that as an inside perform of the agent).

And right here is the code.

import streamlit as st
import asyncio

from brokers import Agent, Runner, handoff

adult_writer_agent = Agent(
    identify="Grownup Author Agent",
    directions=f"""Write the article based mostly on the data provided that it's appropriate for adults keen on tradition.
                    """, 
    mannequin="o4-mini",
)

teen_writer_agent = Agent(
    identify="Teen Author Agent",
    directions=f"""Write the article based mostly on the data provided that it's appropriate for youngsters who wish to have a cool time.
                    """, 
    mannequin="o4-mini",
)

kid_writer_agent = Agent(
    identify="Child Author Agent",
    directions=f"""Write the article based mostly on the data provided that it's appropriate for youths of round 8 years outdated. 
                    Be enthusiastic!
                    """, 
    mannequin="o4-mini",
)

researcher_agent = Agent(
    identify="Analysis agent",
    directions=f"""Discover data on the subject(s) given.""",

    mannequin="o4-mini",
    handoffs = [kid_writer_agent, teen_writer_agent, adult_writer_agent]
)

async def run_agent(input_string):
    end result = await Runner.run(researcher_agent, input_string)
    return end result

# Streamlit UI

st.title("Author Agent3")
st.write("Write stuff for adults, youngsters or youngsters.")

user_input = st.text_input("Enter a question and press 'Ship':")

st.write("Response:")
response_container = st.container(peak=300, border=True)

if st.button("Ship"):
    response = asyncio.run(run_agent(user_input))
    with response_container:
        st.markdown(response.final_output)
    st.write(response)
    st.json(response.raw_responses)

This system’s construction is comparable, however now now we have a set of brokers at hand off to and a listing of them within the Analysis Agent. The directions within the numerous brokers are self-explanatory, and this system will appropriately reply to a immediate akin to “Write an essay about Paris, France for youths” or “…for youngsters” or “…for adults”. The Analysis Agent will appropriately select the suitable Author Agent for the duty.

The screenshot beneath reveals an instance of writing for youngsters.

The prompts offered on this instance are easy. Extra subtle prompts would possible yield a greater and extra constant end result, however the purpose right here is to indicate the methods fairly than to construct a intelligent app.

That’s one kind of collaboration; one other is to make use of different brokers as instruments. This isn’t too dissimilar to the programmatic handoff we noticed earlier.

Brokers as instruments

Operating an agent is asking a perform in the identical approach as calling a device. So why not use brokers as clever instruments?

As a substitute of giving management over to a brand new agent, we use it as a perform that we go data to and get data again from.

Beneath is a knowledge circulate diagram that illustrates the thought. Not like a handoff, the primary agent doesn’t go general management to a different agent; as an alternative, it intelligently chooses to name an agent as if it had been a device. The referred to as agent does its job after which passes management again to the calling agent. Once more, the info flows to an LLM have been omitted for readability.

Beneath is a screenshot of a modified model of the earlier program. We modified the character of the app just a little. The primary agent is now a journey agent; it expects the consumer to offer it a vacation spot and the age group for which it ought to write. The UI is modified in order that the age group is chosen by way of a radio button. The textual content enter area needs to be a vacation spot.

A lot of modifications have been made to the logic of the app. The UI adjustments the way in which the data is enter, and that is mirrored in the way in which that the immediate is constructed – we use an f-string to include the 2 items of information into the immediate.

Moreover, we now have an additional agent that codecs the textual content. The opposite brokers are related (however notice that the prompts have been refined), and we additionally use a structured output to make sure that the textual content that we output is exactly what we anticipate.

Basically, although, we see that the author brokers and the formatting agent are specified as instruments within the researcher agent.

import streamlit as st
import asyncio
from brokers import Agent, Runner, function_tool
from pydantic import BaseModel

class PRArticle(BaseModel):
    article_text: str
    commentary: str

adult_writer_agent = Agent(
    identify="Grownup Author Agent",
    directions="""Write the article based mostly on the data provided that it's appropriate for adults keen on tradition. 
                    Be mature.""", 
    mannequin="gpt-4o",
)

teen_writer_agent = Agent(
    identify="Teen Author Agent",
    directions="""Write the article based mostly on the data provided that it's appropriate for youngsters who wish to have a great time. 
                    Be cool!""", 
    mannequin="gpt-4o",
)

kid_writer_agent = Agent(
    identify="Child Author Agent",
    directions="""Write the article based mostly on the data provided that it's appropriate for youths of round 8 years outdated. 
                    Be enthusiastic!""", 
    mannequin="gpt-4o",
)

format_agent = Agent(
    identify="Format Agent",
    directions=f"""Edit the article so as to add a title and subtitles and make sure the textual content is formatted as Markdown. Return solely the textual content of article.""", 
    mannequin="gpt-4o",
)

researcher_agent = Agent(
    identify="Analysis agent",
    directions="""You're a Journey Agent who will discover helpful data on your prospects of all ages.
                    Discover data on the vacation spot(s) given. 
                    When you've got a end result ship it to the suitable author agent to supply a brief PR textual content.
                    When you've got the end result ship it to the Format agent for remaining processing.
                    """,
    mannequin="gpt-4o",
    instruments = [kid_writer_agent.as_tool(
                tool_name="kids_article_writer",
                tool_description="Write an essay for kids",), 
            teen_writer_agent.as_tool(
                tool_name="teen_article_writer",
                tool_description="Write an essay for teens",), 
            adult_writer_agent.as_tool(
                tool_name="adult_article_writer",
                tool_description="Write an essay for adults",),
            format_agent.as_tool(
                tool_name="format_article",
                tool_description="Add titles and subtitles and format as Markdown",
        ),],
    output_type = PRArticle
)

async def run_agent(input_string):
    end result = await Runner.run(researcher_agent, input_string)
    return end result

# Streamlit UI

st.title("Journey Agent")
st.write("The journey agent will write about locations for various audiences.")

vacation spot = st.text_input("Enter a vacation spot, choose the age group and press 'Ship':")
age_group = st.radio(
    "What age group is the reader?",
    ["Adult", "Teenager", "Child"],
    horizontal=True,
)

st.write("Response:")
response_container = st.container(peak=500, border=True)

if st.button("Ship"):
    response = asyncio.run(run_agent(f"The vacation spot is {vacation spot} and reader the age group is {age_group}"))
    with response_container:
        st.markdown(response.final_output.article_text)
    st.write(response)
    st.json(response.raw_responses)

The instruments checklist is just a little totally different to the one we noticed earlier:

• The device identify is the agent identify plus .agent_as_tool(), a way that makes the agent appropriate with different instruments .
• The device wants a few parameters — a reputation and an outline.

One different addition, which could be very helpful, is using structured outputs, as talked about above. This separates the textual content that we would like from some other commentary that the LLM would possibly wish to insert. For those who run the code, you’ll be able to see within the raw_responses the extra data that the LLM generates.

Utilizing structured outputs helps to supply constant outcomes and solves an issue that could be a explicit bugbear of mine.

I’ve requested the output to be run by way of a formatter agent that can construction the end result as Markdown. It is dependent upon the LLM, it is dependent upon the immediate, and who is aware of, possibly it is dependent upon the time of day or the climate, however each time I believe I’ve bought it proper, an LLM will instantly insert Markdown fencing. So as an alternative of a clear:

## It is a header

That is some textual content

I as an alternative get:

Right here is your textual content formatted as Markdown:


''' Markdown  

# It is a header

That is some textual content  
'''

Infuriating!

Anyway, the reply appears to be to make use of structured outputs. For those who requested it to format the response because the textual content of what you need, plus a second area referred to as ‘commentary’ or some such factor, it seems to do the precise factor. Any extraneous stuff the LLM decides to spout goes within the second area, and the unadulterated Markdown goes within the textual content area.

OK, Shakespeare, it isn’t: adjusting the directions in order that they’re extra detailed would possibly give higher outcomes (the present prompts are quite simple). However it works properly sufficient as an instance the tactic.

Conclusion

That’s scratched the floor of OpenAI’s Brokers SDK. Thanks for studying, and I hope you discovered it helpful. We have now seen the way to create brokers and the way to mix them in numerous methods, and we took a really fast have a look at structured outputs.

The examples are, in fact, easy, however I hope they illustrate the simple approach that brokers could be orchestrated merely with out resorting to complicated abstractions and unwieldy frameworks.

The code right here makes use of the Response API as a result of that’s the default. Nevertheless, it ought to run the identical approach with the Completions API, as properly. Which suggests that you’re not restricted to ChatGPT and, with a little bit of jiggery-pokery, this SDK can be utilized with any LLM that helps the OpenAI Completions API.

There’s loads extra to seek out out in OpenAI’s documentation.

• Photos are by the writer until in any other case said.