diff --git a/sources/platform/integrations/ai/agno.md b/sources/platform/integrations/ai/agno.md index 70722f095..786efe190 100644 --- a/sources/platform/integrations/ai/agno.md +++ b/sources/platform/integrations/ai/agno.md @@ -2,7 +2,7 @@ title: Agno Integration sidebar_label: Agno description: Integrate Apify with Agno to power AI agents with web scraping, automation, and data insights. -sidebar_position: 17 +sidebar_position: 19 slug: /integrations/agno --- diff --git a/sources/platform/integrations/ai/aws_bedrock.md b/sources/platform/integrations/ai/aws_bedrock.md index 35b94944c..0b22a86f2 100644 --- a/sources/platform/integrations/ai/aws_bedrock.md +++ b/sources/platform/integrations/ai/aws_bedrock.md @@ -2,7 +2,7 @@ title: Amazon Bedrock integrations sidebar_label: Amazon Bedrock description: Learn how to integrate Apify with Amazon Bedrock Agents to provide web data for AI agents -sidebar_position: 11 +sidebar_position: 15 slug: /integrations/aws_bedrock --- diff --git a/sources/platform/integrations/ai/chatgpt.md b/sources/platform/integrations/ai/chatgpt.md new file mode 100644 index 000000000..a29674f9c --- /dev/null +++ b/sources/platform/integrations/ai/chatgpt.md @@ -0,0 +1,107 @@ +--- +title: ChatGPT Integration +sidebar_label: ChatGPT +description: Learn how to integrate Apify with ChatGPT to provide web context in real-time. +sidebar_position: 12 +slug: /integrations/chatgpt +--- + +**Learn how to integrate Apify Actors with ChatGPT to provide web context in real-time.** + +--- + +The _ChatGPT_ integration enables you to connect ChatGPT to Apify's extensive library of Actors through the [Model Context Protocol (MCP)](https://openai.com/research/model-context-protocol). This allows ChatGPT to use Apify tools and Actors directly in conversations, providing real-time web data access and automation capabilities. + +In this tutorial, you'll learn how to connect ChatGPT to the **Apify MCP server** using a custom connector. + +## Prerequisites + +Before connecting ChatGPT to Apify, you'll need: + +- _An Apify account_ - Sign up for an Apify account, if you don't have one. +- _Apify API token_ - Get your API token from the **Integrations** section in [Apify Console](https://console.apify.com/account#/integrations). This token authorizes the MCP server to run Actors on your behalf. Make sure to keep it secure. +- _An OpenAI account with access to ChatGPT_ - You need an OpenAI account to use ChatGPT. +- _ChatGPT with Developer Mode enabled_ - You must enable [Developer Mode](https://platform.openai.com/docs/guides/developer-mode) to add custom connectors. + +## Enable developer mode in ChatGPT + +You must enable [Developer Mode](https://platform.openai.com/docs/guides/developer-mode) in ChatGPT to add custom connectors like the Apify MCP server. +When Developer Mode is active, the message input box is outlined in orange. + +## Create an MCP connector + +1. In ChatGPT, go to **Settings → Apps & Connectors → Create**. If you don't see the **Create** button, enable developer mode or reload the page. + +2. Fill in the following fields: + + - **Name** – a user-facing title, e.g., `apify-mcp` + - **Description** – a short description of what the connector does + - **MCP Server URL** – choose one of the following: + - `https://mcp.apify.com` — use the default set of Apify tools + - `https://mcp.apify.com?tools=actors,docs,mtrunkat/url-list-download-html` — use specific tools + (See [mcp.apify.com](https://mcp.apify.com) for details.) + - **Authentication** – OAuth, you don’t need to provide a client ID or secret. + +![ChatGPT Create connect](../images/chatgpt-connector.png) + +:::caution ChatGPT currently cannot change selected tools after connector creation +At the moment, ChatGPT does not allow modifying the selected tools after the connector is created. +If you need to add or remove tools later, you’ll need to create a new connector. + +If you try to use a connector with social media scrapers like Instagram or TikTok, you may see the error: +"Something went wrong with setting up the connection." +This error typically occurs for social media scrapers like Instagram, TikTok, etc. +But you can still use these Actors with ChatGPT by including them when initially setting up the connector. + +::: + +## Authorize access + +Click **Create** to proceed to the authentication page. +You’ll be redirected to the Apify website to authorize ChatGPT to access your Apify account. +Ensure you're logged into the correct Apify account before approving access. + +Once authorized, you’ll return to ChatGPT and see a success message with a list of tools available from the Apify MCP server. + +## Try the MCP connector in ChatGPT + +Once your connector is ready: + +1. Open a **new chat** in ChatGPT. +2. Click the **+** button near the message composer and select **More**. +3. Choose your **Apify MCP connector** to add it to the conversation. +4. Ask ChatGPT to use Apify tools, for example: + + > “Search the web and summarize recent trends in AI agents” + +You’ll need to grant permission for each Apify tool when it’s used for the first time. +You should see ChatGPT calling Apify tools — such as the [RAG Web Browser](https://apify.com/apify/rag-web-browser) — to gather information. + +![ChatGPT Apify tools](../images/chatgpt-with-rag-web-browser.png) + +## Use any Actor with ChatGPT + +You can use any [Apify Actor](https://apify.com/store) with ChatGPT. By default, the Apify MCP server exposes a set of tools that lets you search and use Actors directly. + +**Example query:** + +> "Find and run an Actor that scrapes Instagram profiles, and get the profile of @natgeo" + + +## Limitations + +- MCP integration in ChatGPT is still in **beta** and may have some limitations or bugs. +- Tool selection and execution can be **slow**, especially with the latest GPT models. +- **Custom connectors** are only available in ChatGPT **Developer Mode**. + +## Related integrations + +- [OpenAI Assistants integration](/platform/integrations/openai-assistants) — Use Apify Actors with OpenAI Assistants API via function calling +- [OpenAI Agents SDK integration](/platform/integrations/openai-agents) — Integrate Apify MCP server with OpenAI Agents SDK + +## Resources + +- [ChatGPT Developer Mode](https://platform.openai.com/docs/guides/developer-mode) - Learn how to enable Developer Mode in ChatGPT +- [Connectors and MCP servers](https://platform.openai.com/docs/guides/tools-connectors-mcp) - Official OpenAI documentation on using MCP servers with ChatGPT +- [Apify MCP server](https://mcp.apify.com) - Interactive configuration tool for the Apify MCP server +- [Apify MCP documentation](/platform/integrations/mcp) - Complete guide to using the Apify MCP server diff --git a/sources/platform/integrations/ai/crewai.md b/sources/platform/integrations/ai/crewai.md index bee4e93ea..1e7f6336a 100644 --- a/sources/platform/integrations/ai/crewai.md +++ b/sources/platform/integrations/ai/crewai.md @@ -2,7 +2,7 @@ title: 🤖🚀 CrewAI integration sidebar_label: CrewAI description: Learn how to build AI Agents with Apify and CrewAI 🤖🚀. -sidebar_position: 2 +sidebar_position: 3 slug: /integrations/crewai --- diff --git a/sources/platform/integrations/ai/flowise.md b/sources/platform/integrations/ai/flowise.md index bc76b41fc..fbffcfa60 100644 --- a/sources/platform/integrations/ai/flowise.md +++ b/sources/platform/integrations/ai/flowise.md @@ -2,7 +2,7 @@ title: Flowise integration description: Learn how to integrate Apify with Flowise. sidebar_label: Flowise -sidebar_position: 9 +sidebar_position: 10 slug: /integrations/flowise --- diff --git a/sources/platform/integrations/ai/haystack.md b/sources/platform/integrations/ai/haystack.md index 32445ada6..2c4993b63 100644 --- a/sources/platform/integrations/ai/haystack.md +++ b/sources/platform/integrations/ai/haystack.md @@ -2,7 +2,7 @@ title: Haystack integration sidebar_label: Haystack description: Learn how to integrate Apify with Haystack to work with web data in the Haystack ecosystem. -sidebar_position: 3 +sidebar_position: 4 slug: /integrations/haystack --- diff --git a/sources/platform/integrations/ai/langchain.md b/sources/platform/integrations/ai/langchain.md index 2f206369a..8d78ea758 100644 --- a/sources/platform/integrations/ai/langchain.md +++ b/sources/platform/integrations/ai/langchain.md @@ -2,7 +2,7 @@ title: 🦜🔗 LangChain integration sidebar_label: LangChain description: Learn how to integrate Apify with 🦜🔗 LangChain, in order to feed vector databases and LLMs with data crawled from the web. -sidebar_position: 4 +sidebar_position: 5 slug: /integrations/langchain --- diff --git a/sources/platform/integrations/ai/langflow.md b/sources/platform/integrations/ai/langflow.md index 331b61a05..e50fa72b7 100644 --- a/sources/platform/integrations/ai/langflow.md +++ b/sources/platform/integrations/ai/langflow.md @@ -2,7 +2,7 @@ title: Langflow integration sidebar_label: Langflow description: Learn how to integrate Apify with Langflow low-code tool to build powerful AI agents and workflows that can use any API, model, or database. -sidebar_position: 5 +sidebar_position: 6 slug: /integrations/langflow --- diff --git a/sources/platform/integrations/ai/langgraph.md b/sources/platform/integrations/ai/langgraph.md index 8690ed08f..27d76d9e4 100644 --- a/sources/platform/integrations/ai/langgraph.md +++ b/sources/platform/integrations/ai/langgraph.md @@ -2,7 +2,7 @@ title: 🦜🔘➡️ LangGraph integration sidebar_label: LangGraph description: Learn how to build AI Agents with Apify and LangGraph 🦜🔘➡️. -sidebar_position: 6 +sidebar_position: 8 slug: /integrations/langgraph --- diff --git a/sources/platform/integrations/ai/lindy.md b/sources/platform/integrations/ai/lindy.md index ccf796b9f..6275507d5 100644 --- a/sources/platform/integrations/ai/lindy.md +++ b/sources/platform/integrations/ai/lindy.md @@ -2,7 +2,7 @@ title: Lindy integration description: Learn how to integrate Apify with Lindy. sidebar_label: Lindy -sidebar_position: 7 +sidebar_position: 9 slug: /integrations/lindy --- diff --git a/sources/platform/integrations/ai/llama.md b/sources/platform/integrations/ai/llama.md index 3a7a642a0..c83d86b65 100644 --- a/sources/platform/integrations/ai/llama.md +++ b/sources/platform/integrations/ai/llama.md @@ -2,7 +2,7 @@ title: LlamaIndex integration description: Learn how to integrate Apify with LlamaIndex in order to feed vector databases and LLMs with data crawled from the web. sidebar_label: LlamaIndex -sidebar_position: 6 +sidebar_position: 7 slug: /integrations/llama-index --- diff --git a/sources/platform/integrations/ai/mastra.md b/sources/platform/integrations/ai/mastra.md index 37e8a36ed..1de306904 100644 --- a/sources/platform/integrations/ai/mastra.md +++ b/sources/platform/integrations/ai/mastra.md @@ -2,7 +2,7 @@ title: Mastra MCP integration sidebar_label: Mastra description: Learn how to build AI Agents with Mastra via Apify Actors MCP server -sidebar_position: 10 +sidebar_position: 11 slug: /integrations/mastra --- diff --git a/sources/platform/integrations/ai/milvus.md b/sources/platform/integrations/ai/milvus.md index 0ed1e293b..af1919832 100644 --- a/sources/platform/integrations/ai/milvus.md +++ b/sources/platform/integrations/ai/milvus.md @@ -2,7 +2,7 @@ title: Milvus integration description: Learn how to integrate Apify with Milvus (Zilliz) to save data scraped from the websites into the Milvus vector database. sidebar_label: Milvus -sidebar_position: 13 +sidebar_position: 16 slug: /integrations/milvus toc_min_heading_level: 2 toc_max_heading_level: 4 diff --git a/sources/platform/integrations/ai/openai_agents.md b/sources/platform/integrations/ai/openai_agents.md new file mode 100644 index 000000000..a631d7ded --- /dev/null +++ b/sources/platform/integrations/ai/openai_agents.md @@ -0,0 +1,285 @@ +--- +title: OpenAI Agents SDK integration +sidebar_label: OpenAI Agents SDK +description: Learn how to integrate Apify with the OpenAI Agents SDK using the Model Context Protocol to build AI agents with web data access. +sidebar_position: 13 +slug: /integrations/openai-agents +--- + +The _OpenAI Agents Python SDK_ enables you to build AI agents powered by OpenAI's language models that can use tools, manage context, and interact with external systems through the [Model Context Protocol (MCP)](https://openai.com/research/model-context-protocol). By connecting to the Apify MCP server, your agents can access Apify's extensive library of Actors to perform web scraping, data extraction, and automation tasks in real time. + +For more details about the OpenAI Agents SDK, see the [official documentation](https://openai.github.io/openai-agents-python/). + +## Prerequisites + +Before integrating Apify with OpenAI Agents SDK, you'll need: + +- _An Apify account_ - Sign up for an Apify account, if you don't have one. +- _Apify API token_ - Get your API token from the **Integrations** section in [Apify Console](https://console.apify.com/account#/integrations). This token authorizes the MCP server to run Actors on your behalf. Make sure to keep it secure. +- _OpenAI API key_ - Get your API key from the [OpenAI platform](https://platform.openai.com/account/api-keys). You need this to use OpenAI Agents SDK. +- _Python packages_ - Install the required packages: + + ```bash + pip install agents openai + ``` + +### Building a web search agent with Apify MCP + +You can connect to the Apify MCP server using streamable HTTP with Bearer token authentication. Use your Apify API token by setting the `Authorization: Bearer ` header in the MCP server configuration. + +```python +import asyncio +import os + +from agents import Agent, Runner +from agents.mcp import MCPServerStreamableHttp + +os.environ["APIFY_TOKEN"] = "Your Apify API token" +os.environ["OPENAI_API_KEY"] = "Your OpenAI API key" + + +async def main() -> None: + # Create MCP server connection with Bearer token + async with MCPServerStreamableHttp( + name="Apify MCP Server", + params={ + "url": "https://mcp.apify.com", + "headers": {"Authorization": f"Bearer {os.environ['APIFY_TOKEN']}"}, + "timeout": 120, + }, + cache_tools_list=True, + max_retry_attempts=3, + ) as server: + # Create agent with MCP server + agent = Agent( + name="Assistant", + instructions="Use the Apify MCP tools to answer questions. Search the web when needed.", + mcp_servers=[server], + ) + + # Run the agent + result = await Runner.run(agent, "Search the web and summarize recent trends in AI agents") + print(result.final_output) + + +if __name__ == "__main__": + asyncio.run(main()) +``` + +#### Configuration options + +The `MCPServerStreamableHttp` connects to the Apify MCP server using streamable HTTP. Key configuration options: + +- `params`: Dictionary containing: + - `url`: The MCP server URL (`https://mcp.apify.com` for Apify) + - `headers`: Authentication headers (Bearer token for Apify) + - `timeout`: Request timeout in seconds +- `cache_tools_list`: Caches the tool list to reduce API calls +- `max_retry_attempts`: Number of retry attempts for failed requests + +:::note Tool execution may take some time + +The agent may take some time to execute tool calls, especially when using web scraping Actors or searching the web. Actor runs can take time to complete depending on their task complexity. + +::: + +### Using specific Actors + +You can configure the Apify MCP server to expose specific Actors by including them in the URL query parameters. For example, to use an Instagram scraper: + +```python +async with MCPServerStreamableHttp( + name="Apify MCP Server", + params={ + "url": "https://mcp.apify.com?tools=apify/instagram-scraper", + "headers": {"Authorization": f"Bearer {os.environ['APIFY_TOKEN']}"}, + "timeout": 120, + }, + cache_tools_list=True, +) as server: + agent = Agent( + name="Assistant", + instructions="Use the Instagram scraper to analyze Instagram profiles and posts.", + mcp_servers=[server], + ) + + # Run the agent + result = await Runner.run( + agent, "Get the latest posts from @natgeo Instagram profile and summarize the content" + ) + print(result.final_output) +``` + +:::tip Easy configuration + +Use the UI configurator `https://mcp.apify.com/` to select your tools visually, then copy the configuration to your code. + +::: + +## Examples + +### Web search agent + +This example demonstrates how to build an agent that can search the web and use Apify Actors to gather information: + +```python +import asyncio +import os + +from agents import Agent, Runner +from agents.mcp import MCPServerStreamableHttp + +os.environ["APIFY_TOKEN"] = "Your Apify API token" +os.environ["OPENAI_API_KEY"] = "Your OpenAI API key" + + +async def main() -> None: + # Create MCP server connection + async with MCPServerStreamableHttp( + name="Apify MCP Server", + params={ + "url": "https://mcp.apify.com", + "headers": {"Authorization": f"Bearer {os.environ['APIFY_TOKEN']}"}, + "timeout": 120, + }, + cache_tools_list=True, + max_retry_attempts=3, + ) as server: + # Create agent with MCP server + agent = Agent( + name="Assistant", + instructions="Use the Apify MCP tools to answer questions. Search the web when needed.", + mcp_servers=[server], + ) + + # Run the agent + result = await Runner.run(agent, "Search the web and summarize recent trends in AI agents") + print(result.final_output) + + +if __name__ == "__main__": + asyncio.run(main()) +``` + +### Instagram profile analysis + +This example shows how to use a specific Actor for Instagram profile analysis: + +```python +import asyncio +import os + +from agents import Agent, Runner +from agents.mcp import MCPServerStreamableHttp + +os.environ["APIFY_TOKEN"] = "Your Apify API token" +os.environ["OPENAI_API_KEY"] = "Your OpenAI API key" + + +async def main() -> None: + # Create MCP server connection with Instagram scraper + async with MCPServerStreamableHttp( + name="Apify MCP Server", + params={ + "url": "https://mcp.apify.com?tools=apify/instagram-scraper", + "headers": {"Authorization": f"Bearer {os.environ['APIFY_TOKEN']}"}, + "timeout": 120, + }, + cache_tools_list=True, + max_retry_attempts=3, + ) as server: + # Create agent with MCP server + agent = Agent( + name="Assistant", + instructions="Use the Instagram scraper to analyze Instagram profiles and posts.", + mcp_servers=[server], + ) + + # Run the agent + result = await Runner.run( + agent, "Get the latest posts from @natgeo Instagram profile and summarize the main themes" + ) + print(result.final_output) + + +if __name__ == "__main__": + asyncio.run(main()) +``` + +### Agent to test MCP servers + +You can use the OpenAI Agents SDK to test MCP servers and verify they're working correctly. The agent can list available tools and execute them to ensure proper functionality: + +```python +import asyncio +import os + +from agents import Agent, Runner +from agents.mcp import MCPServerStreamableHttp + +os.environ["APIFY_TOKEN"] = "Your Apify API token" +os.environ["OPENAI_API_KEY"] = "Your OpenAI API key" + + +async def main() -> None: + # Connect to Apify MCP server for testing + async with MCPServerStreamableHttp( + name="Apify MCP Server", + params={ + "url": "https://mcp.apify.com", + "headers": {"Authorization": f"Bearer {os.environ['APIFY_TOKEN']}"}, + "timeout": 120, + }, + cache_tools_list=True, + max_retry_attempts=3, + ) as server: + # List available tools + tools = await server.list_tools() + print(f"Available tools: {[tool.name for tool in tools]}") + + # Create a test agent + agent = Agent( + name="Tester", + instructions="Test the available MCP tools by calling them with appropriate parameters.", + mcp_servers=[server], + ) + + # Test a simple query + result = await Runner.run( + agent, "List all available tools and test the search-actors tool with query 'instagram'" + ) + print(result.final_output) + + +if __name__ == "__main__": + asyncio.run(main()) +``` + +For a comprehensive example with error handling and reporting, refer to the [OpenAI Agent MCP Tester](https://apify.com/jiri.spilka/openai-agent-mcp-tester) Actor. This Actor automates the testing process using OpenAI agents with a two-agent orchestrator pattern (planner and executor), generating detailed reports with pass/fail status for each tool. + +## Troubleshooting + +### Authentication errors + +- _Check your API token_: Verify that your Apify API token is correct. You can find it in the **Integrations** section of the [Apify Console](https://console.apify.com/account#/integrations). Without a valid token, the server cannot start Actor runs. +- _Set environment variables_: When running your agent, ensure you have set the `APIFY_TOKEN` and `OPENAI_API_KEY` environment variables. + +### Agent execution issues + +- _No response or long delays_: Actor runs can take time to complete depending on their task. If you're experiencing long delays, check the Actor's logs in Apify Console. The logs will provide insight into the Actor's status and show if it's processing a long operation or has encountered an error. +- _Tool execution timeout_: If tool calls are timing out, increase the `timeout` parameter in the `MCPServerStreamableHttp` configuration. + +## Related integrations + +- [ChatGPT integration](/platform/integrations/chatgpt) — Add Apify MCP server as a custom connector in ChatGPT +- [OpenAI Assistants integration](/platform/integrations/openai-assistants) — Use Apify Actors with OpenAI Assistants API via function calling + +## Resources + +- [OpenAI Agents Python SDK documentation](https://openai.github.io/openai-agents-python/) - Official documentation for the OpenAI Agents SDK +- [OpenAI Agents MCP guide](https://openai.github.io/openai-agents-python/mcp/) - Learn how to use MCP with OpenAI Agents SDK +- [OpenAI Agent MCP Tester Actor](https://apify.com/jiri.spilka/openai-agent-mcp-tester) - A specialized Actor for testing MCP server integration +- [OpenAI Agent MCP Tester GitHub repository](https://github.com/apify/openai-agent-mcp-tester) - Source code for the MCP tester Actor +- [Apify MCP server](https://mcp.apify.com) - Interactive configuration tool for the Apify MCP server +- [Apify MCP documentation](/platform/integrations/mcp) - Complete guide to using the Apify MCP server +- [Model Context Protocol specification](https://modelcontextprotocol.io/) - Learn about the MCP specification diff --git a/sources/platform/integrations/ai/openai_assistants.md b/sources/platform/integrations/ai/openai_assistants.md index 4074f77c5..708e1fb0f 100644 --- a/sources/platform/integrations/ai/openai_assistants.md +++ b/sources/platform/integrations/ai/openai_assistants.md @@ -2,7 +2,7 @@ title: OpenAI Assistants integration sidebar_label: OpenAI Assistants description: Learn how to integrate Apify with OpenAI Assistants to provide real-time search data and to save them into OpenAI Vector Store -sidebar_position: 12 +sidebar_position: 14 slug: /integrations/openai-assistants --- @@ -413,6 +413,11 @@ for m in client.beta.threads.messages.list(thread_id=run.thread_id): +## Related integrations + +- [ChatGPT integration](/platform/integrations/chatgpt) — Add Apify MCP server as a custom connector in ChatGPT +- [OpenAI Agents SDK integration](/platform/integrations/openai-agents) — Integrate Apify MCP server with OpenAI Agents SDK + ## Resources - [OpenAI Assistants](https://platform.openai.com/docs/assistants/overview) diff --git a/sources/platform/integrations/ai/pinecone.md b/sources/platform/integrations/ai/pinecone.md index e87b447cd..f2325a55b 100644 --- a/sources/platform/integrations/ai/pinecone.md +++ b/sources/platform/integrations/ai/pinecone.md @@ -2,7 +2,7 @@ title: Pinecone integration description: Learn how to integrate Apify with Pinecone to feed data crawled from the web into the Pinecone vector database. sidebar_label: Pinecone -sidebar_position: 14 +sidebar_position: 17 slug: /integrations/pinecone toc_min_heading_level: 2 toc_max_heading_level: 4 diff --git a/sources/platform/integrations/ai/qdrant.md b/sources/platform/integrations/ai/qdrant.md index 66f4e74c0..de8ad5bbd 100644 --- a/sources/platform/integrations/ai/qdrant.md +++ b/sources/platform/integrations/ai/qdrant.md @@ -2,7 +2,7 @@ title: Qdrant integration description: Learn how to integrate Apify with Qdrant to feed data crawled from the web into the Qdrant vector database. sidebar_label: Qdrant -sidebar_position: 15 +sidebar_position: 18 slug: /integrations/qdrant toc_min_heading_level: 2 toc_max_heading_level: 4 diff --git a/sources/platform/integrations/images/chatgpt-connector.png b/sources/platform/integrations/images/chatgpt-connector.png new file mode 100644 index 000000000..373dbbe31 Binary files /dev/null and b/sources/platform/integrations/images/chatgpt-connector.png differ diff --git a/sources/platform/integrations/images/chatgpt-with-rag-web-browser.png b/sources/platform/integrations/images/chatgpt-with-rag-web-browser.png new file mode 100644 index 000000000..e23210b62 Binary files /dev/null and b/sources/platform/integrations/images/chatgpt-with-rag-web-browser.png differ