Jockey, an open-source conversational video agent, has been significantly improved through the integration of Twelve Labs APIs and LangGraph. This combination aims to provide more intelligent and efficient video processing capabilities, according to a recent study LangChain blog post.
Overview of Twelve Labs APIs
Twelve Labs provides advanced video understanding APIs that extract rich insights and information directly from video content. These advanced video base models (VFMs) work natively with video, bypassing intermediate representations such as pre-generated captions. This allows for a more accurate and contextual understanding of video content, including images, audio, on-screen text and temporal relationships.
The APIs support various functionalities such as video search, classification, summary and question answering. They can be integrated into applications for content discovery, video editing automation, interactive video FAQs, and AI-generated highlights. With enterprise-grade security and scalability, Twelve Labs APIs open new possibilities for video-powered applications.
LangGraph v0.1 and LangGraph Cloud launch
LangChain has introduced LangGraph v0.1, a framework designed for building agentic and multi-agent applications with improved control and precision. Unlike its predecessor, LangChain AgentExecutor, LangGraph provides a flexible API for custom cognitive architectures, allowing developers to control code flow, prompts, and LLM calls. It also supports human-agent collaboration through a built-in persistence layer, enabling human approval before task execution and ‘time travel’ for editing and resuming agent actions.
To complement this framework, LangChain has also been launched LangGraph cloud, currently in closed beta. This service provides a scalable infrastructure for deploying LangGraph agents, managing horizontally scalable servers and job queues to handle numerous concurrent users and store large states. LangGraph Cloud integrates with LangGraph Studio for visualizing and debugging agent journeys, enabling rapid iteration and feedback for developers.
How Jockey uses the LangGraph and Twelve Labs APIs
Jockey now uses LangGraph in the latest version v1.1 for improved scalability and functionality. Originally built on LangChain, Jockey’s new architecture provides more efficient and precise control over complex video workflows. This transition marks a significant advancement, allowing better management of video processing tasks.
Jockey combines Large Language Models (LLMs) with Twelve Labs’ specialized video APIs through LangGraph’s flexible framework. The intricate network of nodes within the LangGraph UI illustrates Jockey’s decision-making process, including components such as the supervisor, scheduler, video editing, video search, and video text generation nodes. This granular control optimizes token usage and guides node responses, resulting in more efficient video processing.
Jockey’s data flow diagram shows how information moves through the system, from initial query input to complex video processing steps. This includes retrieving videos from Twelve Labs’ APIs, segmenting the content as necessary, and presenting the final results to the user.
Jockey architecture overview
Jockey’s architecture is designed to perform complex video-related tasks through a multi-agent system consisting of the supervisor, scheduler, and workers. The Supervisor acts as a central coordinator, directing tasks between nodes and managing the workflow. The Planner creates detailed plans for complex requests, while the Workers perform tasks using specialized tools such as video search, text generation and editing.
This architecture allows Jockey to dynamically adapt to different queries, from simple text responses to complex video manipulation tasks. LangGraph’s framework helps manage state between nodes, optimize token usage, and provide granular control over every step in the video processing workflow.
Adjust jockey
Jockey’s modular design allows for customization and expansion. Developers can change prompts, extend states for more complex scenarios, or add new workers to address specific use cases. This flexibility makes Jockey a versatile foundation for building advanced video AI applications.
For example, developers can create prompts that instruct Jockey to identify specific scenes from videos without changing the core system. More substantial customizations could include changing prompts, expanding state management, or adding new specialized workers for tasks like advanced video effects or video generation.
Conclusion
Jockey represents a powerful convergence of LangGraph’s agent framework and Twelve Labs’ video insights APIs, opening up new possibilities for intelligent video processing and interaction. Developers can explore the possibilities of Jockey by visiting the Jockey GitHub repository or access the LangGraph documentation for more details.
Image source: Shutterstock