Ingesting Documents for AI: The Missing Piece
Building AI-powered applications in Laravel often involves complex pipelines: chunking text, creating embeddings, and performing vector searches. Laravel 13's AI SDK streamlines much of this, but a critical gap remained: getting clean, usable text out of diverse document formats, especially scanned PDFs. This tutorial addresses that gap, providing an end-to-end solution from document upload to AI-driven question answering.
The goal is to enable users to upload documents, have them converted to Markdown, split into manageable chunks, embedded into a PostgreSQL database using the pgvector extension, and then queried using an AI agent. This approach keeps the entire data pipeline within a single PostgreSQL instance, avoiding the need for a separate vector database.
Setup and Dependencies
To implement this, you'll need Laravel 13 and a PostgreSQL database with the pgvector extension enabled. Two key packages are required: the official laravel/ai package and a third-party package, parseforartisans/laravel, which handles the heavy lifting of document parsing.
composer require laravel/ai parseforartisans/laravel
After installation, you'll need to configure the necessary service providers and potentially publish configuration files. The parseforartisans/laravel package typically requires minimal configuration, often relying on sensible defaults for common document types. Ensure your .env file is correctly set up with your database credentials, and that the pgvector extension is active in your PostgreSQL instance.
Enabling the pgvector extension is crucial. This can usually be done by connecting to your PostgreSQL database as an administrator and running the SQL command: CREATE EXTENSION vector;. If you are using a managed PostgreSQL service, consult their documentation for enabling extensions.
The Document Ingestion Pipeline
The process begins when a user uploads a document. This could be a PDF, a scanned image containing text, or other formats. The parseforartisans/laravel package acts as the initial gatekeeper, responsible for extracting raw text from these diverse sources.
For traditional PDFs, the package leverages robust libraries to extract text layers. For scanned documents or image-based PDFs, it employs optical character recognition (OCR) to convert visual information into machine-readable text. The output of this stage is clean, structured text, ready for the next steps.
Once the text is extracted, it is passed to the Laravel AI pipeline. The first step here is chunking. Large documents are broken down into smaller, semantically coherent pieces. This is vital for effective embedding and subsequent querying, as AI models perform best with focused context. The optimal chunk size can vary, but common practice involves chunks of a few hundred words, often with overlapping sections to preserve context across boundaries.
Following chunking, each piece of text is converted into a numerical vector representation – an embedding. The laravel/ai package integrates with various embedding models to achieve this. These embeddings capture the semantic meaning of the text chunks. These vectors are then stored in a PostgreSQL table equipped with the pgvector extension. This extension allows PostgreSQL to efficiently store and query high-dimensional vectors, serving as a capable vector database replacement.
Querying and Answering with Agents
With the documents processed and embedded, the system is ready to answer user questions. A user can ask a question through the Laravel application's interface. This question is also converted into an embedding using the same model used for the document chunks.
The laravel/ai package's agent functionality then takes over. The agent performs a similarity search within the pgvector column of your PostgreSQL database. It identifies the document chunks whose embeddings are most similar to the question's embedding. These relevant chunks are retrieved.
Finally, the retrieved chunks, along with the original user question, are fed into a large language model (LLM) orchestrated by the AI agent. The LLM synthesizes an answer based on the provided context. This creates a conversational experience where users can ask questions about their uploaded documents and receive accurate, contextually relevant answers.
End-to-End Workflow Example
Consider a scenario where a user uploads a 40-page scanned PDF manual for a piece of machinery. The parseforartisans/laravel package uses OCR to extract all text. This raw text is then chunked into approximately 500-word segments. Each segment is embedded and stored in PostgreSQL.
Later, a user asks, "What is the recommended maintenance schedule for the XYZ component?" The question is embedded. The system finds the chunks from the manual that are semantically closest to this question, likely sections detailing maintenance procedures and schedules. These chunks are passed to the LLM, which then generates a precise answer, potentially quoting directly from the manual or summarizing the relevant information.
This entire process, from raw document upload to a synthesized answer, is managed within the Laravel framework, leveraging the AI SDK and the parsing package. The surprising detail here is the seamless integration of OCR and advanced AI querying capabilities, making complex document analysis accessible within a familiar PHP framework.
Broader Implications
This combination of tools democratizes the creation of RAG (Retrieval-Augmented Generation) systems. Developers no longer need to cobble together disparate services for OCR, text extraction, vector storage, and LLM orchestration. By integrating these capabilities into Laravel, the barrier to entry for building sophisticated AI applications that understand user-provided documents is significantly lowered. This enables startups and established businesses alike to quickly develop internal knowledge bases, customer support bots, and document analysis tools.
The reliance on PostgreSQL with pgvector also offers a cost and complexity advantage. For many use cases, it eliminates the need for separate, often expensive, vector database subscriptions. This makes advanced AI features more accessible and manageable, particularly for projects with budget constraints or those prioritizing a consolidated data infrastructure.