MarChat Technical Documentation

Maritime shipboard AI assistant with RAG, multi-mode chat, and pre-arrival form automation

What is MarChat?

MarChat is a maritime AI assistant purpose-built for shipboard use. It provides Retrieval-Augmented Generation (RAG) over maritime regulations including SOLAS, MARPOL, STCW, and other IMO conventions. The system supports multi-mode chat with context-aware system prompts tailored to different operational scenarios, and includes a pre-arrival form auto-fill engine that populates port documentation from natural language context and vessel profile data.

All processing runs locally through Ollama, ensuring that sensitive vessel and operational data never leaves the shipboard network. The hybrid search pipeline combines semantic understanding with keyword matching, delivering precise results even for highly technical regulatory queries.

Architecture

Component Overview

MarChat follows a standard client-server architecture with a Next.js frontend communicating with a FastAPI backend. The backend orchestrates all AI operations including embedding generation, vector search, BM25 retrieval, cross-encoder reranking, and LLM inference. Persistent storage is split between Qdrant (vector embeddings and metadata) and SQLite (relational data such as documents, collections, queries, and form templates).

Data Flow: RAG Query

When a user submits a query, the system executes a multi-stage retrieval pipeline that combines semantic and keyword search, fuses the results, and reranks them before generating a response.

Data Flow: Form Auto-Fill

The form auto-fill pipeline processes uploaded templates, extracts fields, and uses the LLM to generate values from user-provided context.

Installation

Prerequisites

• Docker & Docker Compose — for containerized deployment
• Ollama — local LLM runtime with the following models:
  • gemma3:12b — language model for chat and form filling
  • mxbai-embed-large — embedding model (1024 dimensions)
• Node.js 18+ — for frontend development mode
• Python 3.12+ — for backend development mode

Docker Compose (Recommended)

git clone https://github.com/SL-Mar/marchat.git
cd marchat
docker compose up -d

Pre-Install Ollama Models

Before starting the application, ensure both required models are available in your Ollama instance:

ollama pull gemma3:12b
ollama pull mxbai-embed-large

Development Mode

For local development with hot-reload, use the launch script:

chmod +x launch.sh
./launch.sh

This starts the backend (FastAPI on port 8005) and frontend (Next.js on port 3005) in development mode with automatic reloading on file changes.

Environment Variables

Create a .env file in the project root with the following configuration:

# Core Services
QDRANT_URL=http://localhost:6333
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=gemma3:12b
EMBEDDING_MODEL=mxbai-embed-large

# Database
DATABASE_URL=sqlite:///./data/marchat.db

# Optional: OpenAI fallback (leave empty for local-only)
OPENAI_API_KEY=

Verifying Installation

After starting the application, verify that all services are healthy:

# Check backend health endpoint
curl http://localhost:8005/health

# Expected response:
{
  "status": "healthy",
  "services": {
    "qdrant": "connected",
    "ollama": "connected",
    "database": "connected"
  }
}

Tech Stack

Backend

Frontend

Storage

AI Models

Hybrid Search

The hybrid search pipeline is the core retrieval mechanism in MarChat. It combines two fundamentally different search strategies — semantic vector search and BM25 keyword matching — to achieve robust retrieval across the full spectrum of maritime regulation queries. This is critical because maritime queries often mix natural language questions ("What are the fire safety requirements?") with precise technical references ("Regulation 3-1, Chapter II-2").

Step 1: Semantic Search

The user's query is encoded into a 1024-dimensional vector using the mxbai-embed-large model via Ollama. This vector is then used to perform a cosine similarity search against the Qdrant vector database. Semantic search excels at understanding the meaning behind a query, finding relevant content even when the exact words differ. For example, "fire protection in engine rooms" will match chunks about "fire detection systems in machinery spaces" because the semantic meaning is similar.

To ensure a broad candidate pool, the semantic search retrieves top_k x 3 results, which are later narrowed down by fusion and reranking.

Step 2: BM25 Keyword Search

In parallel, the query is tokenized using simple lowercase word splitting and matched against the BM25 inverted index. BM25 is a probabilistic ranking function that scores documents based on term frequency, inverse document frequency, and document length normalization. This approach is critical for capturing exact regulation numbers, specific technical terms, and abbreviations that semantic search might miss. For example, "Reg. 14.1.2.3" or "SOLAS II-2" are best matched by keyword search.

Like semantic search, BM25 retrieves top_k x 3 results to provide ample candidates for fusion.

Step 3: Reciprocal Rank Fusion (RRF)

The results from both search methods are combined using Reciprocal Rank Fusion. RRF is a rank-based fusion method that does not require score normalization across different retrieval systems. Each document receives a score based on its rank position in each result list, with the constant k = 60 dampening the influence of high-ranking positions. Documents appearing in both lists receive combined scores, naturally boosting results that are relevant by both criteria.

After fusion, the top top_k x 2 candidates are passed forward for reranking.

Step 4: Cross-Encoder Reranking

The fused candidates are reranked using a cross-encoder model (cross-encoder/ms-marco-MiniLM-L-6-v2). Unlike bi-encoder similarity (used in semantic search), the cross-encoder processes the query and each candidate document jointly, allowing it to capture fine-grained interactions between the query and document text. This produces much more accurate relevance scores at the cost of higher computational overhead — which is why it is applied only to the already-filtered candidate set.

Documents scoring below MIN_RERANK_SCORE = -2.0 are filtered out, and the remaining results are returned sorted by relevance score, limited to top_k.

BM25 Engine

The BM25 engine provides keyword-based retrieval as a complement to semantic vector search. It maintains a persistent inverted index that is incrementally updated as new documents are indexed.

Tokenization

The engine uses a simple tokenization strategy: text is converted to lowercase and split on whitespace and punctuation boundaries. This straightforward approach works well for maritime regulatory text, which contains many technical terms and abbreviations that benefit from exact matching.

Persistence

The BM25 index is serialized using Python's pickle module and stored at data/bm25_index.pkl. This allows the index to persist across application restarts without requiring re-indexing of all documents.

Incremental Indexing

When new documents are indexed, the BM25 index is updated incrementally rather than rebuilt from scratch. New chunk texts and their corresponding IDs are appended to the existing index, and the BM25 statistics (term frequencies, document lengths) are recalculated.

Key Methods

# BM25 index location
data/bm25_index.pkl

# Index is loaded at startup if the file exists
# Re-created automatically if missing or corrupted

Smart Chunking

Maritime regulations follow a strict hierarchical structure defined by the International Maritime Organization (IMO). MarChat's smart chunking engine detects this hierarchy during document processing and preserves it as metadata on each chunk, enabling precise citations in RAG responses.

IMO Hierarchy Detection

The chunker uses regular expression patterns to detect hierarchy levels in the document text. As it processes each line, it maintains a cascading state machine that tracks the current position in the hierarchy.

# Hierarchy detection regex patterns

convention:  (SOLAS|MARPOL|STCW|COLREG|LOADLINE|TONNAGE|SFV|STP|SAR)
annex:       ANNEX\s+[IVX]+|ANNEX\s+[0-9]+
chapter:     CHAPTER\s+\w+
regulation:  Regulation\s+\d+|Reg\.\s*\d+
paragraph:   \d+\.\d+(\.\d+)*

Processing Steps

1. Split PDF by pages — Extract text from each page of the uploaded PDF document.
2. Process line-by-line — Scan each line for hierarchy markers using the regex patterns above.
3. Detect convention from filename — If the filename contains a convention name (e.g., SOLAS_consolidated.pdf), set the convention metadata automatically.
4. Maintain hierarchical state — Track the current position as: convention → annex → chapter → regulation → paragraph. Each new detection updates the corresponding level and all chunks inherit the full hierarchy path.
5. Build chunks at threshold — Accumulate text until the chunk reaches 2400 characters, then split with a 100-word overlap to preserve context across chunk boundaries.
6. Cascading reset — When a higher-level marker is detected (e.g., a new annex), all lower levels (chapter, regulation, paragraph) are reset. This prevents stale metadata from carrying over into a new section.

Metadata Output

Each chunk is stored with the following metadata fields:

Cross-Encoder Reranking

After the Reciprocal Rank Fusion step produces a merged candidate set, the cross-encoder model provides a final precision pass by jointly scoring each (query, document) pair.

Model Details

How It Works

Unlike bi-encoder models (which encode query and document independently and compare their vectors), a cross-encoder processes the query and document text together through the same transformer pass. This allows the model to attend across both inputs simultaneously, capturing word-level interactions between the query and the document. The result is significantly more accurate relevance judgments, at the cost of being computationally more expensive (which is why it is only applied to the already-narrowed candidate set, not the entire corpus).

Filtering

After scoring, any candidate with a relevance score below -2.0 is removed from the results. This threshold was determined empirically to filter out genuinely irrelevant content while retaining borderline-relevant chunks that may contain useful context. The remaining candidates are sorted by score in descending order and truncated to top_k.

Multi-Mode Chat

MarChat supports five specialized chat modes, each with a tailored system prompt that controls the LLM's behavior, citation format, and response structure. The mode is selected per-query, allowing users to switch between different operational contexts within the same session.

Example: Regulatory Mode Query

# Request
POST /api/rag/query
{
  "query": "What are the requirements for fire detection in machinery spaces?",
  "mode": "regulatory",
  "collection_name": "maritime_v1",
  "top_k": 5
}

# Response includes citations like:
# "According to SOLAS > Chapter II-2 > Regulation 7 > Para. 2.1 (Page 203),
#  fixed fire detection and fire alarm systems shall be provided
#  in machinery spaces of category A..."

System Prompts

Each chat mode uses a carefully crafted system prompt that defines the LLM's persona, response structure, citation requirements, and content guidelines. The system prompt is prepended to the user's query along with the retrieved context chunks.

Prompt Structure

Every system prompt follows a common structure:

1. Role definition — who the assistant is and its area of expertise
2. Citation requirements — how to reference source material
3. Response format — expected structure, tone, and length
4. Content guidelines — what to include, what to avoid
5. Fallback behavior — how to handle queries outside the knowledge base

Regulatory Mode Prompt (Example)

You are a maritime regulatory expert specializing in IMO conventions.
Answer questions using ONLY the provided context from maritime regulations.

CITATION FORMAT (MANDATORY):
- Always cite the specific regulation using the full hierarchy:
  Convention > Chapter > Regulation > Paragraph, Page
- Example: "SOLAS > Chapter II-1 > Regulation 3-1 > Para. 2.1, Page 145"

RESPONSE STRUCTURE:
1. Direct answer to the question
2. Relevant regulatory text with citations
3. Additional context or related regulations if applicable

IMPORTANT:
- If the context does not contain sufficient information, state this clearly
- Do not fabricate or assume regulatory content
- Use exact text from regulations when possible

Forms Mode Prompt (Example)

You are a pre-arrival documentation specialist.
Given the form fields and context provided, generate a JSON object
mapping each field name to its appropriate value.

OUTPUT FORMAT:
Return ONLY a valid JSON object with field names as keys:
{
  "vessel_name": "MV Pacific Star",
  "imo_number": "9123456",
  "eta": "2026-03-15T08:00:00",
  "last_port": "Singapore"
}

IMPORTANT:
- Use the vessel profile data when available
- Infer values from the natural language context
- Use ISO 8601 format for dates
- Leave unknown fields as empty strings

Mode-Specific Behaviors

• Regulatory mode enforces hierarchical citations and uses exact regulatory text whenever available in the context.
• Medical mode includes safety disclaimers and references to TMAS (Telemedical Assistance Service) protocols for remote medical guidance.
• General mode provides practical, actionable advice while still referencing source material.
• Forms mode outputs strictly valid JSON, mapping field names to inferred values from the provided context.
• Research mode adopts an academic tone, tracks sources carefully, and provides page-level citations.

LLM Providers

MarChat uses a factory pattern to abstract LLM provider details, allowing the system to switch between different backends without changing application logic.

Provider Factory

# Factory function
def get_llm_provider(provider_type: str, **kwargs) -> LLMProvider:
    """
    Returns an LLM provider instance based on the specified type.

    Args:
        provider_type: "ollama" or "openai"
        **kwargs: Provider-specific configuration

    Returns:
        LLMProvider instance with generate() and embed() methods
    """
    if provider_type == "ollama":
        return OllamaProvider(**kwargs)
    elif provider_type == "openai":
        return OpenAIProvider(**kwargs)
    else:
        raise ValueError(f"Unknown provider: {provider_type}")

Ollama Provider (Default)

The Ollama provider communicates with a locally running Ollama instance via its HTTP API. It is the default and recommended provider for MarChat, as it keeps all data on the local machine.

# Ollama API call for chat generation
POST http://localhost:11434/api/chat
{
  "model": "gemma3:12b",
  "messages": [
    {"role": "system", "content": "...system prompt..."},
    {"role": "user", "content": "...context + query..."}
  ],
  "options": {
    "temperature": 0.3,
    "num_predict": 2048,
    "repeat_penalty": 1.1,
    "top_k": 40,
    "top_p": 0.9
  },
  "stream": false
}

OpenAI Provider (Optional)

The OpenAI provider is available as a fallback for environments where Ollama is not available. It requires an OPENAI_API_KEY environment variable and uses the openai.ChatCompletion API. Note that using OpenAI sends data to external servers, which may not be appropriate for sensitive vessel information.

Template Processing

The template processing engine handles Excel (.xlsx) and DOCX files used for pre-arrival port documentation. It detects form fields using placeholder patterns and highlighted cells, then provides a structured representation of the template's fields for filling.

Field Detection

The engine searches for two types of field markers in uploaded templates:

• Placeholder patterns — {{field_name}} markers in cell values or document text
• Highlighted cells — Yellow-highlighted cells in Excel spreadsheets (common in port authority templates)

Field Type Inference

Field types are automatically inferred from the field name using keyword matching:

FormField Data Model

@dataclass
class FormField:
    name: str               # Field identifier (e.g., "vessel_name")
    field_type: str         # text, date, number, boolean, list
    location: str           # Cell reference (Excel) or placeholder position (DOCX)
    required: bool          # Whether the field is mandatory
    default_value: Optional[str]  # Pre-populated value if available
    description: str        # Human-readable field description

Fill Process

1. Load template — Open the uploaded Excel or DOCX file using openpyxl or python-docx.
2. Find placeholders — Scan all cells (Excel) or paragraphs (DOCX) for {{...}} patterns.
3. Replace with values — Substitute each placeholder with the corresponding value from the fill data.
4. Save filled document — Write the filled template to disk and make it available for download.

# Example: Excel placeholder detection
for sheet in workbook.sheetnames:
    ws = workbook[sheet]
    for row in ws.iter_rows():
        for cell in row:
            if cell.value and "{{" in str(cell.value):
                # Extract field name from {{field_name}}
                match = re.findall(r'\{\{(\w+)\}\}', str(cell.value))
                for field_name in match:
                    fields.append(FormField(
                        name=field_name,
                        field_type=infer_type(field_name),
                        location=f"{sheet}!{cell.coordinate}",
                        required=True,
                        default_value=None,
                        description=field_name.replace("_", " ").title()
                    ))

Auto-Fill Engine

The auto-fill engine uses the LLM to intelligently populate form fields from unstructured natural language context. Instead of manually filling each field, the user provides a free-text description of the voyage, vessel, and port information, and the LLM extracts the relevant values.

How It Works

1. Extract field list — The system retrieves all detected fields from the uploaded template.
2. Build prompt — A structured prompt is constructed containing the field list, the user's natural language context, and optionally the vessel profile data.
3. LLM generation — The forms mode system prompt instructs the LLM to output a JSON object mapping each field name to its inferred value.
4. Parse response — The engine handles markdown code block wrapping (strips ```json delimiters if present) and parses the JSON.
5. Fill template — The parsed values are applied to the template placeholders.

Example Context Input

We are MV Pacific Star, IMO 9123456, Panama flag, arriving at
Rotterdam on March 15th 2026 at approximately 0800 local time.
Coming from Singapore via Suez Canal. Gross tonnage 45,230.
We have 22 crew members on board, all healthy. No dangerous
goods. Last port state control inspection was in Singapore on
February 28th, no deficiencies found. Agent is MaritimePort BV.

Example LLM Output

{
  "vessel_name": "MV Pacific Star",
  "imo_number": "9123456",
  "flag_state": "Panama",
  "port_of_arrival": "Rotterdam",
  "eta": "2026-03-15T08:00:00",
  "last_port": "Singapore",
  "gross_tonnage": "45230",
  "crew_count": "22",
  "has_dangerous_goods": "No",
  "last_psc_inspection_port": "Singapore",
  "last_psc_inspection_date": "2026-02-28",
  "psc_deficiencies": "None",
  "agent_name": "MaritimePort BV"
}

Vessel Profile Pre-Population

If a vessel profile is associated with the fill request, its static fields (vessel name, IMO number, flag state, call sign, MMSI, tonnage, etc.) are automatically included in the prompt context. This reduces the amount of information the user needs to provide and ensures consistency across multiple form fills.

Vessel Profiles

Vessel profiles store reusable vessel information that can be pre-populated into forms, eliminating the need to re-enter static data for every port call.

Profile Fields

The extra_fields_json column allows storing any additional vessel-specific data without schema changes. This is useful for fields that vary by vessel type, flag state requirements, or specific port authority needs.

RAG API

Endpoints for executing RAG queries against indexed document collections.

Request Body

{
  "query": "What are the requirements for fire detection in machinery spaces?",
  "mode": "regulatory",
  "collection_name": "maritime_v1",
  "top_k": 5
}

Response

{
  "answer": "According to SOLAS > Chapter II-2 > Regulation 7 > Para. 2.1 (Page 203), fixed fire detection and fire alarm systems shall be provided in machinery spaces of category A. The system must be capable of rapidly detecting the onset of fire...",
  "retrieved_chunks": [
    {
      "text": "2.1 A fixed fire detection and fire alarm system...",
      "metadata": {
        "convention": "SOLAS",
        "chapter": "CHAPTER II-2",
        "regulation": "Regulation 7",
        "paragraph": "2.1",
        "page": 203
      },
      "score": 0.847
    }
  ],
  "total_chunks_retrieved": 5,
  "response_time_ms": 2340,
  "mode": "regulatory"
}

Query Parameters

Response

[
  {
    "id": 42,
    "query_text": "Fire detection in machinery spaces",
    "answer_text": "According to SOLAS...",
    "mode": "regulatory",
    "response_time_ms": 2340,
    "created_at": "2026-02-07T14:30:00Z"
  }
]

Documents API

Endpoints for uploading, indexing, and managing PDF documents within collections.

Request (multipart/form-data)

Response

{
  "id": 7,
  "title": "SOLAS_Consolidated_2024",
  "filename": "SOLAS_Consolidated_2024.pdf",
  "status": "indexing",
  "collection_id": 1,
  "total_chunks": 0,
  "created_at": "2026-02-07T10:15:00Z"
}

Collections API

Endpoints for managing document collections. Each collection maps to a Qdrant collection and has its own schema, search configuration, and document set.

Response

[
  {
    "id": 1,
    "name": "Maritime Regulations",
    "description": "SOLAS, MARPOL, STCW and other IMO conventions",
    "schema_id": "maritime_v1",
    "qdrant_collection_name": "maritime_v1",
    "document_count": 12,
    "created_at": "2026-01-15T09:00:00Z"
  }
]

Request Body

{
  "name": "Medical Guidelines",
  "description": "IMGS and maritime medical references",
  "schema_id": "medical_v1"
}

Available Schemas

Chat API

Endpoints for the multi-mode chat system. Chat sessions maintain conversation history for context continuity.

Request Body

{
  "session_id": "sess_abc123",
  "message": "What are the minimum rest hour requirements for watchkeepers?",
  "mode": "regulatory",
  "use_rag": true,
  "collection_name": "maritime_v1"
}

Response

{
  "session_id": "sess_abc123",
  "response": "According to STCW > Section A-VIII/1 > Para. 2 (Page 312), the minimum rest period for seafarers assigned to watchkeeping duties is:\n\n1. A minimum of 10 hours of rest in any 24-hour period\n2. A minimum of 77 hours in any 7-day period\n\nThe rest period may be divided into no more than two periods, one of which shall be at least 6 hours in length...",
  "mode": "regulatory",
  "sources": [
    {
      "convention": "STCW",
      "chapter": "Section A-VIII/1",
      "paragraph": "2",
      "page": 312
    }
  ]
}

Forms API

Endpoints for managing form templates, filling forms (manually or with LLM assistance), and managing vessel profiles.

Template Management

Request (multipart/form-data)

Response

{
  "id": 3,
  "name": "Rotterdam Pre-Arrival Form",
  "file_type": "xlsx",
  "field_count": 28,
  "fields": [
    {"name": "vessel_name", "field_type": "text", "location": "Sheet1!B3"},
    {"name": "imo_number", "field_type": "text", "location": "Sheet1!B4"},
    {"name": "eta", "field_type": "date", "location": "Sheet1!B7"},
    {"name": "gross_tonnage", "field_type": "number", "location": "Sheet1!B10"},
    {"name": "has_dangerous_goods", "field_type": "boolean", "location": "Sheet1!B15"}
  ],
  "created_at": "2026-02-07T11:00:00Z"
}

Form Filling

Request Body

{
  "template_id": 3,
  "field_values": {
    "vessel_name": "MV Pacific Star",
    "imo_number": "9123456",
    "eta": "2026-03-15T08:00:00",
    "gross_tonnage": "45230",
    "has_dangerous_goods": "No"
  },
  "vessel_profile_id": 1
}

Request Body

{
  "template_id": 3,
  "context": "We are MV Pacific Star, IMO 9123456, Panama flag, arriving at Rotterdam on March 15th 2026 at approximately 0800 local time. Coming from Singapore via Suez Canal. Gross tonnage 45,230. 22 crew members, all healthy. No dangerous goods.",
  "vessel_profile_id": 1
}

Response

{
  "id": 15,
  "template_id": 3,
  "vessel_profile_id": 1,
  "field_values": {
    "vessel_name": "MV Pacific Star",
    "imo_number": "9123456",
    "flag_state": "Panama",
    "port_of_arrival": "Rotterdam",
    "eta": "2026-03-15T08:00:00",
    "last_port": "Singapore",
    "gross_tonnage": "45230",
    "crew_count": "22",
    "has_dangerous_goods": "No"
  },
  "fields_filled": 9,
  "fields_total": 28,
  "created_at": "2026-02-07T11:30:00Z"
}

Vessel Profile Management

Request Body

{
  "vessel_name": "MV Pacific Star",
  "imo_number": "9123456",
  "flag_state": "Panama",
  "call_sign": "3FXY7",
  "mmsi": "354123456",
  "gross_tonnage": 45230,
  "net_tonnage": 22615,
  "deadweight": 52000,
  "vessel_type": "Bulk Carrier",
  "year_built": 2018,
  "owner": "Pacific Shipping Ltd",
  "operator": "StarBulk Management",
  "classification_society": "Lloyd's Register"
}

Database Schema

MarChat uses SQLite for relational data storage with 11 tables organized into RAG-related and form-related groups. The database file is located at data/marchat.db.

RAG Tables

DocumentSchema

Defines the metadata structure and chunking strategy for a collection type.

Collection

Represents a document collection backed by a Qdrant vector collection.

CollectionSearchConfig

Per-collection search configuration for the hybrid pipeline.

Document

Represents an uploaded PDF document.

Query

Stores RAG query history for analytics and debugging.

Form Tables

VesselProfile

Stores reusable vessel information for form pre-population.

FormTemplate

Stores uploaded form templates with their detected fields.

FilledForm

Stores completed form instances with their field values.

Settings_DB

Key-value store for application settings.

Collection Schemas

Collection schemas define how documents are chunked, what metadata fields are extracted, and how the hierarchy detection works for each collection type. MarChat ships with four built-in schemas optimized for different document types.

System Schemas

Schema Configuration Example

# maritime_v1 schema definition
{
  "schema_id": "maritime_v1",
  "name": "Maritime Regulations",
  "metadata_fields": [
    {"name": "convention", "type": "string", "required": true},
    {"name": "annex", "type": "string", "required": false},
    {"name": "chapter", "type": "string", "required": false},
    {"name": "regulation", "type": "string", "required": false},
    {"name": "paragraph", "type": "string", "required": false},
    {"name": "page", "type": "integer", "required": true}
  ],
  "hierarchy_config": {
    "levels": ["convention", "annex", "chapter", "regulation", "paragraph"],
    "patterns": {
      "convention": "(SOLAS|MARPOL|STCW|COLREG|LOADLINE|TONNAGE|SFV|STP|SAR)",
      "annex": "ANNEX\\s+[IVX]+|ANNEX\\s+[0-9]+",
      "chapter": "CHAPTER\\s+\\w+",
      "regulation": "Regulation\\s+\\d+|Reg\\.\\s*\\d+",
      "paragraph": "\\d+\\.\\d+(\\.\\d+)*"
    },
    "cascade_reset": true
  },
  "chunking_strategy": {
    "type": "hierarchical",
    "chunk_size": 800,
    "overlap": 100,
    "respect_boundaries": true
  }
}

Schema Validation Rules

• schema_id pattern — Must match [a-z_]+_v[0-9]+ (lowercase with version suffix)
• name length — Minimum 3 characters
• regex patterns — Must be valid regular expressions (tested at schema creation time)
• chunk_size — Must be at least 100 characters
• overlap — Must be less than chunk_size
• metadata fields — At least one field must be defined per schema

Validation Steps

This section provides a comprehensive validation checklist for verifying that all MarChat components are functioning correctly. Follow these steps after initial installation or after making configuration changes.

1. Service Health

• GET /health returns all services green (qdrant: connected, ollama: connected, database: connected)
• GET /api/settings/health shows service connectivity status with version information
• Ollama responds with the correct model (gemma3:12b) when queried via /api/tags
• Qdrant has the correct vector dimension (1024) for all created collections

# Verify service health
curl http://localhost:8005/health

# Verify Ollama models
curl http://localhost:11434/api/tags | python -m json.tool

# Verify Qdrant collections
curl http://localhost:6333/collections | python -m json.tool

2. Document Processing

• Upload a PDF document → status changes from uploaded to indexing to indexed
• total_chunks is greater than 0 after indexing completes
• BM25 index is updated (data/bm25_index.pkl file modified timestamp changes)
• Qdrant collection has a matching point count (number of vectors equals total chunks)

# Upload and verify
curl -X POST http://localhost:8005/api/documents/upload \
  -F "file=@SOLAS_Chapter_II-2.pdf" \
  -F "collection_name=maritime_v1"

# Check document status
curl http://localhost:8005/api/documents/7

3. RAG Query Pipeline

• Semantic search returns relevant chunks based on meaning
• BM25 search returns keyword-matching chunks for exact terms (regulation numbers, abbreviations)
• RRF fusion combines both result sets, boosting documents that appear in both
• Cross-encoder reranking improves precision by filtering low-relevance candidates
• Generated answer includes proper citations with hierarchy metadata
• Response time is logged in the query history table

# Test RAG query
curl -X POST http://localhost:8005/api/rag/query \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What are the fire detection requirements for machinery spaces?",
    "mode": "regulatory",
    "collection_name": "maritime_v1",
    "top_k": 5
  }'

4. Multi-Mode Chat

• Each mode produces appropriately formatted responses matching its system prompt
• Regulatory mode includes full convention hierarchy citations (Convention > Chapter > Regulation > Para, Page)
• Medical mode references IMGS protocols and includes appropriate safety disclaimers
• General mode provides practical operational advice with source references
• Forms mode outputs valid JSON that can be parsed by the auto-fill engine
• Research mode uses academic tone with page-level citations and source attribution

5. Pre-Arrival Forms

• Template upload extracts the correct field count from the uploaded Excel or DOCX file
• {{placeholder}} patterns are correctly detected in both Excel cells and DOCX paragraphs
• Manual fill replaces all placeholders with the provided values
• Auto-fill produces valid JSON from natural language context
• Downloaded file has all placeholder values filled correctly
• Vessel profile data is correctly pre-populated into form fields

# Test template upload
curl -X POST http://localhost:8005/api/forms/templates/upload \
  -F "file=@Rotterdam_PreArrival.xlsx" \
  -F "name=Rotterdam Pre-Arrival"

# Test auto-fill
curl -X POST http://localhost:8005/api/forms/fill/auto \
  -H "Content-Type: application/json" \
  -d '{
    "template_id": 3,
    "context": "MV Pacific Star, IMO 9123456, arriving Rotterdam March 15th 2026",
    "vessel_profile_id": 1
  }'

6. Collection Management

• Create collection with schema → corresponding Qdrant collection is created with correct vector config (1024 dim, cosine distance)
• Upload document to specific collection → chunks are indexed in the correct Qdrant collection
• Query with collection_name parameter → search is scoped to only that collection
• Delete collection → removes both Qdrant vectors and all related database records (documents, chunks, search config)