Hub_of_Epstein_Files_Directory

Epstein Files Hub - Bot Usage Guide

This guide explains how the AI agents work and how to interact with them effectively.

Overview

The Epstein Files Hub uses 26 specialized AI agents that work 24/7 to maintain and organize over 30,000 documents and 20,000 images.

Agent Categories

1. Document Management Agents (7 agents)

Document Indexing Agent

OCR Processing Agent

Document Analysis Agent

Document Verification Agent

Document Summarization Agent

Cross-Reference Agent

Document Classification Agent

Total Document Processing Capacity: 42,000+ operations/day

2. Image Management Agents (5 agents)

Image Indexing Agent

Image Analysis Agent

Image Verification Agent

Image Organization Agent

Image Maintenance Agent

Total Image Processing Capacity: 26,000+ operations/day

3. Search & Retrieval Agents (3 agents - Backend Only)

Web Search Agent

Image Search Agent

Internal Search Agent

Note: External search engines are NOT exposed to users. Users only access the Internal Search Agent through the web UI.

4. Quality Control Agents (3 agents)

Fact-Checking Agent

Source Verification Agent

Content Moderation Agent

5. Organization Agents (4 agents)

Collection Management Agent

Timeline Generation Agent

Relationship Mapping Agent

Auto-Tagging Agent

6. Monitoring Agents (2 agents)

System Health Agent

Performance Optimization Agent

7. User Support Agents (2 agents)

Search Assistant Agent

Help & Documentation Agent

How to Use the Bots

For End Users

Using the Search Feature

  1. Go to Search Page
  2. Enter your search criteria
  3. Internal Search Agent processes your query
  4. Results display with relevance scores
  5. Click to view full documents

Search Tips:

Uploading Documents

  1. Visit Upload Page
  2. Select PDF file(s)
  3. PDF Analysis Agent automatically analyzes
  4. Document Routing Agent decides:
    • ≥70% relevance → Accepted & indexed
    • <70% relevance → Rejected & trashed
  5. Receive notification of results

Upload Guidelines:

For Contributors

Submitting via GitHub

# 1. Fork repository
git clone https://github.com/YOUR_USERNAME/Hub_of_Epstein_Files_Directory.git

# 2. Add document
cp document.pdf data/uploads/

# 3. Create metadata
cat > data/uploads/document.json << EOF
{
  "title": "Document Title",
  "date": "2024-01-01",
  "source": "Court Records",
  "relevance": "High"
}
EOF

# 4. Commit and push
git add data/uploads/
git commit -m "Add new court filing"
git push origin main

# 5. Create pull request

Automated Processing:

For Developers

Adding New Bots

# bots/your-new-bot/bot.py

from typing import Dict, Any
from azure.ai import DocumentAnalysisClient

class YourNewBot:
    """
    Description of what this bot does.
    
    Capacity: X operations/day
    Dependencies: Azure service, etc.
    """
    
    def __init__(self, config: Dict[str, Any]):
        self.config = config
        # Initialize services
        
    def process(self, input_data: Any) -> Dict[str, Any]:
        """
        Main processing function.
        
        Args:
            input_data: Input to process
            
        Returns:
            Processing results
        """
        # Implementation
        pass

Steps:

  1. Create bot directory in bots/
  2. Implement bot class
  3. Create README documenting bot
  4. Add to bots/AGENT_INFRASTRUCTURE.md
  5. Create GitHub Actions workflow
  6. Write tests
  7. Submit PR

Monitoring Bot Performance

Check Application Insights:

# Azure CLI
az monitor metrics list \
  --resource /subscriptions/SUB_ID/resourceGroups/RG/providers/Microsoft.Insights/components/APP_INSIGHTS \
  --metric-names requests/count \
  --aggregation Average

Dashboard available at: Monitor Dashboard

Bot Coordination

Agents work together in workflows:

Document Upload Workflow

  1. Upload → PDF Analysis Bot
  2. Analysis → Document Routing Bot
  3. If accepted:
    • OCR → OCR Processing Agent
    • Index → Document Indexing Agent
    • Analyze → Document Analysis Agent
    • Verify → Document Verification Agent
    • Summarize → Summarization Agent
    • Link → Cross-Reference Agent
    • Classify → Classification Agent
  4. Complete → Notification

Image Upload Workflow

  1. Upload → Image Analysis Bot
  2. Analysis → Image Routing Bot
  3. If accepted:
    • Index → Image Indexing Agent
    • Analyze → Image Analysis Agent
    • Verify → Image Verification Agent
    • Organize → Organization Agent
  4. Complete → Notification

Search Query Workflow

  1. Query → Search Assistant Agent
  2. Process → Internal Search Agent
  3. Rank → Relevance Scoring
  4. Results → User Interface

Troubleshooting

Bot Not Processing

Check:

  1. GitHub Actions status
  2. Azure service health
  3. Processing queue length
  4. Error logs in App Insights

Solution:

Slow Processing

Causes:

Solution:

Document Rejected

Reasons:

Solution:

API Documentation

For developers integrating with the system:

Search API

GET /api/search?q=query&type=document&from=0&size=20

Upload API

POST /api/upload
Content-Type: multipart/form-data

file: <binary data>
metadata: <json>

Status API

GET /api/status/{upload_id}

Full API docs: API Documentation

Best Practices

  1. Upload Guidelines:
    • Verify documents before uploading
    • Include source information
    • Use descriptive filenames
    • Check for duplicates first
  2. Search Tips:
    • Start broad, then refine
    • Use filters effectively
    • Check related documents
    • Review source verification levels
  3. Contributing:
    • Follow contribution guidelines
    • Test before submitting
    • Document your changes
    • Be patient with review process

Support

Last Updated: December 2024