Edit page in Livemark
(2026-02-07 15:31)

Dribdat v3 Roadmap

Contents:

Modernization Proposal

This document outlines a major rework of the Dribdat tech stack to modernize the application, improve developer and user experience, and ensure the codebase is optimized for agentic development.

Generated with Google Jules based on a complete analysis of the source code and documentation, as well as an interview with @loleg

1. Vision & Goals

2. Modernized Tech Stack

Backend: FastAPI

Data Layer: SQLModel

Frontend: Vue.js 3 (Integrated Backboard)

Storage & Media

3. Agentic Development Process

To make the codebase "conducive to the use of agentic development," we recommend the following process and structure:

Modular "Domain" Directory Structure

Instead of a monolithic models.py or views.py, the app should be organized by domain:

dribdat_vnext/
├── domains/
│   ├── event/        # Models, Services, API routes for Events
│   │   ├── models.py
│   │   ├── service.py
│   │   ├── router.py
│   │   └── AGENTS.md # Specific instructions for AI agents working here
│   ├── project/      # Project management & Syncing
│   ├── user/         # Auth & Profiles
│   └── activity/     # Dribs & Logs
├── core/             # Base configurations, DB setup, generic utils
└── main.py           # Application entry point

Strict Coding Standards

  1. 100% Type Coverage: Every function must have type hints for arguments and return values.
  2. Pydantic for Data Validation: All external data must be validated through Pydantic schemas.
  3. Explicit Side Effects: Business logic should reside in "Service" layers, keeping Routers (API endpoints) thin.
  4. AGENTS.md Files: Each module contains an AGENTS.md file explaining its purpose, key patterns, and "gotchas."

4. Full Set of Requirements for Dribdat vNext

R1: Core Event Management

R2: Project & Team Formation

R3: Intelligent Data Sync (The "Sync Engine")

R4: Real-time Activity Log ("Dribs")

R5: Presentation Mode

R6: Data Portability & Migration

5. Migration Strategy

  1. Export: Use the existing Dribdat API to export data into a standard Data Package format.
  2. Schema Mapping: Map legacy SQLAlchemy models to new SQLModel classes.
  3. Import: Create a CLI tool in vNext that reads the Data Package and populates the new PostgreSQL database.
  4. Proxy/Co-existence: (Optional) Run the legacy app in read-only mode for historical events while using vNext for new events.

Sync Engine Redesign

The Project Sync Engine is responsible for aggregating data from various external sources (GitHub, GitLab, etc.) into the Dribdat project dashboard.

1. Current Architecture

Currently, synchronization is synchronous and happens during web requests or via simple CLI commands. It uses multiple libraries (requests, pyquery, etc.) and lacks a standardized plugin system.

2. Proposed vNext Architecture

Asynchronous & Task-Based

Syncing should be non-blocking and managed by a task queue:

Provider-Based Plugin System

Each external data source should be a "Provider" class implementing a standard interface:

class BaseProvider(ABC):
    @abstractmethod
    async def fetch_metadata(self, url: str) -> ProjectMetadata:
        """Fetch basic info: name, summary, image, etc."""
        pass

    @abstractmethod
    async def fetch_content(self, url: str) -> str:
        """Fetch long-form content (README.md)."""
        pass

    @abstractmethod
    async def fetch_activity(self, url: str, since: datetime) -> List[Activity]:
        """Fetch recent commits/updates."""
        pass

Supported Providers (Priority)

  1. GitHub Provider: Uses GitHub API (with optional token support to avoid rate limiting).
  2. GitLab Provider: Supports both gitlab.com and self-hosted instances.
  3. HuggingFace Provider: Syncs model/dataset cards.
  4. Codeberg/Gitea Provider: Generic Forgejo/Gitea support.
  5. Generic Web/OpenGraph Provider: Fallback for any URL (extracting Title, Description, and OG images).

3. Data Flow

  1. Trigger: A user clicks "Sync", or a scheduled job runs.
  2. Dispatch: The Sync Engine identifies the provider based on the autotext_url.
  3. Fetch: The provider fetches data asynchronously.
  4. Transform: Raw data is transformed into Pydantic models (e.g., SyncData).
  5. Reconcile:
    • If a field is empty in Dribdat but present in the remote source, it is filled.
    • If a field is marked as "Always Sync," it is overwritten.
    • Commits are added as Activity objects if they don't already exist.
  6. Notify: The user is notified via WebSocket/SSE that the sync is complete.

4. Agentic Advantage

Migration Strategy

Moving from the legacy Flask stack to the modern FastAPI stack requires a robust data migration path. We utilize the Frictionless Data Package format as our intermediate "bridge."

1. Phase 1: Export from Legacy Dribdat

The legacy application already includes some support for data exports. We recommend enhancing the existing dribdat/apipackage.py to produce a full Data Package:

2. Phase 2: Data Package Validation

Before importing into vNext, the exported package can be validated using the frictionless CLI:

frictionless validate datapackage.json

This ensures that the data conforms to the expected schema and that there are no broken relationships (e.g., a project pointing to a non-existent event).

3. Phase 3: Import into Dribdat vNext

The vNext application will include a migration CLI:

dribdat-cli migrate --source datapackage.json

Key Import Steps

  1. Dependency Sorting: Import Events first, then Categories, then Users, then Projects, then Activities.
  2. Password Handling: Since FastAPI (via Passlib/Bcrypt) might use different salt/rounds than Flask-Bcrypt, we recommend:
    • Carrying over the hashes and attempting compatibility.
    • OR flagging users to "Reset Password on First Login."
  3. ID Mapping: Maintain a mapping of old IDs to new IDs if the primary keys change during the import process.

4. Phase 4: S3 Media Migration

If S3 was used in the legacy app, the image_url fields in the database will likely remain valid if the same S3 bucket is used. If moving buckets:

  1. Use a script to iterate through all image_url and logo_url fields.
  2. Download from the old bucket and upload to the new bucket.
  3. Update the database record in vNext.

5. Why this strategy?