# Ruturaj Chaubey

> Full content of https://www.ruturaj.xyz, for LLMs.

Full Stack Engineer with 3+ years of experience building high-performance, decoupled web platforms. At QED42, he architects headless Drupal and Next.js solutions for enterprise clients including UNICEF and US-based organisations. Beyond client work, he's an active Drupal community speaker and open source contributor based in Pune, India.

- GitHub: https://atlas.ruturaj.xyz/s/ruturaj-github
- LinkedIn: https://atlas.ruturaj.xyz/s/ruturaj-linkedin
- X (Twitter): https://atlas.ruturaj.xyz/s/ruturaj-x
- Drupal: https://atlas.ruturaj.xyz/s/ruturaj-drupal

## Experience

### Engineer - Full Stack, QED42 (May 2023 – Present)

- Architected and led the front-end development for a UNICEF-affiliated product using Next.js, focusing on performance, scalability, and clean UX.
- Explored and implemented AI technologies, building proof-of-concept applications to strengthen understanding of applied AI and emerging trends.
- Led a web development internship program, training and mentoring fresh graduates into industry-ready full stack engineers.
- Conducted internal technical sessions, demonstrating Headless Drupal starter kits like Octahedriod to enhance the team's technical proficiency.
- Contributed to Drupal Open Source, delivering code solutions and improvements for community-reported issues.

Tech: Next.js, Tailwind CSS, Sass, Drupal, PHP, Fast API, Python, Supabase, Postman, Figma, Cloudflare, Vercel

### Associate Engineer - Full Stack, QED42 (May 2022 – Apr 2023)

- Developed and deployed a full-featured e-commerce platform for a U.S.-based client using Drupal Commerce and Gatsby.
- Optimized performance across the stack, cutting page load times by 25% through improved caching, build optimization, and API tuning.
- Engineered robust integrations between Gatsby and Drupal Commerce, enabling seamless data synchronization across product listings.
- Led internal training sessions on React and Headless Drupal architecture, upskilling the team on modern web development practices.

Tech: React, Redux, Gatsby, Tailwind CSS, Drupal, PHP, MySQL, GraphQL, Figma

### Intern, QED42 (Aug 2021 – Apr 2022)

- Integrated React with Drupal CMS to enhance content workflows and elevate overall site interactivity.
- Contributed patches to the Drupal Open Source community and participated in strategic development initiatives.
- Developed hands-on expertise in Drupal CMS (module and theme development), Headless Drupal, and working with REST and JSON APIs.

Tech: Drupal, PHP, React, Figma

## Projects

### Neuron

A shared backend platform that exposes reusable services — a URL shortener and queued email notifications with templates — behind a single API key. Includes a dashboard with Google sign-in for managing keys, using services directly, and tracking per-key usage analytics.

Tech: NestJS, Next.js, TypeScript, PostgreSQL, Prisma, Supabase, Redis, Resend, Tailwind CSS, Docker

Live: https://neuron.ruturaj.xyz/
Source: https://github.com/RChaubey16/neuron

### Echo

A private library for the quotes that stay with you, and what they meant to you at the time. Save a quote in seconds with quick capture, add a reflection, and organize with tags, collections and favorites. Full-text search finds anything, and Today's Echo, 'Echo me something' and scheduled Revisits bring quotes back over time. Google sign-in only, every request checked for ownership, and JSON/CSV export so your data stays yours.

Tech: Next.js, React, TypeScript, PostgreSQL, Prisma, Supabase, Upstash Redis, Tailwind CSS, Vercel

Live: https://echo.ruturaj.xyz/
Source: https://github.com/RChaubey16/echo

### Roast My Drupal

Paste a drupal.org username or profile URL and get roasted by an LLM using only your public Drupal activity: bio, badges, maintained projects (with release and issue health), and contribution stats. Pick a comedic persona like Chandler Bing or Gordon Ramsay. A single Next.js API route scrapes, parses, caches, and streams the roast from Gemini, with a live stats card alongside.

Tech: Next.js, TypeScript, React, Vercel AI SDK, Gemini, Cheerio, Vercel KV, Vitest

Live: https://roast-my-drupal.vercel.app/
Source: https://github.com/RChaubey16/roast-my-drupal

### What the Hex

A fast, minimal color lookup tool for developers and designers. Paste a hex code, get the color name, preview, and ready-to-use CSS/SCSS variables — all in one place.

Tech: React, TypeScript, Tailwind CSS

Live: https://what-the-hex.ruturaj.xyz
Source: https://github.com/RChaubey16/what-the-hex

## Talks

### Inside Drupal CMS 2.0: Visual Building, AI, and Site Templates

Drupal Pune Meetup, Feb 2026

Drupal CMS 2.0 introduces visual building, built-in AI, and ready-to-use site templates. A live walkthrough showing what these features mean for developers, content teams, and digital organizations.

Event: https://www.drupal.org/community/events/drupal-pune-meetup-february-21-2026-2026-06-21

### Everything AI in Drupal: Past, Present, and What's Next

DrupalCamp Pune 2025, Sep 2025

A look at how AI has evolved within the Drupal ecosystem — from early integrations to the Drupal AI module. Covers where the project stands today and what the community can expect next.

Event: https://www.drupal.org/community/events/drupalcamp-pune-2025-2025-09-13

### From Static to Dynamic: Next.js Rendering Patterns with Drupal CMS

DrupalCamp Pune 2025, Sep 2025

The Next.js App Router offers SSR, SSG, and ISR — but which fits which content type? Explores how to pair each rendering strategy with Drupal's content model when building a headless site.

Event: https://www.drupal.org/community/events/drupalcamp-pune-2025-2025-09-13

### All about Drupal CMS

Pune Drupal CMS Launch Party, Jan 2025

A first look at Drupal CMS 1.0, built on Drupal core. Covers what it is, who it's for, and how it simplifies site building for editors and developers.

Event: https://www.drupal.org/community/events/pune-drupal-cms-launch-party-2025-01-15

### Transforming Experiences: A Guide to Transform API and Decoupled Layout Builder in Drupal

Drupal Pune Meetup, Dec 2023

The Transform API bridges Drupal's content and frontend frameworks in headless architectures. Explores how it delivers structured JSON and handles responsive images for better performance on decoupled sites.

Event: https://www.drupal.org/community/events/drupal-pune-december23-meetup-2023-12-16

## Case Studies

---

# Global EdTech Discovery Platform

> Led the complete frontend build of a UNICEF-backed global EdTech discovery platform — a headless Drupal CMS paired with a Next.js frontend — designed to surface expert-curated digital learning tools for educators, policymakers, and governments worldwide.

- **Organisation:** QED42
- **Role:** Lead Frontend Engineer
- **Duration:** May 2024 – Present
- **Client:** Confidential
- **Tags:** Next.js, Drupal, TypeScript, JSON:API, PHP, Twig

**Outcome:** Beta launched at Slush 2024 with 500+ expert-evaluated EdTech tools, now serving 2,000+ monthly visitors across a UNICEF-backed multi-institutional platform

_Some details are omitted or anonymised under a non-disclosure agreement._

Source: https://www.ruturaj.xyz/work/edtech-discovery-platform

## Overview

A major global initiative — developed by UNICEF's Global Learning Innovation Hub in partnership with the Ministry for Foreign Affairs of Finland, the Asian Development Bank (ADB), and Arm — needed a platform to surface high-quality, rigorously evaluated EdTech tools for educators, governments, and policymakers worldwide.

The platform is built on the EdTech for Good Framework: every tool listed undergoes expert evaluation across learning impact, scalability, inclusivity, and data security before being published. The result needed to be a fast, searchable discovery experience backed by a CMS capable of managing a continuously growing catalogue — without developer involvement in day-to-day content operations.

The stack: headless Drupal 10 as the content and editorial backend, Next.js App Router as the public-facing frontend.

### System Architecture

```mermaid
graph TD
    subgraph CMS["Drupal 10 · Headless CMS"]
        Admin[Editorial Team]
        Theme[Drupal Theme\nTwig · Admin & Editorial UI]
        Content[Content Model\nContent Types · Taxonomies · Moderation]
        JAPI[JSON:API]
        Admin --> Theme
        Theme --> Content
        Content --> JAPI
    end

    subgraph Transform["Normalisation Layer"]
        NL[Field flattening · Relationship resolution\nImage variant mapping · Typed output]
    end

    subgraph Build["Build Pipeline"]
        SSG[Next.js Static Build\nAll pages pre-rendered at deploy time]
        CDN[CDN · Static Hosting]
        SSG --> CDN
    end

    subgraph Runtime["Runtime · Client-side"]
        CSR[Search & Filters · CSR\nDynamic queries at runtime]
    end

    Users([Educators · Policymakers · Governments])

    JAPI --> NL
    NL -->|build-time fetch| SSG
    Content -->|content publish triggers new build| SSG
    CDN --> Users
    CSR -->|runtime fetch| NL
    Users -->|search| CSR
```

## The Problem

Three distinct challenges shaped the architecture:

- **Structured catalogue at scale**: 500+ EdTech tools, each evaluated across multiple dimensions, needed a flexible content model supporting rich filtering, taxonomies, expert scoring, and editorial workflows — all manageable by non-technical staff.
- **Decoupled yet editorially complete**: The editorial team needed full CMS control (moderation states, previews, structured fields) while the frontend had to be independently deployable and optimised for a global audience — many of whom are in bandwidth-constrained environments.
- **Dual-mode Drupal**: The project required both a traditional Drupal theme for certain admin and editorial interfaces, and a fully decoupled Next.js frontend for the public site. Both paradigms had to coexist cleanly within the same Drupal installation.

## My Role

As Lead Frontend Engineer, I held end-to-end ownership of the frontend stack and contributed to back-end architecture alongside the Drupal team:

- Architecting and building the Next.js App Router frontend from scratch
- Building and maintaining the Drupal theme (Twig templates, CSS, JS) for editorial and admin-facing views
- Designing the content delivery layer between Drupal's JSON:API and Next.js
- Contributing to Drupal content modelling — content types, field configuration, taxonomy structures, and moderation workflows
- Setting up deployment pipelines and environment configurations

## Approach

### Content modelling in Drupal

Each EdTech tool in the catalogue is a structured Drupal content type with fields covering tool metadata, evaluation scores, target audience, subject areas, supported languages, offline capability, and infrastructure requirements. Getting this model right early was critical — it directly shaped what the frontend could filter, search, and display.

Taxonomy vocabularies handled classification (subject, age group, geography, connectivity requirements), enabling multi-dimensional filtering without custom query logic on the frontend.

### Headless delivery via JSON:API

Rather than exposing Drupal's raw JSON:API responses directly to Next.js, we built a normalisation layer that flattened nested relationships and resolved entity references into a predictable, typed structure. This meant the frontend components were insulated from Drupal's internal data shape — content type changes in Drupal didn't ripple into the Next.js codebase.

```ts
type EdTechTool = {
  id: string;
  title: string;
  summary: string;
  evaluationScores: {
    learningImpact: number;
    scalability: number;
    inclusivity: number;
    dataSecurity: number;
  };
  tags: string[];
  languages: string[];
  offlineCapable: boolean;
  targetAgeGroups: string[];
  image: {
    src: string;
    alt: string;
  };
};
```

### Drupal theme for editorial interfaces

Not every view needed to be decoupled. Certain admin and editorial interfaces — tool submission forms, moderation dashboards, evaluation workflows — were better served by Drupal's native theming system. I built and maintained a Drupal theme using Twig templates that gave the editorial team a consistent, branded experience within the CMS while keeping those views out of the Next.js surface area.

### Rendering strategy

The entire site is statically generated at build time. Every page — tool catalogue, individual tool detail, and static content — is SSG-rendered and deployed as a static build. Content updates in Drupal trigger a new build and deployment, keeping the public site in sync with the editorial team's approved changes.

| Page type            | Strategy | Rationale                                           |
| -------------------- | -------- | --------------------------------------------------- |
| Tool catalogue       | SSG      | Built at deploy time, consistent global performance |
| Tool detail          | SSG      | 500+ tool pages pre-rendered at build               |
| Static / about pages | SSG      | Infrequent changes, ideal for static generation     |
| Search & filters     | CSR      | Fully dynamic, user-driven queries                  |

## Key Deliverables

- **Next.js App Router frontend** — full public site implementation with per-page rendering strategies
- **Drupal theme** — Twig-based theme for editorial and admin interfaces within the CMS
- **JSON:API normalisation layer** — typed, predictable content delivery contract between Drupal and Next.js
- **Filtering and discovery UI** — multi-dimensional search and filter across 500+ tools by subject, age group, language, connectivity, and evaluation score
- **Build-triggered deployment pipeline** — Drupal content updates trigger a new static build and deployment
- **Content model contributions** — Drupal content types, taxonomy vocabularies, and moderation workflow configuration

## Results & Impact

- **Beta launched at Slush 2024** — delivered on time for a high-profile international conference debut, representing UNICEF's Global Learning Innovation Hub alongside partners including the Ministry for Foreign Affairs of Finland, ADB, and Arm
- **500+ EdTech tools evaluated and published** — the catalogue went live with a substantial, fully structured dataset, immediately useful to governments and educators
- **Editorial independence** — the content team can manage the full tool lifecycle (submission, evaluation, publication) without engineering involvement
- **2,000+ monthly visitors** — the platform consistently attracts educators, policymakers, and governments from around the world
- **Global reach** — the platform serves users across low-bandwidth and connectivity-constrained environments, a core requirement given its audience in emerging education markets

## Lessons Learned

**Content modelling is frontend architecture.** The Drupal content model directly determines what's possible in the frontend. Early investment in getting taxonomy structures, field relationships, and moderation states right — before writing a single React component — paid off significantly as the catalogue scaled.

**Own the contract between systems.** The normalisation layer between Drupal's JSON:API and Next.js was the most important architectural decision on the frontend side. Every time content types evolved in Drupal, the frontend stayed untouched. Without it, each CMS change would have required parallel frontend updates.

**Dual-mode Drupal is a valid pattern.** Running a Drupal theme alongside a decoupled frontend in the same installation is often dismissed in favour of going fully headless. In practice, keeping editorial and admin interfaces in Drupal's native theme — where Drupal's UX is strongest — while decoupling only the public-facing site gave the team the best of both worlds.

---

# Prette — Ecommerce Marketplace for Sewists

> Built the full-stack integration layer for Prette, a Drupal Commerce-powered marketplace for sewists, connecting a Gatsby JS frontend to a headless Drupal backend via GraphQL — covering authentication, cart, wishlist, product pages, orders, and address management.

- **Organisation:** QED42
- **Role:** Full Stack Engineer
- **Duration:** July 2022 – May 2023
- **Client:** Patternry / Prette
- **Tags:** Drupal, Gatsby JS, React, TypeScript, GraphQL, Drupal Commerce

**Outcome:** Delivered a fully integrated ecommerce storefront for a niche marketplace, enabling sewists to discover and purchase materials for their projects

_Some details are omitted or anonymised under a non-disclosure agreement._

Source: https://www.ruturaj.xyz/work/prette-ecommerce

## Overview

Prette is a marketplace designed for sewists — people who make their own clothes. The platform makes it easy for them to find and purchase everything needed to create the garments they imagine, from fabrics and patterns to notions and accessories.

The project was built on a decoupled architecture: a **Drupal Commerce** backend serving as the content, product catalogue, and commerce engine, paired with a **Gatsby JS** frontend consuming content and commerce data via GraphQL. My role as Full Stack Engineer covered the full breadth of the Drupal–Gatsby integration: building the API connections, implementing commerce features, developing custom Drupal modules, and ensuring a seamless experience from product discovery through checkout and order management.

## The Problem

Building a commerce frontend on top of Drupal Commerce in a decoupled setup is non-trivial. Drupal Commerce's native tools — cart management, authentication, wishlists, address books, orders — are built for server-rendered Drupal themes. Exposing them cleanly to a Gatsby JS frontend required:

- **Custom API surface**: Drupal Commerce doesn't expose all commerce operations through JSON:API or GraphQL by default. Cart mutations, wishlist operations, and address management needed custom API endpoints.
- **Decoupled authentication**: Standard Drupal session authentication doesn't work in a decoupled context. A token-based authentication flow had to be implemented and integrated into the Gatsby frontend.
- **Static + dynamic content**: Gatsby's static generation model works well for product catalogues, but cart, wishlist, and account pages are inherently dynamic. The architecture had to accommodate both.
- **Drupal View integrations**: Several listing and filtering experiences on the frontend were powered by Drupal Views — exposing these as GraphQL-consumable endpoints required configuration and custom module work.

## My Role

As Full Stack Engineer, I was responsible for the full integration layer between Drupal Commerce and the Gatsby frontend:

- Drupal API integrations with UI components
- Decoupled Drupal Authentication
- Custom module development
- Wishlist and Cart Integration
- Drupal View integrations
- Order pages integration
- Address-book integration
- Header Menu integration
- Custom paragraph types
- Product Detail Page and Store Page integration in Gatsby via GraphQL
- Static Site Generation (SSG) and Server-Side Rendering (SSR) implementation
- Custom tokens in Drupal backend
- Reusable templates for Drupal content types
- Google Analytics implementation
- Contributed patches on drupal.org for extended functionality

## Approach

### Decoupled Authentication

Standard Drupal session-based authentication doesn't translate to a Gatsby frontend. I implemented a token-based authentication flow using Drupal's Simple OAuth module, issuing access and refresh tokens to the Gatsby frontend. This allowed user-specific commerce operations — cart, wishlist, orders, account management — to work reliably in the decoupled context.

The authentication layer was integrated into the Gatsby frontend so that protected routes and user-specific API calls handled token attachment and refresh transparently.

### Cart and Wishlist Integration

Drupal Commerce's cart management is designed for server-rendered Drupal pages. Exposing it to the Gatsby frontend required building a custom API layer in Drupal that accepted cart mutation requests (add, update, remove) from the frontend and returned structured responses the React components could consume.

Wishlist functionality followed a similar pattern — custom Drupal module code to expose wishlist read and write operations as REST endpoints, with corresponding React components on the Gatsby side.

### Product Detail Page and Store Page via GraphQL

The Product Detail Page and Store Page were integrated into the Gatsby frontend using GraphQL. The Gatsby GraphQL data layer was configured to pull product and catalogue data from Drupal at build time for static generation, while dynamic data (stock, pricing, user-specific state) was fetched client-side.

This approach meant product pages loaded instantly from the CDN for anonymous users, while authenticated users saw personalised state layered on top.

### Rendering Strategy

| Page type                      | Strategy | Rationale                                                          |
| ------------------------------ | -------- | ------------------------------------------------------------------ |
| Product catalogue / Store page | SSG      | Catalogue data is stable; pre-render at build time for performance |
| Product Detail Page            | SSG      | Individual product pages pre-rendered at build                     |
| Cart / Wishlist                | CSR      | User-specific, dynamic; must reflect live state                    |
| Order history / Account        | SSR      | Authenticated, user-specific content                               |

### Drupal View Integrations

Several listing and filtering experiences — category pages, search results, curated collections — were powered by Drupal Views exposed as REST endpoints. I configured Views to expose the correct fields and filters, then built the Gatsby-side data fetching and UI components to consume them.

### Custom Module Development

Several commerce and content requirements weren't met by Drupal's contributed modules out of the box. I built custom Drupal modules to:

- Expose cart, wishlist, and address-book operations as structured REST endpoints
- Implement custom tokens for use in transactional emails and content
- Extend existing module functionality where patches on drupal.org weren't yet merged

I also contributed patches upstream to drupal.org where I identified bugs or missing functionality in contributed modules the project depended on.

### Custom Paragraph Types and Reusable Templates

The content editorial experience relied on Drupal's Paragraph types for structured, reusable content blocks. I created custom paragraph types to support the design requirements and built corresponding Gatsby templates that rendered content generated by these Drupal content types — ensuring editors could build pages without developer involvement for routine content updates.

### Header Menu Integration

The site navigation was managed in Drupal and consumed by the Gatsby frontend via GraphQL. This allowed the editorial team to update navigation without frontend deployments, while the Gatsby build system kept the static site in sync on each rebuild.

### Google Analytics

Integrated Google Analytics tracking across the Gatsby frontend, covering page views, ecommerce events (product views, add-to-cart, purchases), and user interactions — giving the Prette team visibility into the customer journey through the storefront.

## Key Deliverables

- **Decoupled authentication** — token-based auth flow connecting Gatsby frontend to Drupal backend, enabling user-specific commerce operations
- **Cart and wishlist** — full add/update/remove cart management and wishlist functionality surfaced in the Gatsby UI via custom Drupal API endpoints
- **Product Detail Page and Store Page** — statically generated product and catalogue pages with client-side dynamic state for authenticated users
- **Order pages** — integrated order history and order detail views for authenticated users
- **Address book** — user address management integrated into the account section
- **Drupal View integrations** — listing and filtering experiences driven by Drupal Views and consumed via GraphQL
- **Header Menu** — CMS-managed navigation consumed by Gatsby, editable without frontend deploys
- **Custom paragraph types and templates** — reusable Drupal content structures with matching Gatsby render templates
- **Custom Drupal modules** — bespoke backend modules for commerce API endpoints, custom tokens, and extended functionality
- **Google Analytics** — ecommerce and behavioural tracking across the storefront
- **Drupal.org contributions** — patches submitted upstream for bugs and missing functionality in contributed modules

## Lessons Learned

**Decoupled Drupal Commerce needs a deliberate API contract.** Drupal Commerce is designed around server-rendered pages. Exposing its operations — cart mutations, wishlist management, address books — to a decoupled frontend requires building an explicit, intentional API surface. Leaning on whatever JSON:API happens to expose leads to brittle integrations; a custom API layer built around the frontend's actual needs is more robust.

**SSG and dynamic commerce can coexist.** Combining static generation for product catalogue pages with client-side rendering for user-specific state (cart, wishlist, account) gives the best of both: fast, cacheable product pages and accurate, live commerce state. The key is being explicit about which data belongs to which rendering mode from the start.

**Contributed module gaps are opportunities.** Several times during the project, a contributed module was missing a feature or had a bug that blocked progress. Contributing patches upstream — rather than maintaining local forks — kept the codebase clean and gave something back to the Drupal community.

---

# Knowbase — Multi-Tenant Knowledge Base SaaS

> Designed and built a multi-tenant knowledge base platform from scratch, combining PostgreSQL full-text search with a fully async RAG pipeline (Jina AI + pgvector + Gemini) to let teams store, search, and query their documents through both keyword and AI-powered interfaces.

- **Organisation:** Personal
- **Role:** Solo Full Stack Engineer
- **Duration:** Dec 2025 – Present
- **Client:** Self-Initiated
- **Tags:** NestJS, Next.js, TypeScript, PostgreSQL, Drizzle ORM, BullMQ, Supabase, pgvector, Jina AI, Gemini, RAG, Tailwind

**Outcome:** End-to-end RAG pipeline processing three document types (text, URL, PDF) with sub-second full-text search and AI-grounded answers via an HNSW vector index on Supabase

Source: https://www.ruturaj.xyz/work/knowbase

## Overview

Knowbase is a multi-tenant SaaS platform that lets teams organise their collective knowledge into Organisations and Workspaces, ingest documents in multiple formats, and retrieve them through two complementary search modes: traditional full-text search and a retrieval-augmented generation (RAG) pipeline that answers natural-language questions using only the content in the knowledge base.

The system is built on a NestJS API with Drizzle ORM against a Supabase-hosted PostgreSQL instance, paired with a Next.js 16 App Router frontend. The RAG pipeline runs fully asynchronously via BullMQ, embedding documents with Jina AI and querying them through Supabase's pgvector HNSW index before generating answers with Gemini.

### System Architecture

```mermaid
graph TD
    subgraph FE["knowbase-app · Next.js 16 · Port 3001"]
        SC[Server Components\nserverFetch]
        CC[Client Components\nclientFetch + SWR]
        SA[Server Actions]
    end

    subgraph API["knowbase-api · NestJS · Port 3000"]
        Auth[Auth Module\nGoogle OAuth + JWT]
        Orgs[Organisations]
        WS[Workspaces]
        Docs[Documents]
        RAG[RAG Module]
    end

    subgraph Queue["Async Queue"]
        R[(Redis)]
        Q[rag-ingestion queue]
        W[IngestionProcessor\nBullMQ Worker]
        R --> Q --> W
    end

    subgraph DB["Supabase · PostgreSQL"]
        PG[(Relational tables\n+ tsvector FTS)]
        PGV[(pgvector\nHNSW index)]
    end

    subgraph AI["External AI Services"]
        Jina[Jina AI\njina-embeddings-v2-base-en\n768 dimensions]
        Gemini[Google Gemini\ngemini-2.0-flash]
    end

    SC -->|"cookies + X-Organisation header"| API
    CC -->|"credentials: include"| API
    SA --> API
    API --> PG
    RAG -->|"enqueue job"| Q
    W -->|"batch embed chunks"| Jina
    W -->|"store embeddings"| PGV
    W -->|"update status"| PG
    RAG -->|"embed query"| Jina
    RAG -->|"match_document_chunks RPC"| PGV
    RAG -->|"generate answer"| Gemini
```

---

## The Problem

Most team knowledge ends up fragmented — scattered across Notion pages, PDFs, internal wikis, and web links that nobody ever searches. The challenge was building a platform where teams could ingest content in whatever format it lives in, search it accurately across large corpora, and — crucially — ask questions against it without getting answers invented from outside their own knowledge base.

Three problems had to be solved in parallel:

- **Ingestion heterogeneity**: Teams store knowledge as typed text, scraped web pages, and uploaded PDFs. Each format needs a different extraction path before any indexing can happen.
- **Search duality**: Keyword search is fast and familiar but misses semantic relationships. Pure vector search is powerful but opaque. Both modes need to coexist in a single coherent UI.
- **RAG hallucination risk**: LLMs will confidently answer outside their context. The pipeline had to strictly ground Gemini's output in retrieved chunks and return a fallback when the answer isn't in the documents.

---

## My Role

Sole architect and engineer across the full stack — API design, database schema, RAG pipeline, async worker infrastructure, and the Next.js frontend.

Primary areas of ownership:

- Multi-tenant data model and role-based access control layer
- Document ingestion pipeline for three content types
- Async RAG ingestion worker (chunking → embedding → pgvector storage)
- Dual search modes: PostgreSQL tsvector full-text and Supabase RPC vector similarity
- Next.js App Router frontend with server/client fetch split and SWR polling

---

## Approach

### Multi-tenant isolation

The tenancy model is three levels deep: **Organisation → Workspace → Document**. Membership at each level is tracked separately, with independent role enums (`owner | admin | member` for orgs; `owner | admin | editor | viewer` for workspaces). The critical design decision was that `workspace_members` links to `organisation_members.id`, not directly to `users` — workspace access is always derived through org membership, making it impossible for a workspace to be accessed by someone outside the parent organisation.

Every document route accepts a workspace identifier that can be either a UUID or a slug, resolved inside `assertWorkspaceAccess` via a single join query that simultaneously validates membership and returns the workspace ID.

```mermaid
erDiagram
    users {
        uuid id PK
        text email
        text googleId
        text refreshTokenHash
    }
    organisations {
        uuid id PK
        text name
        text slug "globally unique"
        uuid createdBy FK
    }
    organisation_members {
        uuid id PK
        uuid organisationId FK
        uuid userId FK
        enum role "owner | admin | member"
    }
    workspaces {
        uuid id PK
        text name
        text slug "unique per org"
        uuid organisationId FK
    }
    workspace_members {
        uuid id PK
        uuid workspaceId FK
        uuid organisationMemberId FK
        enum role "owner | admin | editor | viewer"
    }
    documents {
        uuid id PK
        uuid workspaceId FK
        uuid createdByMemberId FK
        text title
        enum type "text | url | pdf"
        enum status "ready | processing | failed"
        timestamp archivedAt "soft delete"
    }
    document_contents {
        uuid documentId PK,FK
        text rawContent
    }
    document_chunks {
        uuid id PK
        uuid documentId FK
        int chunkIndex
        text content
    }
    document_chunk_embeddings {
        uuid chunkId PK,FK
        vector embedding "768-dim HNSW"
        text model
    }

    users ||--o{ organisation_members : "belongs to"
    organisations ||--o{ organisation_members : "has"
    organisation_members ||--o{ workspace_members : "maps to"
    workspaces ||--o{ workspace_members : "has"
    organisations ||--o{ workspaces : "contains"
    workspaces ||--o{ documents : "contains"
    documents ||--|| document_contents : "has"
    documents ||--o{ document_chunks : "split into"
    document_chunks ||--|| document_chunk_embeddings : "embedded as"
```

### Auth flow

```mermaid
sequenceDiagram
    actor User
    participant App as Next.js App
    participant API as NestJS API
    participant Google as Google OAuth
    participant DB as PostgreSQL

    User->>App: Click "Sign in with Google"
    App->>API: GET /auth/google
    API->>Google: Redirect to OAuth consent screen
    Google-->>API: Callback with auth code
    API->>Google: Exchange code for user profile
    API->>DB: Find or create user record
    API->>DB: Store bcrypt(refreshToken)
    API-->>App: Set httpOnly cookies
    Note right of API: accessToken (15m)<br/>refreshToken (7d)
    App-->>User: Authenticated session

    Note over App,API: On 401 — access token expired
    App->>API: POST /auth/refresh (singleton promise)
    API->>DB: bcrypt.compare(refreshToken, hash)
    API->>DB: Update refreshTokenHash
    API-->>App: Rotate both cookies
    App->>API: Retry original request
```

### Document ingestion — three paths, one pipeline

| Type   | Extraction                                                                                       |
| ------ | ------------------------------------------------------------------------------------------------ |
| `text` | Raw content from request body, stored directly                                                   |
| `url`  | Cheerio scrapes the page, strips `script / style / nav / header / footer`, normalises whitespace |
| `pdf`  | `pdf-parse` v2 class API (`new PDFParse({ data: buffer }).getText()`) extracts body text         |

All three paths converge at the same transaction: insert into `documents`, insert raw content into `document_contents`, then optionally enqueue for RAG indexing. The `isIndexed` flag on the DTO controls whether the document enters the async pipeline immediately or stays as a plain searchable document.

```mermaid
flowchart TD
    A([POST /documents\nor /documents/upload]) --> B{type?}
    B -->|text| C[Use request body\nas rawContent]
    B -->|url| D[Cheerio fetch + scrape\nstrip nav/script/style]
    B -->|pdf| E[pdf-parse v2\nextract text from buffer]
    C & D & E --> F

    F[DB transaction\ninsert documents\ninsert document_contents] --> G{isIndexed?}
    G -->|false| H([status: ready\nFTS only])
    G -->|true| I([status: processing\nEnqueue BullMQ job])
```

### RAG pipeline design

When `isIndexed: true`, the document status is set to `processing` and a BullMQ job is enqueued on the `rag-ingestion` queue. The `IngestionProcessor` worker handles each job in five steps:

1. **Chunking** — `ChunkingService` splits text into sentence-aware chunks of 1500 chars with 150-char overlap, preventing mid-sentence cuts at chunk boundaries.
2. **Insert chunks** — chunk records written to `document_chunks` with `chunkIndex` for ordered retrieval.
3. **Embed** — `EmbeddingService` sends all chunks in a single batch request to Jina AI's `jina-embeddings-v2-base-en` model (768 dimensions).
4. **Store embeddings** — `document_chunk_embeddings` receives chunk IDs paired with their embedding vectors.
5. **Status update** — document status set to `ready` on success, `failed` on error. BullMQ handles retries (3 attempts, exponential backoff at 2s).

Force re-indexing deletes stale chunks first (cascade removes their embeddings), then re-enqueues. This makes re-indexing safe to call idempotently.

```mermaid
flowchart TD
    A([BullMQ job received\ningest-manual-document]) --> B[ChunkingService\n1500 char chunks\n150 char sentence-aware overlap]
    B --> C[Insert document_chunks\nwith chunkIndex]
    C --> D[EmbeddingService\nBatch POST to Jina AI\njina-embeddings-v2-base-en]
    D --> E[Insert document_chunk_embeddings\n768-dim vectors\nHNSW indexed]
    E --> F([Update status → ready])

    A -->|exception| G([Update status → failed])
    G -->|attempt < 3| H([BullMQ retry\nexponential backoff 2s])
    H --> A

    style F fill:#16a34a,color:#fff
    style G fill:#dc2626,color:#fff
```

### Dual search

Full-text search uses PostgreSQL's `tsvector` columns on both `documents` and `document_contents`, combined at query time:

```sql
WHERE (d.search_vector || dc.search_vector)
      @@ plainto_tsquery('english', $1)
ORDER BY ts_rank(d.search_vector || dc.search_vector, plainto_tsquery('english', $1)) DESC
```

RAG mode embeds the query with the same Jina model, calls Supabase's `match_document_chunks` RPC (cosine similarity on the HNSW index, top-K chunks), and passes the retrieved context to Gemini with an explicit grounding prompt that instructs it to answer only from the provided context and return a fallback if the answer isn't present.

```mermaid
flowchart TD
    A([User query]) --> B{Search mode}

    B -->|simple| C[plainto_tsquery\n'english']
    C --> D[tsvector match across\ndocuments + document_contents]
    D --> E[ts_rank ORDER BY DESC]
    E --> F([Ranked document list])

    B -->|rag| G[EmbeddingService\nJina AI embed query\n768-dim vector]
    G --> H[Supabase RPC\nmatch_document_chunks\ncosine similarity on HNSW]
    H --> I[Top-K chunks retrieved]
    I --> J[GeminiService\ngrounded prompt\nanswer from context only]
    J --> K{Answer found\nin context?}
    K -->|yes| L([AI-generated answer])
    K -->|no| M(["I don't know based\non the provided information."])

    style L fill:#16a34a,color:#fff
    style M fill:#6b7280,color:#fff
```

### Frontend architecture

The Next.js app separates server and client fetch paths at the module level using `"server-only"`. `serverFetch` reads the `X-Organisation` cookie and forwards it as a header alongside all session cookies — no client-side header management needed in server components. `clientFetch` implements a singleton refresh-promise to prevent concurrent token refresh races when multiple SWR hooks fire simultaneously on a 401.

The search UI surfaces both modes in a single page: simple mode renders a standard results list; RAG mode renders a chat-style interface with message history, auto-scroll, auto-refocus after the assistant responds, and a clear-conversation button. Documents with `status === "processing"` trigger a 4-second SWR polling interval that stops automatically once all documents reach a terminal state.

```mermaid
flowchart TD
    subgraph Server["Server-side (RSC + Server Actions)"]
        SC[Server Component] -->|serverFetch| API
        SA[Server Action] -->|serverFetch| API
        SF[serverFetch\nreads cookies\nforwards X-Organisation header]
        SC --> SF
        SA --> SF
    end

    subgraph Client["Client-side (SWR hooks)"]
        CC[Client Component] -->|useDocuments\nuseWorkspaces\nuseSearch| SWR
        SWR -->|clientFetch| API
        CF[clientFetch\ncredentials: include\nsingleton refresh promise]
        SWR --> CF
    end

    API[(NestJS API\nPort 3000)]

    CF -->|401| Refresh[POST /auth/refresh\nrotate cookies]
    Refresh -->|success| Retry[Retry original request]
    Refresh -->|fail| Login([Redirect /login])

    SF -->|401| Login
```

---

## Key Deliverables

- **Multi-tenant API** — NestJS with Drizzle ORM, two-level membership model, guard-based context injection, soft-delete for documents
- **Three-format ingestion** — text, URL (Cheerio), PDF (`pdf-parse`) converging into a single document/content schema
- **Async RAG pipeline** — BullMQ worker with chunking, Jina AI batch embedding, pgvector storage, and document lifecycle state machine
- **Dual search interface** — PostgreSQL tsvector full-text and Supabase HNSW vector similarity with grounded Gemini generation
- **Next.js App Router frontend** — server/client fetch split, SWR with status polling, markdown rendering, table/card view toggle with View Transition API
- **Settings pages** — org and workspace rename/delete (owner-only, with role-checked redirects server-side)

---

## Technical Highlights

### Grounded RAG prompt

Gemini is given a strict system prompt that prevents it from answering outside the retrieved context — a deliberate choice to make the knowledge base trustworthy rather than impressively fluent.

```ts
buildGroundedPrompt(input: GroundedPromptInput): string {
  const context = input.chunks
    .map((c, i) => `Context ${i + 1}:\n${c.content}`)
    .join("\n\n");

  return `
    You are an assistant that answers questions using ONLY the provided context.

    Rules:
    - Use only the context below
    - If the answer is not in the context, say: "I don't know based on the provided information."
    - Do not use prior knowledge.

    Context:
    ${context}

    Question:
    ${input.query}
    `;
}
```

### Processing status polling

SWR's `refreshInterval` is set conditionally — it only polls when there's at least one document in the `processing` state, dropping to zero once all documents settle. This avoids constant background requests for quiescent workspaces.

```ts
refreshInterval: documents?.some((d) => d.status === "processing") ? 4000 : 0;
```

### Membership-scoped access check

Every document operation resolves the workspace through a membership join in one query, which simultaneously validates access and returns the workspace ID. No separate permission check is needed.

```ts
const [member] = await this.db
  .select({ id: schema.workspaces.id })
  .from(schema.workspaceMembers)
  .innerJoin(
    schema.workspaces,
    eq(schema.workspaceMembers.workspaceId, schema.workspaces.id),
  )
  .where(
    and(
      isUuid
        ? eq(schema.workspaces.id, workspaceIdentifier)
        : eq(schema.workspaces.slug, workspaceIdentifier),
      eq(schema.workspaceMembers.organisationMemberId, organisationMemberId),
      eq(schema.workspaces.organisationId, organisationId),
    ),
  )
  .limit(1);

if (!member)
  throw new ForbiddenException("You do not have access to this workspace");
```

---

## Results & Impact

- **Full RAG pipeline end-to-end** — documents are chunked, embedded, and stored asynchronously; queries return Gemini-generated answers grounded strictly in the workspace's own content
- **Three ingestion formats** — teams can add knowledge without reformatting it; URL scraping and PDF extraction happen server-side with no client involvement
- **Sub-second full-text search** — `tsvector` with `plainto_tsquery` and `ts_rank` returns ranked results across document titles and full content bodies
- **Zero hallucination by design** — the RAG prompt's grounding rules mean the assistant explicitly declines to answer when retrieved context doesn't contain the answer

---

## Lessons Learned

**Separate the membership model from the user model early.** Making `workspace_members` link to `organisation_members` (not `users`) meant workspace access is always derived through org membership. This felt like over-engineering at first but prevented an entire class of cross-tenant access bugs later.

**Async status needs a state machine, not a boolean.** Using `processing | ready | failed` instead of a simple `isIndexed` boolean gave the frontend meaningful states to render (spinner, success, retry action) without polling the embeddings table directly. The status column became the single source of truth for pipeline progress.

**Chunk at sentence boundaries, not character boundaries.** A naive character-split chunker would cut mid-sentence, degrading embedding quality and making retrieval less accurate. The sentence-aware chunker with overlap keeps semantic units intact across chunk boundaries, at the cost of slightly more complexity in the splitting logic.

**BullMQ's job lifecycle events are the right observability hook.** Rather than building a separate progress-tracking system, `@OnWorkerEvent('active' | 'completed' | 'failed')` gives enough signal to log and debug pipeline failures without adding another dependency.