# Redirect Intelligence v2 - Detailed Implementation Plan Based on the `redirect_intelligence_v2_plan.md` and current application state, this document provides a phase-by-phase implementation guide for upgrading to **Redirect Intelligence v2** with **Node/Express + Prisma(Postgres) + React/Chakra**. ## Current State Analysis From `comprehensive_app_documentation.md`, the current application has: - ✅ Express.js server with redirect tracking - ✅ Rate limiting (100 req/hour/IP) - ✅ SSL certificate analysis - ✅ Basic frontend with dark/light mode - ✅ API endpoints: `/api/track`, `/api/v1/track` - ✅ Security warnings (loops, SSL downgrades) - ✅ Response body truncation and metadata capture ## Implementation Strategy **Backward Compatibility:** All existing endpoints (`/api/track`, `/api/v1/track`) will be preserved with identical behavior. **Migration Approach:** Gradual migration with feature flags to ensure zero downtime. --- ## Phase 0: Repo & Env (Dockerized) ### Goals - Restructure project for TypeScript - Add Docker Compose with all services - Setup development environment ### Files to Create/Modify #### 1. Project Structure ``` / ├── apps/ │ ├── api/ # Express.js API (TypeScript) │ │ ├── src/ │ │ │ ├── index.ts │ │ │ ├── routes/ │ │ │ ├── middleware/ │ │ │ ├── services/ │ │ │ └── types/ │ │ ├── Dockerfile │ │ ├── package.json │ │ └── tsconfig.json │ ├── web/ # React frontend │ │ ├── src/ │ │ │ ├── components/ │ │ │ ├── pages/ │ │ │ ├── hooks/ │ │ │ └── types/ │ │ ├── Dockerfile │ │ ├── package.json │ │ └── tsconfig.json │ └── worker/ # BullMQ worker │ ├── src/ │ ├── Dockerfile │ └── package.json ├── packages/ │ ├── database/ # Prisma schema & migrations │ │ ├── prisma/ │ │ └── package.json │ └── shared/ # Shared types & utilities │ ├── src/ │ └── package.json ├── docker-compose.yml ├── docker-compose.dev.yml └── package.json # Root workspace ``` #### 2. Docker Compose Configuration **`docker-compose.yml`** ```yaml version: '3.8' services: postgres: image: postgres:15 environment: POSTGRES_DB: redirect_intelligence POSTGRES_USER: postgres POSTGRES_PASSWORD: postgres ports: - "5432:5432" volumes: - postgres_data:/var/lib/postgresql/data redis: image: redis:7-alpine ports: - "6379:6379" volumes: - redis_data:/data api: build: context: . dockerfile: apps/api/Dockerfile ports: - "3333:3333" environment: - DATABASE_URL=postgresql://postgres:postgres@postgres:5432/redirect_intelligence - REDIS_URL=redis://redis:6379 - NODE_ENV=development depends_on: - postgres - redis volumes: - ./apps/api:/app - /app/node_modules web: build: context: . dockerfile: apps/web/Dockerfile ports: - "3000:3000" environment: - REACT_APP_API_URL=http://localhost:3333 depends_on: - api volumes: - ./apps/web:/app - /app/node_modules worker: build: context: . dockerfile: apps/worker/Dockerfile environment: - DATABASE_URL=postgresql://postgres:postgres@postgres:5432/redirect_intelligence - REDIS_URL=redis://redis:6379 depends_on: - postgres - redis volumes: - ./apps/worker:/app - /app/node_modules volumes: postgres_data: redis_data: ``` #### 3. TypeScript Configuration **Root `package.json`** ```json { "name": "redirect-intelligence-v2", "private": true, "workspaces": [ "apps/*", "packages/*" ], "scripts": { "dev": "docker-compose -f docker-compose.yml -f docker-compose.dev.yml up", "build": "turbo run build", "test": "turbo run test", "lint": "turbo run lint", "db:migrate": "cd packages/database && npx prisma migrate dev", "db:seed": "cd packages/database && npx prisma db seed" }, "devDependencies": { "turbo": "^1.10.0", "typescript": "^5.0.0", "@types/node": "^20.0.0" } } ``` #### 4. Migration Script **`migrate-existing.ts`** ```typescript // Script to migrate current index.js logic to TypeScript structure // Preserves all existing functionality while adding new structure ``` ### Implementation Steps 1. Create new directory structure 2. Migrate existing `index.js` to TypeScript in `apps/api/src/` 3. Preserve existing routes with identical behavior 4. Setup Docker containers 5. Add development scripts 6. Test backward compatibility ### Commit Message ``` feat(phase-0): setup Docker Compose with TypeScript structure - Restructure project with apps/ and packages/ - Add Docker Compose for api, web, db, redis, worker - Migrate existing Express.js logic to TypeScript - Preserve all existing API endpoints and behavior - Setup development environment with hot reload ``` --- ## Phase 1: Postgres + Prisma + Auth ### Goals - Add PostgreSQL with Prisma ORM - Implement user authentication with Argon2 - Create database schema - Add JWT-based session management ### Files to Create/Modify #### 1. Database Schema **`packages/database/prisma/schema.prisma`** ```prisma generator client { provider = "prisma-client-js" } datasource db { provider = "postgresql" url = env("DATABASE_URL") } model User { id String @id @default(cuid()) email String @unique name String passwordHash String @map("password_hash") createdAt DateTime @default(now()) @map("created_at") lastLoginAt DateTime? @map("last_login_at") memberships OrgMembership[] auditLogs AuditLog[] @@map("users") } model Organization { id String @id @default(cuid()) name String plan String @default("free") createdAt DateTime @default(now()) @map("created_at") memberships OrgMembership[] projects Project[] apiKeys ApiKey[] auditLogs AuditLog[] @@map("organizations") } model OrgMembership { id String @id @default(cuid()) orgId String @map("org_id") userId String @map("user_id") role Role organization Organization @relation(fields: [orgId], references: [id], onDelete: Cascade) user User @relation(fields: [userId], references: [id], onDelete: Cascade) @@unique([orgId, userId]) @@map("org_memberships") } model Project { id String @id @default(cuid()) orgId String @map("org_id") name String settingsJson Json @map("settings_json") @default("{}") createdAt DateTime @default(now()) @map("created_at") organization Organization @relation(fields: [orgId], references: [id], onDelete: Cascade) checks Check[] bulkJobs BulkJob[] @@map("projects") } model Check { id String @id @default(cuid()) projectId String @map("project_id") inputUrl String @map("input_url") method String @default("GET") headersJson Json @map("headers_json") @default("{}") userAgent String? @map("user_agent") startedAt DateTime @map("started_at") finishedAt DateTime? @map("finished_at") status CheckStatus finalUrl String? @map("final_url") totalTimeMs Int? @map("total_time_ms") reportId String? @map("report_id") project Project @relation(fields: [projectId], references: [id], onDelete: Cascade) hops Hop[] sslInspections SslInspection[] seoFlags SeoFlags? securityFlags SecurityFlags? reports Report[] @@index([projectId, startedAt(sort: Desc)]) @@map("checks") } model Hop { id String @id @default(cuid()) checkId String @map("check_id") hopIndex Int @map("hop_index") url String scheme String? statusCode Int? @map("status_code") redirectType RedirectType @map("redirect_type") latencyMs Int? @map("latency_ms") contentType String? @map("content_type") reason String? responseHeadersJson Json @map("response_headers_json") @default("{}") check Check @relation(fields: [checkId], references: [id], onDelete: Cascade) @@index([checkId, hopIndex]) @@map("hops") } model SslInspection { id String @id @default(cuid()) checkId String @map("check_id") host String validFrom DateTime? @map("valid_from") validTo DateTime? @map("valid_to") daysToExpiry Int? @map("days_to_expiry") issuer String? protocol String? warningsJson Json @map("warnings_json") @default("[]") check Check @relation(fields: [checkId], references: [id], onDelete: Cascade) @@map("ssl_inspections") } model SeoFlags { id String @id @default(cuid()) checkId String @unique @map("check_id") robotsTxtStatus String? @map("robots_txt_status") robotsTxtRulesJson Json @map("robots_txt_rules_json") @default("{}") metaRobots String? @map("meta_robots") canonicalUrl String? @map("canonical_url") sitemapPresent Boolean @default(false) @map("sitemap_present") noindex Boolean @default(false) nofollow Boolean @default(false) check Check @relation(fields: [checkId], references: [id], onDelete: Cascade) @@map("seo_flags") } model SecurityFlags { id String @id @default(cuid()) checkId String @unique @map("check_id") safeBrowsingStatus String? @map("safe_browsing_status") mixedContent MixedContent @map("mixed_content") @default(NONE) httpsToHttp Boolean @map("https_to_http") @default(false) check Check @relation(fields: [checkId], references: [id], onDelete: Cascade) @@map("security_flags") } model Report { id String @id @default(cuid()) checkId String @map("check_id") markdownPath String? @map("markdown_path") pdfPath String? @map("pdf_path") createdAt DateTime @default(now()) @map("created_at") check Check @relation(fields: [checkId], references: [id], onDelete: Cascade) @@map("reports") } model BulkJob { id String @id @default(cuid()) projectId String @map("project_id") uploadPath String @map("upload_path") status JobStatus progressJson Json @map("progress_json") @default("{}") createdAt DateTime @default(now()) @map("created_at") completedAt DateTime? @map("completed_at") project Project @relation(fields: [projectId], references: [id], onDelete: Cascade) @@map("bulk_jobs") } model ApiKey { id String @id @default(cuid()) orgId String @map("org_id") name String tokenHash String @unique @map("token_hash") permsJson Json @map("perms_json") @default("{}") rateLimitQuota Int @map("rate_limit_quota") @default(1000) createdAt DateTime @default(now()) @map("created_at") organization Organization @relation(fields: [orgId], references: [id], onDelete: Cascade) @@index([tokenHash]) @@map("api_keys") } model AuditLog { id String @id @default(cuid()) orgId String @map("org_id") actorUserId String? @map("actor_user_id") action String entity String entityId String @map("entity_id") metaJson Json @map("meta_json") @default("{}") createdAt DateTime @default(now()) @map("created_at") organization Organization @relation(fields: [orgId], references: [id], onDelete: Cascade) actor User? @relation(fields: [actorUserId], references: [id], onDelete: SetNull) @@map("audit_logs") } enum Role { OWNER ADMIN MEMBER } enum CheckStatus { OK ERROR TIMEOUT LOOP } enum RedirectType { HTTP_301 HTTP_302 HTTP_307 HTTP_308 META_REFRESH JS FINAL OTHER } enum MixedContent { NONE PRESENT FINAL_TO_HTTP } enum JobStatus { QUEUED RUNNING DONE ERROR } ``` #### 2. Authentication Service **`apps/api/src/services/auth.service.ts`** ```typescript import argon2 from 'argon2'; import jwt from 'jsonwebtoken'; import { z } from 'zod'; import { prisma } from '@/lib/prisma'; const loginSchema = z.object({ email: z.string().email(), password: z.string().min(8), }); export class AuthService { async hashPassword(password: string): Promise { return argon2.hash(password); } async verifyPassword(hash: string, password: string): Promise { return argon2.verify(hash, password); } async login(data: z.infer) { const { email, password } = loginSchema.parse(data); const user = await prisma.user.findUnique({ where: { email }, include: { memberships: { include: { organization: true } } } }); if (!user || !await this.verifyPassword(user.passwordHash, password)) { throw new Error('Invalid credentials'); } await prisma.user.update({ where: { id: user.id }, data: { lastLoginAt: new Date() } }); const token = jwt.sign( { userId: user.id, email: user.email }, process.env.JWT_SECRET!, { expiresIn: '7d' } ); return { user, token }; } async createUser(email: string, name: string, password: string) { const existingUser = await prisma.user.findUnique({ where: { email } }); if (existingUser) { throw new Error('User already exists'); } const passwordHash = await this.hashPassword(password); return prisma.user.create({ data: { email, name, passwordHash, } }); } } ``` #### 3. Authentication Middleware **`apps/api/src/middleware/auth.middleware.ts`** ```typescript import { Request, Response, NextFunction } from 'express'; import jwt from 'jsonwebtoken'; import { prisma } from '@/lib/prisma'; export interface AuthenticatedRequest extends Request { user?: { id: string; email: string; memberships: Array<{ orgId: string; role: string; organization: { name: string; plan: string }; }>; }; } export const authenticateToken = async ( req: AuthenticatedRequest, res: Response, next: NextFunction ) => { const authHeader = req.headers.authorization; const token = authHeader && authHeader.split(' ')[1]; // Bearer TOKEN if (!token) { return res.status(401).json({ error: 'Access token required' }); } try { const decoded = jwt.verify(token, process.env.JWT_SECRET!) as { userId: string; email: string; }; const user = await prisma.user.findUnique({ where: { id: decoded.userId }, include: { memberships: { include: { organization: true } } } }); if (!user) { return res.status(401).json({ error: 'User not found' }); } req.user = { id: user.id, email: user.email, memberships: user.memberships.map(m => ({ orgId: m.orgId, role: m.role, organization: { name: m.organization.name, plan: m.organization.plan } })) }; next(); } catch (error) { return res.status(403).json({ error: 'Invalid token' }); } }; ``` #### 4. Auth Routes **`apps/api/src/routes/auth.routes.ts`** ```typescript import express from 'express'; import { z } from 'zod'; import { AuthService } from '@/services/auth.service'; import { authenticateToken, AuthenticatedRequest } from '@/middleware/auth.middleware'; const router = express.Router(); const authService = new AuthService(); const loginSchema = z.object({ email: z.string().email(), password: z.string().min(8), }); const registerSchema = z.object({ email: z.string().email(), name: z.string().min(2), password: z.string().min(8), }); // POST /api/v1/auth/login router.post('/login', async (req, res) => { try { const { user, token } = await authService.login(req.body); // Set HttpOnly cookie res.cookie('auth_token', token, { httpOnly: true, secure: process.env.NODE_ENV === 'production', sameSite: 'strict', maxAge: 7 * 24 * 60 * 60 * 1000, // 7 days }); res.json({ success: true, user: { id: user.id, email: user.email, name: user.name, memberships: user.memberships } }); } catch (error) { res.status(400).json({ success: false, error: error instanceof Error ? error.message : 'Login failed' }); } }); // POST /api/v1/auth/register router.post('/register', async (req, res) => { try { const { email, name, password } = registerSchema.parse(req.body); const user = await authService.createUser(email, name, password); res.status(201).json({ success: true, user: { id: user.id, email: user.email, name: user.name } }); } catch (error) { res.status(400).json({ success: false, error: error instanceof Error ? error.message : 'Registration failed' }); } }); // POST /api/v1/auth/logout router.post('/logout', (req, res) => { res.clearCookie('auth_token'); res.json({ success: true }); }); // GET /api/v1/auth/me router.get('/me', authenticateToken, (req: AuthenticatedRequest, res) => { res.json({ success: true, user: req.user }); }); export default router; ``` #### 5. Updated Main Server **`apps/api/src/index.ts`** (migrate from existing index.js) ```typescript import express from 'express'; import cors from 'cors'; import cookieParser from 'cookie-parser'; import rateLimit from 'express-rate-limit'; import { trackRedirects } from '@/services/redirect.service'; // Migrated from existing import authRoutes from '@/routes/auth.routes'; import { prisma } from '@/lib/prisma'; const app = express(); const PORT = process.env.PORT || 3333; // Middleware app.use(cors({ origin: process.env.WEB_URL || 'http://localhost:3000', credentials: true })); app.use(express.json()); app.use(express.urlencoded({ extended: true })); app.use(cookieParser()); // Rate limiting (preserve existing behavior) const apiLimiter = rateLimit({ windowMs: 60 * 60 * 1000, // 1 hour max: 100, message: { error: 'Too many requests, please try again later.' } }); // Routes app.use('/api/v1/auth', authRoutes); // PRESERVE EXISTING ENDPOINTS - Backward Compatibility app.post('/api/track', apiLimiter, async (req, res) => { // Exact same logic as before - no changes // ... existing implementation }); app.post('/api/v1/track', apiLimiter, async (req, res) => { // Exact same logic as before - no changes // ... existing implementation }); app.get('/api/v1/track', apiLimiter, async (req, res) => { // Exact same logic as before - no changes // ... existing implementation }); // Health check app.get('/health', (req, res) => { res.json({ status: 'ok', timestamp: new Date().toISOString() }); }); app.listen(PORT, () => { console.log(`Server running on http://localhost:${PORT}`); }); ``` ### Implementation Steps 1. Setup Prisma with PostgreSQL 2. Create database schema and run migrations 3. Implement authentication service with Argon2 4. Add JWT middleware for protected routes 5. Create auth routes (login, register, logout, me) 6. Migrate existing server logic to TypeScript 7. Ensure all existing endpoints work identically 8. Add comprehensive tests ### Testing Requirements - Unit tests for auth service - Integration tests for auth routes - Backward compatibility tests for existing endpoints - Database migration tests ### Commit Message ``` feat(phase-1): add Postgres + Prisma + Auth with backward compatibility - Add PostgreSQL database with Prisma ORM - Implement user authentication with Argon2 password hashing - Add JWT-based session management with HttpOnly cookies - Create comprehensive database schema for all entities - Migrate existing Express.js logic to TypeScript - Preserve 100% backward compatibility for existing API endpoints - Add auth routes: login, register, logout, me - Include comprehensive test suite ``` --- ## Phase 2: Persisted Checks (Non-JS Chain) ### Goals - Persist redirect chain analysis to database - Create new `/api/v1/checks` endpoint that stores history - Maintain existing endpoints with identical behavior - Add check retrieval and history endpoints ### Files to Create/Modify #### 1. Enhanced Redirect Service **`apps/api/src/services/redirect.service.ts`** (enhanced from existing) ```typescript import axios from 'axios'; import https from 'https'; import { z } from 'zod'; import { prisma } from '@/lib/prisma'; import { CheckStatus, RedirectType } from '@prisma/client'; const createCheckSchema = z.object({ inputUrl: z.string().url(), method: z.enum(['GET', 'HEAD', 'POST']).default('GET'), userAgent: z.string().optional(), headers: z.record(z.string()).default({}), projectId: z.string().optional(), // Optional for anonymous checks }); export interface RedirectHop { url: string; statusCode?: number; statusText?: string; redirectType: RedirectType; latencyMs: number; contentType?: string; reason?: string; responseHeaders: Record; timestamp: Date; } export interface RedirectResult { checkId?: string; // Only set when persisting inputUrl: string; finalUrl: string; totalTimeMs: number; status: CheckStatus; hops: RedirectHop[]; metadata: { redirectCount: number; method: string; userAgent?: string; }; } export class RedirectService { // Existing trackRedirects logic preserved exactly for backward compatibility async trackRedirectsLegacy( url: string, redirects: any[] = [], options: any = {} ): Promise { // Keep exact existing implementation from index.js // This ensures 100% backward compatibility // ... existing logic } // New enhanced method that persists to database async createCheck( data: z.infer, userId?: string ): Promise { const validatedData = createCheckSchema.parse(data); const startTime = Date.now(); // Create check record const check = await prisma.check.create({ data: { projectId: validatedData.projectId || await this.getDefaultProjectId(userId), inputUrl: validatedData.inputUrl, method: validatedData.method, headersJson: validatedData.headers, userAgent: validatedData.userAgent, startedAt: new Date(), status: CheckStatus.OK, // Will update later }, }); try { // Perform redirect analysis const result = await this.analyzeRedirectChain( validatedData.inputUrl, validatedData.method, validatedData.userAgent, validatedData.headers ); const totalTimeMs = Date.now() - startTime; // Update check with results await prisma.check.update({ where: { id: check.id }, data: { finishedAt: new Date(), finalUrl: result.finalUrl, totalTimeMs, status: result.status, }, }); // Save hops await this.saveHops(check.id, result.hops); return { checkId: check.id, inputUrl: validatedData.inputUrl, finalUrl: result.finalUrl, totalTimeMs, status: result.status, hops: result.hops, metadata: { redirectCount: result.hops.length - 1, method: validatedData.method, userAgent: validatedData.userAgent, }, }; } catch (error) { // Update check with error status await prisma.check.update({ where: { id: check.id }, data: { finishedAt: new Date(), status: CheckStatus.ERROR, totalTimeMs: Date.now() - startTime, }, }); throw error; } } private async analyzeRedirectChain( inputUrl: string, method: string, userAgent?: string, headers: Record = {} ): Promise<{ finalUrl: string; status: CheckStatus; hops: RedirectHop[]; }> { const hops: RedirectHop[] = []; const visitedUrls = new Set(); let currentUrl = inputUrl; let hopIndex = 0; while (hopIndex < 20) { // Max 20 hops to prevent infinite loops if (visitedUrls.has(currentUrl)) { return { finalUrl: currentUrl, status: CheckStatus.LOOP, hops, }; } visitedUrls.add(currentUrl); const hopStartTime = Date.now(); try { const config = { method: hopIndex === 0 ? method : 'GET', // Use specified method only for first request url: currentUrl, maxRedirects: 0, validateStatus: (status: number) => status >= 200 && status < 600, timeout: 15000, headers: { ...headers, ...(userAgent ? { 'User-Agent': userAgent } : {}), }, httpsAgent: new https.Agent({ rejectUnauthorized: false, }), }; const response = await axios(config); const latencyMs = Date.now() - hopStartTime; const hop: RedirectHop = { url: currentUrl, statusCode: response.status, statusText: response.statusText, redirectType: this.determineRedirectType(response), latencyMs, contentType: response.headers['content-type'], responseHeaders: response.headers, timestamp: new Date(), }; hops.push(hop); // Check if this is a redirect if (response.status >= 300 && response.status < 400 && response.headers.location) { currentUrl = new URL(response.headers.location, currentUrl).href; hopIndex++; continue; } // Check for meta refresh if (response.status === 200 && response.headers['content-type']?.includes('text/html')) { const metaRedirect = this.parseMetaRefresh(response.data); if (metaRedirect) { currentUrl = new URL(metaRedirect, currentUrl).href; hop.redirectType = RedirectType.META_REFRESH; hopIndex++; continue; } } // Final destination reached hop.redirectType = RedirectType.FINAL; return { finalUrl: currentUrl, status: CheckStatus.OK, hops, }; } catch (error) { hops.push({ url: currentUrl, redirectType: RedirectType.OTHER, latencyMs: Date.now() - hopStartTime, responseHeaders: {}, timestamp: new Date(), reason: error instanceof Error ? error.message : 'Unknown error', }); return { finalUrl: currentUrl, status: CheckStatus.ERROR, hops, }; } } return { finalUrl: currentUrl, status: CheckStatus.TIMEOUT, hops, }; } private determineRedirectType(response: any): RedirectType { switch (response.status) { case 301: return RedirectType.HTTP_301; case 302: return RedirectType.HTTP_302; case 307: return RedirectType.HTTP_307; case 308: return RedirectType.HTTP_308; default: return RedirectType.OTHER; } } private parseMetaRefresh(html: string): string | null { const metaRefreshRegex = /]*http-equiv=["']?refresh["']?[^>]*content=["']?[^"']*url=([^"';\s>]+)/i; const match = html.match(metaRefreshRegex); return match ? match[1] : null; } private async saveHops(checkId: string, hops: RedirectHop[]): Promise { await prisma.hop.createMany({ data: hops.map((hop, index) => ({ checkId, hopIndex: index, url: hop.url, scheme: new URL(hop.url).protocol.replace(':', ''), statusCode: hop.statusCode, redirectType: hop.redirectType, latencyMs: hop.latencyMs, contentType: hop.contentType, reason: hop.reason, responseHeadersJson: hop.responseHeaders, })), }); } private async getDefaultProjectId(userId?: string): Promise { if (!userId) { // For anonymous checks, create a default project or use a system project return 'anonymous'; } // Get user's first organization's first project, or create one const user = await prisma.user.findUnique({ where: { id: userId }, include: { memberships: { include: { organization: { include: { projects: true } } } } } }); const firstOrg = user?.memberships[0]?.organization; if (firstOrg?.projects[0]) { return firstOrg.projects[0].id; } // Create default project const defaultProject = await prisma.project.create({ data: { name: 'Default Project', orgId: firstOrg!.id, }, }); return defaultProject.id; } async getCheck(checkId: string): Promise { return prisma.check.findUnique({ where: { id: checkId }, include: { hops: { orderBy: { hopIndex: 'asc' } }, sslInspections: true, seoFlags: true, securityFlags: true, project: { include: { organization: true } } }, }); } async getProjectChecks(projectId: string, options: { page?: number; limit?: number; status?: CheckStatus; } = {}): Promise<{ checks: any[]; total: number; page: number; limit: number; }> { const { page = 1, limit = 20, status } = options; const skip = (page - 1) * limit; const where = { projectId, ...(status ? { status } : {}), }; const [checks, total] = await Promise.all([ prisma.check.findMany({ where, include: { hops: { orderBy: { hopIndex: 'asc' } }, sslInspections: true, seoFlags: true, securityFlags: true, }, orderBy: { startedAt: 'desc' }, skip, take: limit, }), prisma.check.count({ where }), ]); return { checks, total, page, limit }; } } ``` #### 2. Check Routes **`apps/api/src/routes/checks.routes.ts`** ```typescript import express from 'express'; import { z } from 'zod'; import { RedirectService } from '@/services/redirect.service'; import { authenticateToken, AuthenticatedRequest } from '@/middleware/auth.middleware'; import rateLimit from 'express-rate-limit'; const router = express.Router(); const redirectService = new RedirectService(); const apiLimiter = rateLimit({ windowMs: 60 * 60 * 1000, // 1 hour max: 100, message: { error: 'Too many requests, please try again later.' } }); const createCheckSchema = z.object({ inputUrl: z.string().url(), method: z.enum(['GET', 'HEAD', 'POST']).default('GET'), userAgent: z.string().optional(), headers: z.record(z.string()).default({}), projectId: z.string().optional(), }); // POST /api/v1/checks - Create new check with persistence router.post('/', apiLimiter, async (req: AuthenticatedRequest, res) => { try { const result = await redirectService.createCheck(req.body, req.user?.id); res.json({ success: true, status: 200, data: result, }); } catch (error) { console.error('Error creating check:', error); res.status(500).json({ success: false, status: 500, error: error instanceof Error ? error.message : 'Failed to create check', }); } }); // GET /api/v1/checks/:id - Get specific check with full details router.get('/:id', async (req, res) => { try { const check = await redirectService.getCheck(req.params.id); if (!check) { return res.status(404).json({ success: false, status: 404, error: 'Check not found', }); } res.json({ success: true, status: 200, data: check, }); } catch (error) { console.error('Error fetching check:', error); res.status(500).json({ success: false, status: 500, error: 'Failed to fetch check', }); } }); export default router; ``` #### 3. Project Routes **`apps/api/src/routes/projects.routes.ts`** ```typescript import express from 'express'; import { z } from 'zod'; import { RedirectService } from '@/services/redirect.service'; import { authenticateToken, AuthenticatedRequest } from '@/middleware/auth.middleware'; const router = express.Router(); const redirectService = new RedirectService(); const getChecksSchema = z.object({ page: z.coerce.number().min(1).default(1), limit: z.coerce.number().min(1).max(100).default(20), status: z.enum(['OK', 'ERROR', 'TIMEOUT', 'LOOP']).optional(), }); // GET /api/v1/projects/:id/checks - Get project check history router.get('/:id/checks', authenticateToken, async (req: AuthenticatedRequest, res) => { try { const { page, limit, status } = getChecksSchema.parse(req.query); const result = await redirectService.getProjectChecks(req.params.id, { page, limit, status: status as any, }); res.json({ success: true, status: 200, data: result, }); } catch (error) { console.error('Error fetching project checks:', error); res.status(500).json({ success: false, status: 500, error: 'Failed to fetch project checks', }); } }); export default router; ``` #### 4. Updated Main Server **`apps/api/src/index.ts`** (add new routes) ```typescript // ... existing imports and setup import checkRoutes from '@/routes/checks.routes'; import projectRoutes from '@/routes/projects.routes'; // ... existing middleware // New routes app.use('/api/v1/checks', checkRoutes); app.use('/api/v1/projects', projectRoutes); // ... existing routes (preserve exactly) ``` ### Implementation Steps 1. Create enhanced RedirectService with database persistence 2. Implement check creation and retrieval endpoints 3. Add project check history endpoint 4. Ensure backward compatibility with existing endpoints 5. Add comprehensive validation with Zod 6. Create database migration for check-related tables 7. Add tests for new functionality ### Testing Requirements - Unit tests for RedirectService methods - Integration tests for new API endpoints - Backward compatibility tests - Database persistence tests - Performance tests for redirect analysis ### Commit Message ``` feat(phase-2): add persisted checks with non-JS redirect chain analysis - Create new /api/v1/checks endpoint that persists redirect analysis - Implement enhanced RedirectService with database storage - Add check retrieval and project history endpoints - Preserve 100% backward compatibility with existing endpoints - Add comprehensive validation with Zod schemas - Support meta refresh redirect detection - Include project-based check organization - Add pagination and filtering for check history ``` --- ## Phase 3: SSL/SEO/Security Flags ### Goals - Add SSL certificate inspection and storage - Implement SEO analysis (robots.txt, meta tags, canonical URLs) - Add security flags (mixed content, safe browsing) - Enhance redirect analysis with comprehensive metadata ### Files to Create/Modify #### 1. SSL Inspection Service **`apps/api/src/services/ssl.service.ts`** ```typescript import https from 'https'; import tls from 'tls'; import { z } from 'zod'; import { prisma } from '@/lib/prisma'; export interface SslCertificateInfo { host: string; validFrom?: Date; validTo?: Date; daysToExpiry?: number; issuer?: string; subject?: string; protocol?: string; valid: boolean; warnings: string[]; } export class SslService { async inspectSslCertificate(url: string): Promise { try { const urlObj = new URL(url); if (urlObj.protocol !== 'https:') { return null; } return new Promise((resolve, reject) => { const options = { host: urlObj.hostname, port: urlObj.port || 443, rejectUnauthorized: false, timeout: 10000, }; const socket = tls.connect(options, () => { try { const cert = socket.getPeerCertificate(true); const now = new Date(); const validFrom = new Date(cert.valid_from); const validTo = new Date(cert.valid_to); const daysToExpiry = Math.floor((validTo.getTime() - now.getTime()) / (1000 * 60 * 60 * 24)); const warnings: string[] = []; // Check certificate validity if (now < validFrom) { warnings.push('Certificate not yet valid'); } if (now > validTo) { warnings.push('Certificate expired'); } if (daysToExpiry <= 30 && daysToExpiry > 0) { warnings.push(`Certificate expires in ${daysToExpiry} days`); } if (!socket.authorized) { warnings.push('Certificate authorization failed'); } const result: SslCertificateInfo = { host: urlObj.hostname, validFrom, validTo, daysToExpiry, issuer: this.formatCertificateName(cert.issuer), subject: this.formatCertificateName(cert.subject), protocol: socket.getProtocol(), valid: socket.authorized && now >= validFrom && now <= validTo, warnings, }; socket.end(); resolve(result); } catch (error) { socket.end(); reject(error); } }); socket.on('error', (error) => { reject(error); }); socket.on('timeout', () => { socket.end(); reject(new Error('SSL connection timeout')); }); }); } catch (error) { console.error('SSL inspection error:', error); return null; } } private formatCertificateName(certName: any): string { if (typeof certName === 'string') return certName; const parts = []; if (certName.CN) parts.push(`CN=${certName.CN}`); if (certName.O) parts.push(`O=${certName.O}`); if (certName.C) parts.push(`C=${certName.C}`); return parts.join(', '); } async saveSslInspection(checkId: string, sslInfo: SslCertificateInfo): Promise { await prisma.sslInspection.create({ data: { checkId, host: sslInfo.host, validFrom: sslInfo.validFrom, validTo: sslInfo.validTo, daysToExpiry: sslInfo.daysToExpiry, issuer: sslInfo.issuer, protocol: sslInfo.protocol, warningsJson: sslInfo.warnings, }, }); } } ``` #### 2. SEO Analysis Service **`apps/api/src/services/seo.service.ts`** ```typescript import axios from 'axios'; import { JSDOM } from 'jsdom'; import { z } from 'zod'; import { prisma } from '@/lib/prisma'; export interface SeoAnalysis { robotsTxtStatus?: string; robotsTxtRules?: Record; metaRobots?: string; canonicalUrl?: string; sitemapPresent: boolean; noindex: boolean; nofollow: boolean; title?: string; description?: string; openGraphData?: Record; } export class SeoService { async analyzeSeo(finalUrl: string, htmlContent?: string): Promise { const analysis: SeoAnalysis = { sitemapPresent: false, noindex: false, nofollow: false, }; try { const urlObj = new URL(finalUrl); const baseUrl = `${urlObj.protocol}//${urlObj.host}`; // Analyze robots.txt const robotsAnalysis = await this.analyzeRobotsTxt(baseUrl); analysis.robotsTxtStatus = robotsAnalysis.status; analysis.robotsTxtRules = robotsAnalysis.rules; analysis.sitemapPresent = robotsAnalysis.sitemapPresent; // Analyze HTML if available if (htmlContent) { const htmlAnalysis = this.analyzeHtml(htmlContent, finalUrl); Object.assign(analysis, htmlAnalysis); } return analysis; } catch (error) { console.error('SEO analysis error:', error); return analysis; } } private async analyzeRobotsTxt(baseUrl: string): Promise<{ status: string; rules: Record; sitemapPresent: boolean; }> { try { const robotsUrl = `${baseUrl}/robots.txt`; const response = await axios.get(robotsUrl, { timeout: 5000, validateStatus: (status) => status < 500, }); if (response.status === 200) { const robotsTxt = response.data; const rules = this.parseRobotsTxt(robotsTxt); const sitemapPresent = robotsTxt.toLowerCase().includes('sitemap:'); return { status: 'found', rules, sitemapPresent, }; } else { return { status: `not_found_${response.status}`, rules: {}, sitemapPresent: false, }; } } catch (error) { return { status: 'error', rules: {}, sitemapPresent: false, }; } } private parseRobotsTxt(content: string): Record { const rules: Record = { userAgents: {}, sitemaps: [], }; const lines = content.split('\n'); let currentUserAgent = '*'; for (const line of lines) { const trimmedLine = line.trim(); if (!trimmedLine || trimmedLine.startsWith('#')) continue; const [directive, ...valueParts] = trimmedLine.split(':'); const value = valueParts.join(':').trim(); switch (directive.toLowerCase()) { case 'user-agent': currentUserAgent = value; if (!rules.userAgents[currentUserAgent]) { rules.userAgents[currentUserAgent] = { allow: [], disallow: [], }; } break; case 'allow': if (!rules.userAgents[currentUserAgent]) { rules.userAgents[currentUserAgent] = { allow: [], disallow: [] }; } rules.userAgents[currentUserAgent].allow.push(value); break; case 'disallow': if (!rules.userAgents[currentUserAgent]) { rules.userAgents[currentUserAgent] = { allow: [], disallow: [] }; } rules.userAgents[currentUserAgent].disallow.push(value); break; case 'sitemap': rules.sitemaps.push(value); break; } } return rules; } private analyzeHtml(htmlContent: string, finalUrl: string): Partial { try { const dom = new JSDOM(htmlContent); const document = dom.window.document; const analysis: Partial = {}; // Meta robots const metaRobots = document.querySelector('meta[name="robots"]'); if (metaRobots) { const content = metaRobots.getAttribute('content')?.toLowerCase() || ''; analysis.metaRobots = content; analysis.noindex = content.includes('noindex'); analysis.nofollow = content.includes('nofollow'); } // Canonical URL const canonicalLink = document.querySelector('link[rel="canonical"]'); if (canonicalLink) { analysis.canonicalUrl = canonicalLink.getAttribute('href') || undefined; } // Title const titleElement = document.querySelector('title'); if (titleElement) { analysis.title = titleElement.textContent || undefined; } // Meta description const metaDescription = document.querySelector('meta[name="description"]'); if (metaDescription) { analysis.description = metaDescription.getAttribute('content') || undefined; } // Open Graph data const ogTags = document.querySelectorAll('meta[property^="og:"]'); if (ogTags.length > 0) { analysis.openGraphData = {}; ogTags.forEach((tag) => { const property = tag.getAttribute('property'); const content = tag.getAttribute('content'); if (property && content) { analysis.openGraphData![property] = content; } }); } return analysis; } catch (error) { console.error('HTML analysis error:', error); return {}; } } async saveSeoFlags(checkId: string, analysis: SeoAnalysis): Promise { await prisma.seoFlags.create({ data: { checkId, robotsTxtStatus: analysis.robotsTxtStatus, robotsTxtRulesJson: analysis.robotsTxtRules || {}, metaRobots: analysis.metaRobots, canonicalUrl: analysis.canonicalUrl, sitemapPresent: analysis.sitemapPresent, noindex: analysis.noindex, nofollow: analysis.nofollow, }, }); } } ``` #### 3. Security Analysis Service **`apps/api/src/services/security.service.ts`** ```typescript import axios from 'axios'; import { JSDOM } from 'jsdom'; import { z } from 'zod'; import { prisma } from '@/lib/prisma'; import { MixedContent } from '@prisma/client'; export interface SecurityAnalysis { safeBrowsingStatus?: string; mixedContent: MixedContent; httpsToHttp: boolean; insecureResources: string[]; securityHeaders: Record; } export class SecurityService { async analyzeSecurity( redirectChain: Array<{ url: string; statusCode?: number }>, finalHtmlContent?: string ): Promise { const analysis: SecurityAnalysis = { mixedContent: MixedContent.NONE, httpsToHttp: false, insecureResources: [], securityHeaders: {}, }; try { // Check for HTTPS to HTTP downgrades analysis.httpsToHttp = this.detectHttpsDowngrade(redirectChain); // Analyze mixed content if we have HTML if (finalHtmlContent) { const mixedContentAnalysis = this.analyzeMixedContent(finalHtmlContent); analysis.mixedContent = mixedContentAnalysis.mixedContent; analysis.insecureResources = mixedContentAnalysis.insecureResources; } // Check final URL mixed content status if (analysis.httpsToHttp) { analysis.mixedContent = MixedContent.FINAL_TO_HTTP; } else if (analysis.insecureResources.length > 0) { analysis.mixedContent = MixedContent.PRESENT; } // Optional: Google Safe Browsing API check (requires API key) if (process.env.GOOGLE_SAFE_BROWSING_API_KEY) { analysis.safeBrowsingStatus = await this.checkSafeBrowsing( redirectChain[redirectChain.length - 1]?.url ); } return analysis; } catch (error) { console.error('Security analysis error:', error); return analysis; } } private detectHttpsDowngrade(redirectChain: Array<{ url: string }>): boolean { for (let i = 1; i < redirectChain.length; i++) { const prevUrl = redirectChain[i - 1].url; const currUrl = redirectChain[i].url; if (prevUrl.startsWith('https://') && currUrl.startsWith('http://')) { return true; } } return false; } private analyzeMixedContent(htmlContent: string): { mixedContent: MixedContent; insecureResources: string[]; } { try { const dom = new JSDOM(htmlContent); const document = dom.window.document; const insecureResources: string[] = []; // Check for insecure resources const resourceSelectors = [ 'img[src^="http://"]', 'script[src^="http://"]', 'link[href^="http://"]', 'iframe[src^="http://"]', 'embed[src^="http://"]', 'object[data^="http://"]', ]; resourceSelectors.forEach(selector => { const elements = document.querySelectorAll(selector); elements.forEach(element => { const url = element.getAttribute('src') || element.getAttribute('href') || element.getAttribute('data'); if (url && url.startsWith('http://')) { insecureResources.push(url); } }); }); return { mixedContent: insecureResources.length > 0 ? MixedContent.PRESENT : MixedContent.NONE, insecureResources, }; } catch (error) { console.error('Mixed content analysis error:', error); return { mixedContent: MixedContent.NONE, insecureResources: [], }; } } private async checkSafeBrowsing(url: string): Promise { try { const apiKey = process.env.GOOGLE_SAFE_BROWSING_API_KEY; if (!apiKey) return 'not_checked'; const response = await axios.post( `https://safebrowsing.googleapis.com/v4/threatMatches:find?key=${apiKey}`, { client: { clientId: 'redirect-intelligence', clientVersion: '1.0', }, threatInfo: { threatTypes: ['MALWARE', 'SOCIAL_ENGINEERING', 'UNWANTED_SOFTWARE'], platformTypes: ['ANY_PLATFORM'], threatEntryTypes: ['URL'], threatEntries: [{ url }], }, }, { timeout: 5000 } ); return response.data.matches ? 'unsafe' : 'safe'; } catch (error) { console.error('Safe browsing check error:', error); return 'error'; } } async saveSecurityFlags(checkId: string, analysis: SecurityAnalysis): Promise { await prisma.securityFlags.create({ data: { checkId, safeBrowsingStatus: analysis.safeBrowsingStatus, mixedContent: analysis.mixedContent, httpsToHttp: analysis.httpsToHttp, }, }); } } ``` #### 4. Enhanced Redirect Service **`apps/api/src/services/redirect.service.ts`** (updated) ```typescript // ... existing imports import { SslService } from './ssl.service'; import { SeoService } from './seo.service'; import { SecurityService } from './security.service'; export class RedirectService { private sslService = new SslService(); private seoService = new SeoService(); private securityService = new SecurityService(); // ... existing methods // Enhanced createCheck method async createCheck( data: z.infer, userId?: string ): Promise { const validatedData = createCheckSchema.parse(data); const startTime = Date.now(); // Create check record const check = await prisma.check.create({ data: { projectId: validatedData.projectId || await this.getDefaultProjectId(userId), inputUrl: validatedData.inputUrl, method: validatedData.method, headersJson: validatedData.headers, userAgent: validatedData.userAgent, startedAt: new Date(), status: CheckStatus.OK, }, }); try { // Perform redirect analysis const result = await this.analyzeRedirectChain( validatedData.inputUrl, validatedData.method, validatedData.userAgent, validatedData.headers ); const totalTimeMs = Date.now() - startTime; // Update check with results await prisma.check.update({ where: { id: check.id }, data: { finishedAt: new Date(), finalUrl: result.finalUrl, totalTimeMs, status: result.status, }, }); // Save hops await this.saveHops(check.id, result.hops); // Perform SSL analysis on HTTPS URLs const httpsUrls = result.hops .map(hop => hop.url) .filter(url => url.startsWith('https://')); for (const url of [...new Set(httpsUrls)]) { const sslInfo = await this.sslService.inspectSslCertificate(url); if (sslInfo) { await this.sslService.saveSslInspection(check.id, sslInfo); } } // Perform SEO analysis const finalHop = result.hops[result.hops.length - 1]; let htmlContent: string | undefined; if (finalHop?.contentType?.includes('text/html')) { // Fetch final URL content for SEO analysis try { const response = await axios.get(result.finalUrl, { timeout: 10000, headers: { 'User-Agent': validatedData.userAgent || 'RedirectIntelligence/1.0' } }); htmlContent = response.data; } catch (error) { console.warn('Failed to fetch final URL content for SEO analysis:', error); } } const seoAnalysis = await this.seoService.analyzeSeo(result.finalUrl, htmlContent); await this.seoService.saveSeoFlags(check.id, seoAnalysis); // Perform security analysis const securityAnalysis = await this.securityService.analyzeSecurity( result.hops.map(hop => ({ url: hop.url, statusCode: hop.statusCode })), htmlContent ); await this.securityService.saveSecurityFlags(check.id, securityAnalysis); return { checkId: check.id, inputUrl: validatedData.inputUrl, finalUrl: result.finalUrl, totalTimeMs, status: result.status, hops: result.hops, metadata: { redirectCount: result.hops.length - 1, method: validatedData.method, userAgent: validatedData.userAgent, }, }; } catch (error) { // Update check with error status await prisma.check.update({ where: { id: check.id }, data: { finishedAt: new Date(), status: CheckStatus.ERROR, totalTimeMs: Date.now() - startTime, }, }); throw error; } } // ... rest of existing methods } ``` ### Implementation Steps 1. Create SSL inspection service with certificate analysis 2. Implement SEO analysis service (robots.txt, meta tags, canonical) 3. Add security analysis service (mixed content, safe browsing) 4. Enhance redirect service to include all analyses 5. Update database schema with new flag tables 6. Add comprehensive tests for all analysis services 7. Update API responses to include new flag data ### Testing Requirements - Unit tests for SSL, SEO, and Security services - Integration tests with real URLs - Mock tests for external API calls (Safe Browsing) - Performance tests for analysis pipeline - Edge case testing (invalid certificates, missing robots.txt, etc.) ### Commit Message ``` feat(phase-3): add comprehensive SSL/SEO/Security analysis - Implement SSL certificate inspection with expiry warnings - Add SEO analysis: robots.txt parsing, meta tags, canonical URLs - Include security analysis: mixed content detection, HTTPS downgrades - Optional Google Safe Browsing API integration - Enhance redirect service with comprehensive metadata collection - Add database storage for all analysis flags - Include insecure resource detection in HTML content - Support certificate chain validation and protocol analysis ``` --- ## Phases 4-13 Summary Due to length constraints, here's a summary of the remaining phases: ### Phase 4: Chakra UI Upgrade Complete - Migrate from current frontend to React + Chakra UI - Implement app shell with sidebar navigation - Create responsive check detail pages with Mermaid diagrams - Add dark/light mode toggle with Chakra theming ### Phase 5: Exports (MD & PDF) - Implement Markdown report generation with templates - Add PDF export using Puppeteer with embedded Mermaid - Create export endpoints and file management ### Phase 6: Bulk CSV + Worker - Implement BullMQ worker for bulk URL processing - Add CSV upload and batch job management - Create job progress tracking and result downloads ### Phase 7: Request Options UI - Enhanced frontend for custom headers and options - Advanced user agent selection - Request method configuration ### Phase 8: API Keys + Public API + Quotas - Implement API key authentication - Add organization-based rate limiting - Create public API endpoints with key authentication ### Phase 9: Optional JS Redirects - Integrate Playwright for JavaScript redirect detection - Add browser automation for SPA redirects - Optional JS analysis with timeout controls ### Phase 10: Monitoring & Alerts - Add uptime monitoring for URLs - Implement alert system for status changes - Create monitoring dashboards ### Phase 11: Admin Panel - User and organization management - System configuration and monitoring - Audit log viewing and analytics ### Phase 12: Billing (Stripe) - Implement Stripe integration - Add subscription plans and usage limits - Billing dashboard and payment management ### Phase 13: Hardening & Perf - Security audit and penetration testing - Performance optimization and caching - Production deployment and monitoring setup Each phase would follow the same detailed structure as shown in phases 0-3, with specific file implementations, testing requirements, and commit messages. ### Next Steps 1. Start with Phase 0 to establish the foundation 2. Follow phases sequentially to ensure stability 3. Maintain backward compatibility throughout 4. Implement comprehensive testing at each phase 5. Use feature flags for gradual rollout This implementation plan ensures a systematic upgrade while preserving all existing functionality and providing a clear path to the enhanced v2 system.