<![CDATA[ freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More ]]> https://www.freecodecamp.org/news/ https://cdn.freecodecamp.org/universal/favicons/favicon.png <![CDATA[ freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More ]]> https://www.freecodecamp.org/news/ Eleventy Fri, 12 Jun 2026 23:04:04 +0000 60 <![CDATA[ How to Build a Case Converter Tool Using HTML, CSS, and JavaScript ]]> https://www.freecodecamp.org/news/how-to-build-a-case-converter-tool/ 6a2bba6e86b91d1d78662a12 Fri, 12 Jun 2026 07:51:10 +0000 If you're looking to level up your front-end development skills by building a practical web utility, this is the guide for you.

We'll code a fully functional Case Converter Tool from scratch using only HTML, CSS, and vanilla JavaScript.

This lightweight application allows users to paste their content and immediately transform it into standard formats like UPPERCASE, lowercase, Title Case, and Sentence case.

Alongside the text formatting, we'll integrate a live character counter and set up functionality to export the final text as a PDF or Word document.

Grab your favorite code editor, and let's dive in.

Prerequisites

Before you begin, you should have a basic familiarity with the following tools and concepts:

  • Core Web Technologies: A fundamental understanding of HTML structure, basic CSS styling, and JavaScript concepts like functions, array methods, and string manipulation.

  • Development Environment: A code editor installed on your computer (for example, Visual Studio Code) and a modern web browser to test your application locally.

Table of Contents

Step 1: Set Up Your Project

Before writing any code, you need to establish a clean directory structure for your application files.

First, you'll need to initialize a workspace. Open your file manager and create a brand new directory to keep your work organized. Let's name this directory case-converter-app.

Then you'll generate the required files. Inside your newly created directory, set up the following three blank files:

  • index.html

  • styles.css

  • script.js

Step 2: Build the HTML Structure

Open the index.html file in your code editor. You'll add the structural foundation of the tool here.

Add the following code into your index.html file:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Case Converter Tool</title>
    <link rel="stylesheet" href="styles.css">
    
    <!-- jsPDF library for generating PDF files -->
    <script src="https://cdnjs.cloudflare.com/ajax/libs/jspdf/2.5.1/jspdf.umd.min.js"></script>
    <!-- Google Fonts for a modern look -->
    <link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap" rel="stylesheet">
</head>
<body>

    <div class="app-container">
        
        <div class="editor-section">
            <div class="textarea-header">
                <span class="tip-badge">πŸ’‘ Tip: Use Download buttons to save results</span>
            </div>
            <textarea id="inputText" placeholder="Type or paste your content here..."></textarea>
        </div>
        
        <!-- Case Conversion Buttons -->
        <div class="button-grid case-buttons">
            <button class="case-btn" onclick="convertCase(event, 'upper')">UPPER CASE</button>
            <button class="case-btn" onclick="convertCase(event, 'lower')">lower case</button>
            <button class="case-btn" onclick="convertCase(event, 'capitalized')">Capitalized Case</button>
            <button class="case-btn" onclick="convertCase(event, 'title')">Title Case</button>
            <button class="case-btn" onclick="convertCase(event, 'sentence')">Sentence case</button>
            <button class="case-btn" onclick="convertCase(event, 'inverse')">iNvErSe CaSe</button>
            <button class="case-btn" onclick="convertCase(event, 'alternate')">aLtErNaTiNg cAsE</button>
        </div>

        <div class="divider"></div>

        <!-- Action Buttons -->
        <div class="button-grid action-buttons">
            <button class="action-btn primary-action copy-btn" onclick="copyToClipboard()">Copy To Clipboard</button>
            <button class="action-btn" onclick="downloadPDF()">Download PDF</button>
            <button class="action-btn" onclick="downloadWord()">Download Word</button>
            <button class="action-btn danger-action" onclick="clearText()">Clear Text</button>
        </div>

        <!-- Real-time Statistics -->
        <div class="stats-panel">
            <div class="stat-box">
                <span class="stat-value" id="charCount">0</span>
                <span class="stat-label">Characters</span>
            </div>
            <div class="stat-box">
                <span class="stat-value" id="wordCount">0</span>
                <span class="stat-label">Words</span>
            </div>
            <div class="stat-box">
                <span class="stat-value" id="paragraphCount">0</span>
                <span class="stat-label">Paragraphs</span>
            </div>
            <div class="stat-box">
                <span class="stat-value" id="sentenceCount">0</span>
                <span class="stat-label">Sentences</span>
            </div>
        </div>

    </div>

    <script src="script.js"></script>
</body>
</html>

Understanding this HTML:

  • <script src="...jspdf..."></script>: This links to an external library that allows JavaScript to generate PDF files directly in the user's browser.

  • <textarea id="inputText">: This creates the main text box where users will paste their content.

  • <div class="stats-panel">: This section contains span elements with unique IDs. You'll target these IDs with JavaScript to update the text statistics in real-time.

Step 3: Style the Tool with CSS

Next, you'll give the tool a clean, professional design. Open your styles.css file and add the following code:

* {
    margin: 0;
    padding: 0;
    box-sizing: border-box;
    font-family: 'Inter', sans-serif;
}

body {
    background: linear-gradient(135deg, #e0eafc 0%, #cfdef3 100%);
    min-height: 100vh;
    display: flex;
    justify-content: center;
    align-items: center;
    padding: 2rem;
    color: #1e293b;
}

.app-container {
    background: #ffffff;
    width: 100%;
    max-width: 900px;
    border-radius: 24px;
    box-shadow: 0 20px 40px rgba(0,0,0,0.08);
    padding: 2.5rem;
}

.textarea-header {
    display: flex;
    justify-content: flex-end;
    margin-bottom: 0.5rem;
}

.tip-badge {
    background: #fef08a;
    color: #854d0e;
    padding: 0.35rem 0.85rem;
    border-radius: 20px;
    font-size: 0.75rem;
    font-weight: 600;
}

textarea {
    width: 100%;
    height: 220px;
    padding: 1.5rem;
    border: 2px solid #e2e8f0;
    border-radius: 16px;
    font-size: 1rem;
    resize: vertical;
    outline: none;
    transition: all 0.3s ease;
    background: #f8fafc;
}

textarea:focus {
    border-color: #007bff;
    background: #fff;
    box-shadow: 0 0 0 4px rgba(0, 123, 255, 0.1);
}

.button-grid {
    display: flex;
    flex-wrap: wrap;
    gap: 0.75rem;
    margin-top: 1.5rem;
}

button {
    padding: 0.75rem 1.25rem;
    border: none;
    border-radius: 12px;
    font-size: 0.875rem;
    font-weight: 600;
    cursor: pointer;
    transition: all 0.2s ease;
}

.case-btn {
    background: #f1f5f9;
    color: #475569;
    border: 1px solid #e2e8f0;
}

.case-btn:hover { 
    background: #e2e8f0; 
}

/* The active class highlights the selected button */
.case-btn.active {
    background: #007bff;
    color: #fff;
    border-color: #007bff;
    box-shadow: 0 4px 12px rgba(0, 123, 255, 0.25);
}

.divider {
    height: 1px;
    background: #e2e8f0;
    margin: 1.5rem 0;
}

.action-btn { 
    background: #fff; 
    border: 1px solid #cbd5e1; 
}

.action-btn:hover { 
    background: #f8fafc; 
    border-color: #94a3b8; 
}

.primary-action { 
    background: #007bff; 
    color: #fff; 
    border-color: #007bff; 
}

.primary-action:hover { 
    background: #0056b3; 
    border-color: #0056b3; 
}

.danger-action { 
    color: #ef4444; 
    border-color: #fca5a5; 
    background: #fef2f2; 
}

.danger-action:hover { 
    background: #fee2e2; 
    border-color: #f87171; 
}

.stats-panel {
    display: grid;
    grid-template-columns: repeat(auto-fit, minmax(130px, 1fr));
    gap: 1rem;
    margin-top: 2rem;
    background: #f8fafc;
    padding: 1.5rem;
    border-radius: 16px;
    border: 1px solid #e2e8f0;
}

.stat-box { 
    display: flex; 
    flex-direction: column; 
    align-items: center; 
}

.stat-value { 
    font-size: 1.75rem; 
    font-weight: 700; 
}

.stat-label { 
    font-size: 0.75rem; 
    color: #64748b; 
    text-transform: uppercase; 
}

Understanding this CSS:

  • body: You use Flexbox to center the tool perfectly on the screen and apply a soft gradient background.

  • .app-container: This creates a white, rounded card with a soft shadow to hold the user interface.

  • .case-btn.active: You define an active state here. You'll use JavaScript to apply this class to the specific button the user clicks.

At this stage, we've completely structured and styled the user interface. The tool will look like this:

Case Converter Tool screenshot

Right now, the front-end is visible, but the buttons are entirely static. To make the transformations actually work, we have to write the logic in JavaScript.

Step 4: Add JavaScript Functionality

Now you need to make the tool interactive. Open the script.js file and add this code:

const textArea = document.getElementById('inputText');

// Listen for typing to update statistics in real-time
textArea.addEventListener('input', updateStats);

function updateStats() {
    const text = textArea.value;
    
    document.getElementById('charCount').textContent = text.length;
    
    const words = text.trim().split(/\s+/).filter(word => word.length > 0);
    document.getElementById('wordCount').textContent = words.length;
    
    const sentences = text.split(/[.!?]+/).filter(sentence => sentence.trim().length > 0);
    document.getElementById('sentenceCount').textContent = sentences.length;
    
    const paragraphs = text.split(/\n+/).filter(paragraph => paragraph.trim().length > 0);
    document.getElementById('paragraphCount').textContent = paragraphs.length;
}

function convertCase(event, type) {
    let text = textArea.value;
    if (!text) return; 

    // Highlight the active button
    const buttons = document.querySelectorAll('.case-btn');
    buttons.forEach(btn => btn.classList.remove('active'));
    if (event) {
        event.target.classList.add('active');
    }

    // Process the text
    switch (type) {
        case 'upper':
            text = text.toUpperCase();
            break;
        case 'lower':
            text = text.toLowerCase();
            break;
        case 'capitalized':
            text = text.toLowerCase().replace(/\b\w/g, c => c.toUpperCase());
            break;
        case 'title':
            const minorWords = ['a', 'an', 'the', 'and', 'but', 'or', 'for', 'nor', 'on', 'at', 'to', 'from', 'by'];
            text = text.toLowerCase().split(' ').map((word, index) => {
                if (index !== 0 && minorWords.includes(word)) return word;
                return word.charAt(0).toUpperCase() + word.slice(1);
            }).join(' ');
            break;
        case 'sentence':
            text = text.toLowerCase().replace(/(^\s*\w|[\.\!\?]\n*\s*\w)/g, c => c.toUpperCase());
            break;
        case 'inverse':
            text = text.split('').map(c => c === c.toUpperCase() ? c.toLowerCase() : c.toUpperCase()).join('');
            break;
        case 'alternate':
            text = text.toLowerCase().split('').map((c, i) => i % 2 === 0 ? c : c.toUpperCase()).join('');
            break;
    }

    textArea.value = text;
    updateStats(); 
}

function copyToClipboard() {
    if (!textArea.value) return;
    textArea.select();
    document.execCommand('copy');
    
    const copyBtn = document.querySelector('.copy-btn');
    copyBtn.textContent = 'Copied!';
    setTimeout(() => copyBtn.textContent = 'Copy To Clipboard', 1500);
}

function clearText() {
    textArea.value = '';
    updateStats();
    document.querySelectorAll('.case-btn').forEach(btn => btn.classList.remove('active'));
}

function downloadWord() {
    if (!textArea.value) return;
    const blob = new Blob([textArea.value], { type: 'application/msword' });
    const url = URL.createObjectURL(blob);
    const a = document.createElement('a');
    a.href = url;
    a.download = 'converted_text.doc';
    a.click();
    URL.revokeObjectURL(url);
}

function downloadPDF() {
    if (!textArea.value) return;
    const { jsPDF } = window.jspdf;
    const doc = new jsPDF();
    const splitText = doc.splitTextToSize(textArea.value, 180);
    doc.text(splitText, 15, 15);
    doc.save('converted_text.pdf');
}

Understanding this JavaScript:

  • addEventListener('input', ...): This listens to every single keystroke. Every time you type, it instantly recalculates the words, characters, and sentences.

  • convertCase(event, type): This function takes the selected style (like upper or sentence) and applies Regular Expressions (Regex) or array mapping to format the string. It also dynamically adds the .active CSS class to the specific button you clicked.

  • document.execCommand('copy'): This is a browser command that copies the selected text directly to the user's clipboard.

  • new Blob(): You use a Blob (Binary Large Object) to construct a file out of the text on the fly. This allows users to download a .doc file without needing a backend server.

Step 5: Test Your Tool

You're now ready to evaluate your code in a real browser environment.

  1. Open the case-converter-app folder on your computer.

  2. Double-click the index.html file to launch the application.

  3. Paste a long paragraph into the text area to verify that the live statistics update accurately.

  4. Switch between the formatting options to observe the immediate DOM manipulation, and test the export buttons to ensure files are downloading correctly.

Conclusion

In this tutorial, you successfully engineered a browser-based Case Converter Tool using vanilla JavaScript.

You learned how to handle continuous user inputs, manipulate string data using Regular Expressions, and trigger local file downloads directly from the front end.

Most importantly, you learned that modern web browsers are highly capable of handling complex document modifications locally, removing the strict need for external backend servers. This method guarantees fast processing speeds and keeps user data completely private.

For a live demonstration of these concepts in a production environment, feel free to test out this Case Converter and experience how seamlessly these text transformations operate.

]]> <![CDATA[ From Flutter to Backend: How to Build Production-Grade REST APIs with Dart and Dart Frog ]]> https://www.freecodecamp.org/news/how-to-build-production-grade-rest-apis-with-dart-and-dart-frog/ 6a2b553bb84c3c44ce471560 Fri, 12 Jun 2026 00:39:23 +0000 Dart backend frameworks exist on a spectrum. At the minimal end sits Shelf, with raw primitives and full control. You wire everything yourself. At the maximal end sits Serverpod. It's a full framework with code generation and opinionated conventions. The framework makes most structural decisions for you.

Dart Frog lives in the middle, and for many Flutter engineers, it's the most natural fit.

Dart Frog is a fast, minimalistic backend framework built on top of Shelf, originally created by Very Good Ventures and now maintained independently. It takes the file-based routing model popularized by Next.js and Remix, applies it to Dart, and wraps it with a clean CLI that handles development server, hot reload, production builds, and Docker generation, all out of the box.

You write a Dart file in the routes/ directory, export an onRequest function, and Dart Frog handles the routing automatically. No router configuration, no handler registration, no mounting. The file system is the router.

In this article, we'll build a User and Profile Management REST API (the same one we built in the linked articles above) using Dart Frog, connect it to PostgreSQL, add JWT authentication, and deploy it to Fly.io.

By the end you'll understand Dart Frog's routing model deeply, and you'll have a clear picture of where it fits compared to Shelf and Serverpod.

Table of Contents

Prerequisites

Before starting, you should have:

  • Comfortable familiarity with Dart and Flutter development

  • Understanding of REST API concepts, endpoints, HTTP methods, status codes

  • Docker Desktop installed and running

  • A Fly.io account for deployment

How Dart Frog Differs from Shelf and Serverpod

Understanding where Dart Frog sits in relation to the other two frameworks helps you make the right choice for each project.

Shelf gives you a Router and you mount handlers manually. Your folder structure has nothing to do with your URL structure. You decide what goes where.

Serverpod generates your routes from endpoint class names and method names. You define a class, run a generator, and the URL is derived automatically.

Dart Frog maps your file system directly to your URL structure. A file at routes/users/index.dart becomes the /users endpoint. A file at routes/users/[id].dart becomes /users/:id. No configuration, no registration, no generation step. The file is the route.

This model will feel immediately intuitive to Flutter engineers who have worked with Next.js or any modern web framework. It's also significantly easier to navigate in a team. You look at the folder structure and you instantly know what endpoints exist.

The other key difference is the RequestContext. Where Shelf passes a raw Request to handlers, Dart Frog wraps it in a RequestContext that carries both the request and any values injected by middleware. This is Dart Frog's dependency injection mechanism, and it's elegant.

Installing Dart Frog

Install the Dart Frog CLI:

dart pub global activate dart_frog_cli

Verify the installation:

dart_frog --version

Creating the Project

dart_frog create user_profile_api
cd user_profile_api

Start the development server with hot reload:

dart_frog dev

Visit http://localhost:8080 and you'll see the default welcome response. The dev server watches for file changes and reloads automatically. No restart needed as you build.

Understanding the Project Structure

user_profile_api/
  routes/
    index.dart              ← GET /
  pubspec.yaml
  analysis_options.yaml

That's the entire starting structure. Clean and minimal. Everything we add will extend from here.

After building our API, the full structure will look like this:

user_profile_api/
  routes/
    _middleware.dart         ← global middleware pipeline
    index.dart               ← GET /
    auth/
      login.dart             ← POST /auth/login
      register.dart          ← POST /auth/register
    users/
      index.dart             ← GET /users
      [id].dart              ← GET, PUT, DELETE /users/:id
      [id]/
        profile.dart         ← GET, POST, PUT /users/:id/profile
  lib/
    config/
      database.dart
      env.dart
    models/
      user.dart
      profile.dart
    repositories/
      user_repository.dart
      profile_repository.dart
    services/
      auth_service.dart
    middleware/
      auth_middleware.dart
      error_middleware.dart
  pubspec.yaml

The routes/ folder is the heart of a Dart Frog project. The lib/ folder holds all shared logic that routes import. This separation is clean and deliberate: routing concerns live in routes/, while business logic lives in lib/.

Dart Frog Core Concepts

File-Based Routing

Every .dart file in the routes/ directory is a route. The file path determines the URL path:

File URL
routes/index.dart /
routes/users/index.dart /users
routes/users/[id].dart /users/:id
routes/auth/login.dart /auth/login
routes/users/[id]/profile.dart /users/:id/profile

Every route file must export an onRequest function:

import 'package:dart_frog/dart_frog.dart';

Future<Response> onRequest(RequestContext context) async {
  return Response.json(body: {'message': 'Hello from Dart Frog'});
}

That's the entire contract. One function, one file, one route. Dart Frog generates the internal routing glue automatically when you run dart_frog dev or dart_frog build.

The RequestContext

RequestContext is the object passed to every route handler and middleware. It's more than just the HTTP request: it's a container for the request and any values that middleware has injected:

Future<Response> onRequest(RequestContext context) async {
  // The raw HTTP request
  final request = context.request;

  // HTTP method
  print(request.method); // GET, POST, etc.

  // Path parameters (for dynamic routes like [id].dart)
  final id = context.request.uri.pathSegments.last;

  // Query parameters
  final page = request.uri.queryParameters['page'];

  // Request body
  final body = await request.json() as Map<String, dynamic>;

  // Values injected by middleware
  final db = context.read<DatabaseConnection>();
  final currentUser = context.read<AuthenticatedUser>();

  return Response.json(body: {'ok': true});
}

context.read() is the dependency injection mechanism. Middleware provides values, and routes consume them. This keeps routes clean and testable: a route handler doesn't know how a database connection was created, it just reads it from context.

Middleware and Dependency Injection

A _middleware.dart file in any route folder applies middleware to all routes in that folder and its subfolders. A _middleware.dart at the root routes/ level applies globally.

Middleware in Dart Frog uses the provider pattern to inject values into the context:

import 'package:dart_frog/dart_frog.dart';

Handler middleware(Handler handler) {
  return handler.use(
    provider<DatabaseConnection>(
      (context) => DatabaseConnection.instance,
    ),
  );
}

Any route in the same folder, or any subfolder, can then call context.read() to get the connection. No global singletons, no manual passing. The context carries it.

Middleware functions can also intercept requests before they reach the route handler, making them perfect for authentication:

Handler middleware(Handler handler) {
  return (context) async {
    final authHeader = context.request.headers['authorization'];

    if (authHeader == null) {
      return Response.json(
        statusCode: 401,
        body: {'error': 'Authorization required'},
      );
    }

    // Verify token and inject user
    final user = verifyToken(authHeader);
    return handler(context.provide<AuthenticatedUser>(() => user));
  };
}

Dynamic Routes

A file named [id].dart matches any single path segment. Inside the handler, extract the parameter from the URL:

Future<Response> onRequest(RequestContext context, String id) async {
  // id is automatically passed as a parameter for dynamic routes
  return Response.json(body: {'userId': id});
}

Dart Frog passes dynamic route parameters as additional arguments to onRequest. This is cleaner than parsing them manually from the URL.

Setting Up the Database

Docker Compose for PostgreSQL

Create docker-compose.yml in the project root:

version: '3.8'

services:
  postgres:
    image: postgres:16-alpine
    container_name: user_profile_db
    environment:
      POSTGRES_DB: user_profile_api
      POSTGRES_USER: dart_user
      POSTGRES_PASSWORD: dart_password
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U dart_user -d user_profile_api"]
      interval: 5s
      timeout: 5s
      retries: 5

volumes:
  postgres_data:

Start the database:

docker compose up -d

Environment Configuration

Add dependencies to pubspec.yaml:

dependencies:
  dart_frog: ^1.4.0
  dart_frog_auth: ^0.1.0
  postgres: ^3.3.0
  dart_jsonwebtoken: ^2.12.0
  bcrypt: ^1.1.3
  dotenv: ^4.1.0

dev_dependencies:
  dart_frog_cli: ^1.2.0
  test: ^1.24.0
  dart_frog_test: ^0.1.0

Run dart pub get.

Create .env:

DB_HOST=localhost
DB_PORT=5432
DB_NAME=user_profile_api
DB_USER=dart_user
DB_PASSWORD=dart_password
JWT_SECRET=your_super_secret_key_change_this_in_production
JWT_EXPIRY_HOURS=24
PORT=8080

Create lib/config/env.dart:

import 'package:dotenv/dotenv.dart';

class Env {
  static late final DotEnv _env;

  static void load() {
    _env = DotEnv(includePlatformEnvironment: true)..load();
  }

  static String get dbHost => _env['DB_HOST'] ?? 'localhost';
  static int get dbPort => int.parse(_env['DB_PORT'] ?? '5432');
  static String get dbName => _env['DB_NAME'] ?? 'user_profile_api';
  static String get dbUser => _env['DB_USER'] ?? 'dart_user';
  static String get dbPassword => _env['DB_PASSWORD'] ?? '';
  static String get jwtSecret => _env['JWT_SECRET'] ?? '';
  static int get jwtExpiryHours =>
      int.parse(_env['JWT_EXPIRY_HOURS'] ?? '24');
}

Database Connection Manager

Create lib/config/database.dart:

import 'package:postgres/postgres.dart';
import 'env.dart';

class Database {
  static Connection? _connection;

  static Future<Connection> get connection async {
    if (_connection != null) return _connection!;
    _connection = await Connection.open(
      Endpoint(
        host: Env.dbHost,
        port: Env.dbPort,
        database: Env.dbName,
        username: Env.dbUser,
        password: Env.dbPassword,
      ),
      settings: const ConnectionSettings(sslMode: SslMode.disable),
    );
    print('Database connected');
    return _connection!;
  }

  static Future<void> runMigrations() async {
    final conn = await connection;
    await conn.execute('''
      CREATE TABLE IF NOT EXISTS users (
        id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
        email VARCHAR(255) UNIQUE NOT NULL,
        password_hash VARCHAR(255) NOT NULL,
        first_name VARCHAR(100) NOT NULL,
        last_name VARCHAR(100) NOT NULL,
        is_active BOOLEAN DEFAULT TRUE,
        created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
        updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
      );

      CREATE INDEX IF NOT EXISTS idx_users_email ON users(email);

      CREATE TABLE IF NOT EXISTS profiles (
        id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
        user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
        bio TEXT,
        avatar_url VARCHAR(500),
        phone VARCHAR(20),
        location VARCHAR(255),
        website VARCHAR(500),
        created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
        updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
        UNIQUE(user_id)
      );

      CREATE INDEX IF NOT EXISTS idx_profiles_user_id ON profiles(user_id);
    ''');
    print('Migrations applied');
  }
}

Migrations

Dart Frog projects have a main.dart entry point generated during dart_frog build. For the development server, migrations are best run from the project entrypoint. Create main.dart in the project root:

import 'dart:io';
import 'package:dart_frog/dart_frog.dart';
import 'lib/config/database.dart';
import 'lib/config/env.dart';

Future<HttpServer> run(Handler handler, InternetAddress ip, int port) async {
  Env.load();
  await Database.runMigrations();
  return serve(handler, ip, port);
}

This run function is Dart Frog's server lifecycle hook. It runs before the server starts accepting requests, giving us the right place to load environment variables and run migrations.

Defining the Models

With the database layer in place, we need Dart classes to represent the data coming in and out of it.

The User model maps to the users table and handles conversion between database rows and Dart objects. The Profile model does the same for the profiles table. Both models follow the same pattern: a factory constructor for reading from the database and a toJson method for sending data back to the client.

Note that toJson on the User model deliberately excludes the password hash. You should never return credential data in an API response.

Create lib/models/user.dart:

class User {
  const User({
    required this.id,
    required this.email,
    required this.passwordHash,
    required this.firstName,
    required this.lastName,
    required this.isActive,
    required this.createdAt,
    required this.updatedAt,
  });

  final String id;
  final String email;
  final String passwordHash;
  final String firstName;
  final String lastName;
  final bool isActive;
  final DateTime createdAt;
  final DateTime updatedAt;

  factory User.fromRow(Map<String, dynamic> row) => User(
        id: row['id'] as String,
        email: row['email'] as String,
        passwordHash: row['password_hash'] as String,
        firstName: row['first_name'] as String,
        lastName: row['last_name'] as String,
        isActive: row['is_active'] as bool,
        createdAt: row['created_at'] as DateTime,
        updatedAt: row['updated_at'] as DateTime,
      );

  Map<String, dynamic> toJson() => {
        'id': id,
        'email': email,
        'firstName': firstName,
        'lastName': lastName,
        'isActive': isActive,
        'createdAt': createdAt.toIso8601String(),
        'updatedAt': updatedAt.toIso8601String(),
      };
}

Create lib/models/profile.dart:

class Profile {
  const Profile({
    required this.id,
    required this.userId,
    this.bio,
    this.avatarUrl,
    this.phone,
    this.location,
    this.website,
    required this.createdAt,
    required this.updatedAt,
  });

  final String id;
  final String userId;
  final String? bio;
  final String? avatarUrl;
  final String? phone;
  final String? location;
  final String? website;
  final DateTime createdAt;
  final DateTime updatedAt;

  factory Profile.fromRow(Map<String, dynamic> row) => Profile(
        id: row['id'] as String,
        userId: row['user_id'] as String,
        bio: row['bio'] as String?,
        avatarUrl: row['avatar_url'] as String?,
        phone: row['phone'] as String?,
        location: row['location'] as String?,
        website: row['website'] as String?,
        createdAt: row['created_at'] as DateTime,
        updatedAt: row['updated_at'] as DateTime,
      );

  Map<String, dynamic> toJson() => {
        'id': id,
        'userId': userId,
        'bio': bio,
        'avatarUrl': avatarUrl,
        'phone': phone,
        'location': location,
        'website': website,
        'createdAt': createdAt.toIso8601String(),
        'updatedAt': updatedAt.toIso8601String(),
      };
}

Building the Repositories

Repositories are the single point of contact between the application and the database. Rather than writing SQL directly inside route handlers, we'll centralise all database operations here. This keeps the handlers clean and makes the data access logic easy to find, maintain, and test independently.

The UserRepository handles every operation on the users table. The ProfileRepository does the same for profiles, using userId as its primary lookup key since profiles are always accessed in the context of a specific user.

User Repository

Create lib/repositories/user_repository.dart:

import 'package:postgres/postgres.dart';
import '../config/database.dart';
import '../models/user.dart';

class UserRepository {
  Future<Connection> get _conn => Database.connection;

  Future<List<User>> findAll() async {
    final conn = await _conn;
    final results = await conn.execute(
      'SELECT * FROM users WHERE is_active = TRUE ORDER BY created_at DESC',
    );
    return results.map((r) => User.fromRow(r.toColumnMap())).toList();
  }

  Future<User?> findById(String id) async {
    final conn = await _conn;
    final results = await conn.execute(
      Sql.named('SELECT * FROM users WHERE id = @id AND is_active = TRUE'),
      parameters: {'id': id},
    );
    if (results.isEmpty) return null;
    return User.fromRow(results.first.toColumnMap());
  }

  Future<User?> findByEmail(String email) async {
    final conn = await _conn;
    final results = await conn.execute(
      Sql.named('SELECT * FROM users WHERE email = @email'),
      parameters: {'email': email},
    );
    if (results.isEmpty) return null;
    return User.fromRow(results.first.toColumnMap());
  }

  Future<User> create({
    required String email,
    required String passwordHash,
    required String firstName,
    required String lastName,
  }) async {
    final conn = await _conn;
    final results = await conn.execute(
      Sql.named('''
        INSERT INTO users (email, password_hash, first_name, last_name)
        VALUES (@email, @passwordHash, @firstName, @lastName)
        RETURNING *
      '''),
      parameters: {
        'email': email,
        'passwordHash': passwordHash,
        'firstName': firstName,
        'lastName': lastName,
      },
    );
    return User.fromRow(results.first.toColumnMap());
  }

  Future<User?> update({
    required String id,
    String? firstName,
    String? lastName,
  }) async {
    final conn = await _conn;
    final results = await conn.execute(
      Sql.named('''
        UPDATE users
        SET
          first_name = COALESCE(@firstName, first_name),
          last_name  = COALESCE(@lastName, last_name),
          updated_at = NOW()
        WHERE id = @id AND is_active = TRUE
        RETURNING *
      '''),
      parameters: {'id': id, 'firstName': firstName, 'lastName': lastName},
    );
    if (results.isEmpty) return null;
    return User.fromRow(results.first.toColumnMap());
  }

  Future<bool> delete(String id) async {
    final conn = await _conn;
    final results = await conn.execute(
      Sql.named('''
        UPDATE users SET is_active = FALSE, updated_at = NOW()
        WHERE id = @id AND is_active = TRUE
        RETURNING id
      '''),
      parameters: {'id': id},
    );
    return results.isNotEmpty;
  }
}

Profile Repository

Create lib/repositories/profile_repository.dart:

import 'package:postgres/postgres.dart';
import '../config/database.dart';
import '../models/profile.dart';

class ProfileRepository {
  Future<Connection> get _conn => Database.connection;

  Future<Profile?> findByUserId(String userId) async {
    final conn = await _conn;
    final results = await conn.execute(
      Sql.named('SELECT * FROM profiles WHERE user_id = @userId'),
      parameters: {'userId': userId},
    );
    if (results.isEmpty) return null;
    return Profile.fromRow(results.first.toColumnMap());
  }

  Future<Profile> create({
    required String userId,
    String? bio,
    String? avatarUrl,
    String? phone,
    String? location,
    String? website,
  }) async {
    final conn = await _conn;
    final results = await conn.execute(
      Sql.named('''
        INSERT INTO profiles (user_id, bio, avatar_url, phone, location, website)
        VALUES (@userId, @bio, @avatarUrl, @phone, @location, @website)
        RETURNING *
      '''),
      parameters: {
        'userId': userId,
        'bio': bio,
        'avatarUrl': avatarUrl,
        'phone': phone,
        'location': location,
        'website': website,
      },
    );
    return Profile.fromRow(results.first.toColumnMap());
  }

  Future<Profile?> update({
    required String userId,
    String? bio,
    String? avatarUrl,
    String? phone,
    String? location,
    String? website,
  }) async {
    final conn = await _conn;
    final results = await conn.execute(
      Sql.named('''
        UPDATE profiles
        SET
          bio        = COALESCE(@bio, bio),
          avatar_url = COALESCE(@avatarUrl, avatar_url),
          phone      = COALESCE(@phone, phone),
          location   = COALESCE(@location, location),
          website    = COALESCE(@website, website),
          updated_at = NOW()
        WHERE user_id = @userId
        RETURNING *
      '''),
      parameters: {
        'userId': userId,
        'bio': bio,
        'avatarUrl': avatarUrl,
        'phone': phone,
        'location': location,
        'website': website,
      },
    );
    if (results.isEmpty) return null;
    return Profile.fromRow(results.first.toColumnMap());
  }
}

Authentication Service

Authentication in this project is handled by a dedicated AuthService that lives in lib/services/. It has one clear responsibility: the cryptographic operations that power auth: hashing passwords before storing them, verifying passwords at login, generating signed JWT tokens on success, and verifying those tokens on protected requests.

Keeping this logic in a service rather than spreading it across route handlers means it can be injected via middleware and consumed cleanly anywhere in the app.

Create lib/services/auth_service.dart:

import 'package:bcrypt/bcrypt.dart';
import 'package:dart_jsonwebtoken/dart_jsonwebtoken.dart';
import '../config/env.dart';
import '../models/user.dart';

class AuthService {
  String hashPassword(String password) =>
      BCrypt.hashpw(password, BCrypt.gensalt());

  bool verifyPassword(String password, String hash) =>
      BCrypt.checkpw(password, hash);

  String generateToken(User user) {
    final jwt = JWT({
      'sub': user.id,
      'email': user.email,
      'iat': DateTime.now().millisecondsSinceEpoch ~/ 1000,
    });
    return jwt.sign(
      SecretKey(Env.jwtSecret),
      expiresIn: Duration(hours: Env.jwtExpiryHours),
    );
  }

  JWT? verifyToken(String token) {
    try {
      return JWT.verify(token, SecretKey(Env.jwtSecret));
    } catch (_) {
      return null;
    }
  }
}

Middleware

Middleware is where Dart Frog's dependency injection model does its most important work. Rather than instantiating repositories and services inside each route handler, we create them once in middleware and make them available to every handler downstream via the RequestContext.

This section defines three pieces of middleware: the database middleware that injects the repositories and auth service, the auth middleware that validates JWT tokens and protects routes, and the error middleware that catches unhandled exceptions and returns consistent error responses across the entire API.

Database Middleware

Create lib/middleware/database_middleware.dart:

import 'package:dart_frog/dart_frog.dart';
import '../repositories/user_repository.dart';
import '../repositories/profile_repository.dart';
import '../services/auth_service.dart';

Middleware databaseMiddleware() {
  return (handler) {
    return handler
        .use(provider<UserRepository>((_) => UserRepository()))
        .use(provider<ProfileRepository>((_) => ProfileRepository()))
        .use(provider<AuthService>((_) => AuthService()));
  };
}

This middleware injects the repositories and auth service into every request context. Routes read them with context.read() without caring how they were created.

Auth Middleware

Create lib/middleware/auth_middleware.dart:

import 'dart:convert';
import 'package:dart_frog/dart_frog.dart';
import '../services/auth_service.dart';

Middleware authMiddleware() {
  return (handler) {
    return (context) async {
      final authHeader = context.request.headers['authorization'];

      if (authHeader == null || !authHeader.startsWith('Bearer ')) {
        return Response.json(
          statusCode: 401,
          body: {'error': 'Authorization header missing or malformed'},
        );
      }

      final token = authHeader.substring(7);
      final authService = context.read<AuthService>();
      final jwt = authService.verifyToken(token);

      if (jwt == null) {
        return Response.json(
          statusCode: 401,
          body: {'error': 'Invalid or expired token'},
        );
      }

      final userId = jwt.payload['sub'] as String;
      final userEmail = jwt.payload['email'] as String;

      return handler(
        context.provide<Map<String, String>>(
          () => {'userId': userId, 'userEmail': userEmail},
        ),
      );
    };
  };
}

Error Middleware

Create lib/middleware/error_middleware.dart:

import 'package:dart_frog/dart_frog.dart';

Middleware errorMiddleware() {
  return (handler) {
    return (context) async {
      try {
        return await handler(context);
      } on FormatException catch (e) {
        return Response.json(
          statusCode: 400,
          body: {'error': 'Invalid request body: ${e.message}'},
        );
      } catch (e, stackTrace) {
        print('Unhandled error: \(e\n\)stackTrace');
        return Response.json(
          statusCode: 500,
          body: {'error': 'An internal server error occurred'},
        );
      }
    };
  };
}

Building the Routes

With the models, repositories, auth service, and middleware all in place, we can now build the route handlers.

In Dart Frog, each file in the routes/ folder is a self-contained endpoint. Routes don't manage dependencies directly. Instead, they read what middleware has already injected into the context and call the appropriate repository or service method.

This section covers three groups of routes: the auth routes for registration and login, the user routes for CRUD operations, and the profile routes nested under a user's ID.

Auth Routes

Create routes/auth/register.dart:

import 'package:dart_frog/dart_frog.dart';
import '../../lib/repositories/user_repository.dart';
import '../../lib/services/auth_service.dart';

Future<Response> onRequest(RequestContext context) async {
  if (context.request.method != HttpMethod.post) {
    return Response.json(statusCode: 405, body: {'error': 'Method not allowed'});
  }

  final body = await context.request.json() as Map<String, dynamic>;
  final email = body['email'] as String?;
  final password = body['password'] as String?;
  final firstName = body['firstName'] as String?;
  final lastName = body['lastName'] as String?;

  if (email == null || password == null ||
      firstName == null || lastName == null) {
    return Response.json(
      statusCode: 400,
      body: {'error': 'email, password, firstName, and lastName are required'},
    );
  }

  if (password.length < 8) {
    return Response.json(
      statusCode: 400,
      body: {'error': 'Password must be at least 8 characters'},
    );
  }

  final userRepo = context.read<UserRepository>();
  final authService = context.read<AuthService>();

  final existing = await userRepo.findByEmail(email);
  if (existing != null) {
    return Response.json(
      statusCode: 409,
      body: {'error': 'An account with this email already exists'},
    );
  }

  final user = await userRepo.create(
    email: email,
    passwordHash: authService.hashPassword(password),
    firstName: firstName,
    lastName: lastName,
  );

  return Response.json(
    statusCode: 201,
    body: {
      'user': user.toJson(),
      'token': authService.generateToken(user),
    },
  );
}

Create routes/auth/login.dart:

import 'package:dart_frog/dart_frog.dart';
import '../../lib/repositories/user_repository.dart';
import '../../lib/services/auth_service.dart';

Future<Response> onRequest(RequestContext context) async {
  if (context.request.method != HttpMethod.post) {
    return Response.json(statusCode: 405, body: {'error': 'Method not allowed'});
  }

  final body = await context.request.json() as Map<String, dynamic>;
  final email = body['email'] as String?;
  final password = body['password'] as String?;

  if (email == null || password == null) {
    return Response.json(
      statusCode: 400,
      body: {'error': 'email and password are required'},
    );
  }

  final userRepo = context.read<UserRepository>();
  final authService = context.read<AuthService>();
  final user = await userRepo.findByEmail(email);

  if (user == null || !authService.verifyPassword(password, user.passwordHash)) {
    return Response.json(
      statusCode: 401,
      body: {'error': 'Invalid email or password'},
    );
  }

  return Response.json(
    body: {
      'user': user.toJson(),
      'token': authService.generateToken(user),
    },
  );
}

User Routes

Create routes/users/index.dart:

import 'package:dart_frog/dart_frog.dart';
import '../../lib/repositories/user_repository.dart';

Future<Response> onRequest(RequestContext context) async {
  if (context.request.method != HttpMethod.get) {
    return Response.json(statusCode: 405, body: {'error': 'Method not allowed'});
  }

  final userRepo = context.read<UserRepository>();
  final users = await userRepo.findAll();

  return Response.json(
    body: users.map((u) => u.toJson()).toList(),
  );
}

Create routes/users/[id].dart:

import 'package:dart_frog/dart_frog.dart';
import '../../lib/repositories/user_repository.dart';

Future<Response> onRequest(RequestContext context, String id) async {
  final userRepo = context.read<UserRepository>();

  switch (context.request.method) {
    case HttpMethod.get:
      return _getUser(userRepo, id);
    case HttpMethod.put:
      return _updateUser(context, userRepo, id);
    case HttpMethod.delete:
      return _deleteUser(userRepo, id);
    default:
      return Response.json(
        statusCode: 405,
        body: {'error': 'Method not allowed'},
      );
  }
}

Future<Response> _getUser(UserRepository repo, String id) async {
  final user = await repo.findById(id);
  if (user == null) {
    return Response.json(statusCode: 404, body: {'error': 'User not found'});
  }
  return Response.json(body: user.toJson());
}

Future<Response> _updateUser(
  RequestContext context,
  UserRepository repo,
  String id,
) async {
  final body = await context.request.json() as Map<String, dynamic>;
  final user = await repo.update(
    id: id,
    firstName: body['firstName'] as String?,
    lastName: body['lastName'] as String?,
  );
  if (user == null) {
    return Response.json(statusCode: 404, body: {'error': 'User not found'});
  }
  return Response.json(body: user.toJson());
}

Future<Response> _deleteUser(UserRepository repo, String id) async {
  final deleted = await repo.delete(id);
  if (!deleted) {
    return Response.json(statusCode: 404, body: {'error': 'User not found'});
  }
  return Response.json(statusCode: 204, body: null);
}

Notice how onRequest receives String id as a second parameter, Dart Frog automatically passes the dynamic path segment to the handler. The switch on context.request.method handles all HTTP methods in a single file which is the idiomatic Dart Frog pattern for CRUD endpoints.

Profile Routes

Create routes/users/[id]/profile.dart:

import 'package:dart_frog/dart_frog.dart';
import '../../../lib/repositories/user_repository.dart';
import '../../../lib/repositories/profile_repository.dart';

Future<Response> onRequest(RequestContext context, String id) async {
  final userRepo = context.read<UserRepository>();
  final profileRepo = context.read<ProfileRepository>();

  final user = await userRepo.findById(id);
  if (user == null) {
    return Response.json(statusCode: 404, body: {'error': 'User not found'});
  }

  switch (context.request.method) {
    case HttpMethod.get:
      return _getProfile(profileRepo, id);
    case HttpMethod.post:
      return _createProfile(context, profileRepo, id);
    case HttpMethod.put:
      return _updateProfile(context, profileRepo, id);
    default:
      return Response.json(
        statusCode: 405,
        body: {'error': 'Method not allowed'},
      );
  }
}

Future<Response> _getProfile(ProfileRepository repo, String userId) async {
  final profile = await repo.findByUserId(userId);
  if (profile == null) {
    return Response.json(statusCode: 404, body: {'error': 'Profile not found'});
  }
  return Response.json(body: profile.toJson());
}

Future<Response> _createProfile(
  RequestContext context,
  ProfileRepository repo,
  String userId,
) async {
  final existing = await repo.findByUserId(userId);
  if (existing != null) {
    return Response.json(
      statusCode: 409,
      body: {'error': 'Profile already exists for this user'},
    );
  }

  final body = await context.request.json() as Map<String, dynamic>;
  final profile = await repo.create(
    userId: userId,
    bio: body['bio'] as String?,
    avatarUrl: body['avatarUrl'] as String?,
    phone: body['phone'] as String?,
    location: body['location'] as String?,
    website: body['website'] as String?,
  );
  return Response.json(statusCode: 201, body: profile.toJson());
}

Future<Response> _updateProfile(
  RequestContext context,
  ProfileRepository repo,
  String userId,
) async {
  final body = await context.request.json() as Map<String, dynamic>;
  final profile = await repo.update(
    userId: userId,
    bio: body['bio'] as String?,
    avatarUrl: body['avatarUrl'] as String?,
    phone: body['phone'] as String?,
    location: body['location'] as String?,
    website: body['website'] as String?,
  );
  if (profile == null) {
    return Response.json(statusCode: 404, body: {'error': 'Profile not found'});
  }
  return Response.json(body: profile.toJson());
}

Wiring the Middleware Pipeline

The routes and middleware are all written, but they aren't connected yet. In Dart Frog, the connection happens through _middleware.dart files placed strategically in the routes/ folder.

To review, a _middleware.dart file at the root level applies to every route in the project. A _middleware.dart inside a subfolder applies only to routes in that folder and below. This gives us precise, folder-scoped control over which middleware runs where without any manual registration or mounting.

Create routes/_middleware.dart for global middleware applied to every route:

import 'package:dart_frog/dart_frog.dart';
import '../lib/middleware/database_middleware.dart';
import '../lib/middleware/error_middleware.dart';

Handler middleware(Handler handler) {
  return handler
      .use(databaseMiddleware())
      .use(errorMiddleware());
}

Create routes/users/_middleware.dart to protect all user routes with authentication:

import 'package:dart_frog/dart_frog.dart';
import '../../lib/middleware/auth_middleware.dart';

Handler middleware(Handler handler) {
  return handler.use(authMiddleware());
}

This is one of the most elegant parts of Dart Frog's model. The routes/users/_middleware.dart file automatically applies auth to every route under routes/users/, including routes/users/index.dart, routes/users/[id].dart, and routes/users/[id]/profile.dart. The auth routes under routes/auth/ are untouched because they live outside the users/ folder.

There's no manual middleware mounting, no array of protected routes, and no route group configuration. The folder structure does the work.

Testing the API

With the server running and all routes wired up, we can verify the full flow end to end. Start the development server and run through each endpoint in order: register a user first to get a token, then use that token on the protected routes. Replace {userId} in the commands below with the actual ID returned from the register response.

Start the development server:

dart_frog dev
# Server is now running at: http://localhost:8080

Register a user:

curl http://localhost:8080/auth/register \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "email": "seyi@example.com",
    "password": "securepassword",
    "firstName": "Seyi",
    "lastName": "Dev"
  }'

Response:

{
  "user": {
    "id": "uuid-here",
    "email": "seyi@example.com",
    "firstName": "Seyi",
    "lastName": "Dev",
    "isActive": true,
    "createdAt": "2025-01-01T00:00:00.000Z",
    "updatedAt": "2025-01-01T00:00:00.000Z"
  },
  "token": "eyJhbGci..."
}

Login:

curl http://localhost:8080/auth/login \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"email": "seyi@example.com", "password": "securepassword"}'

Get all users:

curl http://localhost:8080/users \
  -H "Authorization: Bearer eyJhbGci..."

Get a specific user:

curl http://localhost:8080/users/{userId} \
  -H "Authorization: Bearer eyJhbGci..."

Create a profile:

curl http://localhost:8080/users/{userId}/profile \
  -X POST \
  -H "Authorization: Bearer eyJhbGci..." \
  -H "Content-Type: application/json" \
  -d '{
    "bio": "Flutter engineer turned backend developer",
    "location": "Lagos, Nigeria",
    "website": "https://example.com"
  }'

Update a user:

curl http://localhost:8080/users/{userId} \
  -X PUT \
  -H "Authorization: Bearer eyJhbGci..." \
  -H "Content-Type: application/json" \
  -d '{"firstName": "Oluwaseyi"}'

Delete a user:

curl http://localhost:8080/users/{userId} \
  -X DELETE \
  -H "Authorization: Bearer eyJhbGci..."

Deployment

With everything tested locally, the final step is getting the API live. Dart Frog makes this straightforward: a single CLI command generates a production-ready Dockerfile, and from there we deploy to Fly.io where the app will run as a containerized service alongside a managed PostgreSQL database.

Production Build

Dart Frog generates a production-ready Docker setup with a single command:

dart_frog build

This creates a build/ directory containing:

build/
  bin/
    server.dart         ← compiled entry point
  Dockerfile            ← production Dockerfile
  pubspec.yaml
  pubspec.lock

The generated Dockerfile is a multi-stage build, compiles to a native binary in the first stage, runs from a minimal Debian image in the second. You do not need to write this yourself.

Deploying to Fly.io

Step 1 β€” Authenticate:

fly auth login

Step 2 β€” Launch from the build directory:

cd build
fly launch

Fly detects the Dockerfile and prompts for configuration. Create a PostgreSQL database when asked.

Step 3 β€” Set secrets:

fly secrets set JWT_SECRET="your_production_jwt_secret"
fly secrets set JWT_EXPIRY_HOURS="24"

Step 4 β€” Deploy:

fly deploy

Step 5 β€” Verify:

curl https://your-app-name.fly.dev/auth/register \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"email":"test@example.com","password":"password123","firstName":"Seyi","lastName":"Dev"}'

Conclusion

Dart Frog sits exactly where it positions itself: between the raw control of Shelf and the full opinions of Serverpod. It takes the file-based routing model that has proven itself in the JavaScript ecosystem and brings it to Dart cleanly, without compromising on the language's strengths.

The routing model is its strongest feature. Looking at the routes/ folder tells you everything about your API: what endpoints exist, how they are grouped, and which middleware applies to which sections. That transparency makes codebases easier to navigate, easier to onboard into, and easier to reason about as they grow.

The RequestContext and the provider pattern for dependency injection are well thought out. Middleware injects, routes consume, and nothing bleeds between the two. The folder-scoped middleware is particularly clean, protecting an entire section of your API is as simple as dropping a _middleware.dart file in the right folder.

For Flutter engineers building APIs that need to serve multiple client types, conform to standard REST conventions, or integrate cleanly with existing frontend infrastructure, Dart Frog hits a practical sweet spot that neither Shelf nor Serverpod reaches as naturally.

Dart is now a full-stack language in the truest sense. The same team, the same language, the same conventions – from the Flutter app to the server that powers it.

Happy Coding!

]]> <![CDATA[ How to Scale Laravel Applications for High-Traffic Production Systems ]]> https://www.freecodecamp.org/news/how-to-scale-laravel-applications-for-high-traffic-production-systems/ 6a2b48a3a381db4fd3f61555 Thu, 11 Jun 2026 23:45:39 +0000 Your first scaling problem rarely arrives with a bang. For a while, everything is fine: pages load fast, the database barely breaks a sweat, and the team ships features without thinking much about infrastructure.

Then traffic climbs. A campaign over-performs. A marketplace onboards a popular seller. A SaaS product signs a couple of enterprise accounts.

Suddenly, /dashboard takes two seconds instead of 300 milliseconds. Queue jobs that used to clear in seconds sit waiting for minutes. You have database CPU spikes every afternoon.

So you add another app server, and response time barely moves because the real culprit was a slow query on a large table all along.

If you have run Laravel in production, you've probably lived some version of this. The good news is that scaling Laravel almost never means abandoning the framework. It means learning where pressure builds and making the application behave predictably under load.

In this guide, you'll learn how to find common bottlenecks, tune the database, use Redis effectively, move slow work onto queues, optimize APIs, and monitor a Laravel application in production.

None of this requires a single heroic rewrite. The biggest wins usually come from practical work: removing inefficient queries, pushing slow tasks onto queues, adding the right indexes, caching carefully chosen data, and measuring whether each change actually helped.

Prerequisites

You'll get the most out of this guide if you're already comfortable with:

  • Building applications with Laravel and PHP

  • Writing Eloquent queries and database migrations

  • Using queues, jobs, and scheduled commands

  • Reading a basic database query plan

  • Deploying Laravel to a production server or platform

  • Working with Redis and either MySQL or PostgreSQL in a production-like setup

Table of Contents

What Happens When Laravel Apps Start Growing

Traffic changes a system's behavior because it turns small inefficiencies into permanent costs. A query that takes 80 milliseconds is harmless when it runs a few hundred times an hour. Run it 30 times per page view on a page that gets thousands of hits a minute, and that same query becomes a capacity problem.

The pressure tends to show up in predictable places. More requests mean more PHP workers, more database connections, more queue volume, and more Redis operations.

The database, whether MySQL or PostgreSQL, is usually the first thing to buckle. Queues back up when work is created faster than workers can drain it. Caches only help when hit rates stay high and misses stay controlled. And scaling everything horizontally can turn sloppy code into an expensive cloud bill.

That's why scaling work has to start with measurement, not guesswork. Before you change anything, you want to know what is actually saturated: request CPU, database I/O, lock contention, Redis latency, queue depth, an external API, or oversized payloads.

A typical request in a growing Laravel app travels through several layers. The user sends a request, a load balancer routes it to an app server, and Laravel checks Redis for a cached result. On a miss, it queries the database, stores the computed result back in Redis, and hands any slow follow-up work to a queue. A worker picks up that job later while Laravel returns the response right away.

Here's the important part: adding more app servers does nothing for a slow query, a missing index, or an overloaded queue. Horizontal scaling only pays off once the shared dependencies behind those servers can keep up.

Common Laravel Bottlenecks

Laravel itself causes very few scaling problems. Most issues come from how application code talks to the database, the network, and background workers.

N+1 Queries

The classic offender is the N+1 query. You load a list of models, then lazily touch a relationship on each one:

use App\Models\Post;

$posts = Post::latest()->take(50)->get();

foreach (\(posts as \)post) {
    echo $post->author->name;
}

That's one query for the posts plus one query per author: 51 queries for a single page. Eager load the relationship instead:

use App\Models\Post;

$posts = Post::with('author')
    ->latest()
    ->take(50)
    ->get();

foreach (\(posts as \)post) {
    echo $post->author->name;
}

In production, these are sneaky. They often hide inside API Resources, Blade components, and authorization checks, where the relationship access isn't obvious from the controller.

Missing Indexes

Adding an index is one of the highest-return fixes you can make. Take a query like this:

\(orders = Order::where('account_id', \)accountId)
    ->where('status', 'paid')
    ->whereBetween('created_at', [\(start, \)end])
    ->latest()
    ->paginate(50);

If orders has millions of rows and no useful compound index, the database scans far more rows than it needs to. Add an index that matches how you actually query:

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration {
    public function up(): void
    {
        Schema::table('orders', function (Blueprint $table) {
            $table->index(['account_id', 'status', 'created_at']);
        });
    }

    public function down(): void
    {
        Schema::table('orders', function (Blueprint $table) {
            $table->dropIndex(['account_id', 'status', 'created_at']);
        });
    }
};

Indexes aren't free, though. They take up space and slow down writes. Add them for real, repeated query patterns, not for every column that ever appears in a where clause.

Inefficient Eager Loading

You can also swing too far the other way. Loading every relationship "just in case" burns memory and ships data the request never uses:

$users = User::with([
    'profile',
    'teams',
    'roles.permissions',
    'invoices.lineItems.product',
])->get();

That might be fine for an admin detail page showing one user. On a list page, it's a liability. Constrain the eager loads and select only the columns you need:

$users = User::query()
    ->select(['id', 'name', 'email'])
    ->with([
        'profile:id,user_id,avatar_url',
        'teams:id,name',
    ])
    ->latest()
    ->paginate(25);

One caveat: tightly scoped select lists can break later code that expects a column you didn't load. Keep this technique close to read-heavy endpoints where the payoff is obvious.

Synchronous Processing

High-traffic apps need short web requests. Sending email, generating PDFs, calling third-party APIs, resizing images, and building exports usually belong outside the request cycle. This version can hurt you:

public function store(Request $request)
{
    \(order = Order::create(\)request->validated());

    Mail::to(\(order->user)->send(new OrderReceipt(\)order));

    return response()->json($order, 201);
}

Push the work onto a queue instead:

public function store(StoreOrderRequest $request)
{
    \(order = Order::create(\)request->validated());

    SendOrderReceipt::dispatch($order->id);

    return response()->json([
        'id' => $order->id,
        'status' => 'accepted',
    ], 202);
}

Now your response time no longer depends on your mail provider. If the provider has a slow afternoon, the queue absorbs it and your users don't have to wait.

Large Payloads

Oversized JSON responses hurt everyone in the chain: the app server serializing them, the network carrying them, and the client parsing them. A frequent mistake is returning whole models when you meant to return a summary:

return User::with('orders', 'invoices', 'teams')->findOrFail($id);

Define an explicit API Resource instead:

use Illuminate\Http\Resources\Json\JsonResource;

class UserSummaryResource extends JsonResource
{
    public function toArray($request): array
    {
        return [
            'id' => $this->id,
            'name' => $this->name,
            'avatar_url' => $this->profile?->avatar_url,
            'plan' => $this->subscription_plan,
        ];
    }
}

A small, deliberate response contract keeps endpoint cost easy to reason about and prevents accidental coupling.

Expensive Joins

Joins are useful, but expensive joins across large tables can dominate your database time, especially when they sort or filter on columns that aren't indexed:

$rows = DB::table('orders')
    ->join('users', 'users.id', '=', 'orders.user_id')
    ->join('accounts', 'accounts.id', '=', 'users.account_id')
    ->where('accounts.region', 'us-east')
    ->where('orders.status', 'paid')
    ->orderByDesc('orders.created_at')
    ->limit(100)
    ->get();

At scale, you may need to denormalize a small field, precompute a reporting table, or move analytics off the primary transactional database entirely. Do not treat denormalization as an admission of defeat. Copying a stable field like account_id onto orders can remove a costly join from a hot path. The price you pay is keeping that duplicated data consistent, which can be a worthwhile trade-off.

How to Optimize the Database

When a Laravel app slows down, the database is usually the first place to look.

Add Indexes Around Real Query Patterns

Start with your slow query log, database metrics, and traces rather than intuition. If the app constantly looks up active subscriptions by account, build a compound index that matches that access pattern:

Schema::table('subscriptions', function (Blueprint $table) {
    $table->index(['account_id', 'status', 'renews_at']);
});

Then write the query so it can actually use the index:

\(subscription = Subscription::where('account_id', \)accountId)
    ->where('status', 'active')
    ->where('renews_at', '>=', now())
    ->orderBy('renews_at')
    ->first();

Get in the habit of running EXPLAIN after you add an index to confirm that the plan changed. An index the optimizer ignores is just write overhead.

Use Eager Loading Deliberately

Match eager loading to what the endpoint actually returns. For list endpoints, keep relationships shallow and constrained:

$projects = Project::query()
    ->select(['id', 'account_id', 'name', 'updated_at'])
    ->withCount('openTasks')
    ->with([
        'owner:id,name',
    ])
    ->where('account_id', $accountId)
    ->latest('updated_at')
    ->paginate(30);

When you only need a number, withCount beats loading a whole relationship to count it:

$teams = Team::query()
    ->withCount([
        'members',
        'invitations as pending_invitations_count' => fn (\(query) => \)query->whereNull('accepted_at'),
    ])
    ->paginate(25);

Your memory footprint stays flat, which matters much more on a list page than on a detail page.

Optimize Queries Before Adding Hardware

A bigger database instance buys you time. It also hides the inefficient queries that put you there until the next traffic jump exposes them again. Before you reach for a larger machine, find your highest-cost queries. In local or staging environments, logging slow ones is easy:

use Illuminate\Database\Events\QueryExecuted;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Log;

DB::listen(function (QueryExecuted $query) {
    if ($query->time > 100) {
        Log::warning('Slow query detected', [
            'sql' => $query->toRawSql(),
            'time_ms' => $query->time,
        ]);
    }
});

Be careful doing this in production. Bindings can contain sensitive data, and verbose logging at high volume can become its own performance problem.

Process Large Tables with Chunking

Never pull an entire large table into memory for a batch job:

User::where('is_active', true)
    ->chunkById(1000, function ($users) {
        foreach (\(users as \)user) {
            RefreshUserSearchIndex::dispatch($user->id);
        }
    });

chunkById is safer than offset-based chunking when rows can change while the job runs, because it tracks the last seen ID instead of a numeric offset. For very large exports, stream the records or write them out in batches.

Use Cursor Pagination for High-Volume Feeds

Offset pagination gets slower the deeper a user scrolls, because the database still has to skip every row it's not returning. For feeds, audit logs, messages, and timelines, cursor pagination is usually the better fit:

$events = AuditEvent::query()
    ->where('account_id', $accountId)
    ->orderByDesc('id')
    ->cursorPaginate(50);

return AuditEventResource::collection($events);

It relies on a stable, indexed ordering column and uses next/previous cursors rather than arbitrary page numbers, which is what an infinite-scroll feed usually needs.

Split Reads with Read Replicas

As read traffic grows, replicas can take load off the primary:

'mysql' => [
    'driver' => 'mysql',
    'read' => [
        'host' => [
            env('DB_READ_HOST', '127.0.0.1'),
        ],
    ],
    'write' => [
        'host' => [
            env('DB_WRITE_HOST', '127.0.0.1'),
        ],
    ],
    'sticky' => true,
    'database' => env('DB_DATABASE', 'laravel'),
    'username' => env('DB_USERNAME', 'root'),
    'password' => env('DB_PASSWORD', ''),
],

The sticky option keeps reads on the write connection after a write within the same request, which helps avoid some read-after-write surprises.

Replicas come with replication lag, and that lag matters. Don't route payment confirmations, password changes, permission checks, or anything else consistency-sensitive to a replica that might be a few seconds stale unless the business flow can genuinely tolerate seeing old data.

How to Scale with Redis

Redis often does a lot in a Laravel production stack: caching, sessions, rate limiting, queues, locks, and Horizon metrics. It's fast, but it still needs thought: sensible key design, expiration policies, memory monitoring, and a real plan for invalidation.

Caching

Cache expensive reads that get requested often and can tolerate being slightly out of date:

use Illuminate\Support\Facades\Cache;

$stats = Cache::remember(
    "accounts:{$account->id}:dashboard-stats",
    now()->addMinutes(5),
    fn () => DashboardStats::forAccount($account)->calculate()
);

Short time-to-live values go a surprisingly long way. A five-minute cache can wipe out thousands of duplicate queries while keeping the data fresh enough for most dashboards.

When the data changes after a known event, invalidate it explicitly:

Order::created(function (Order $order) {
    Cache::forget("accounts:{$order->account_id}:dashboard-stats");
});

Caching works best when your keys are predictable and your invalidation is tied to domain events rather than guesswork.

Sessions

For horizontally scaled app servers, file-based sessions are a trap: the next request can land on a different server that has never seen the session. Store sessions in Redis or a database so any server can handle any request:

SESSION_DRIVER=redis
CACHE_STORE=redis
QUEUE_CONNECTION=redis

Rate Limiting

Rate limits protect you from abusive clients, runaway loops, and endpoints that get hammered:

use Illuminate\Cache\RateLimiting\Limit;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\RateLimiter;

RateLimiter::for('api', function (Request $request) {
    return Limit::perMinute(120)->by(
        optional(\(request->user())->id ?: \)request->ip()
    );
});

Expensive endpoints deserve stricter limits:

RateLimiter::for('exports', function (Request $request) {
    return Limit::perHour(10)->by($request->user()->id);
});

Let business cost drive the numbers. Login, search, export, and webhook endpoints rarely need the same limit.

Queues

Redis is a common queue backend because it's quick and Horizon supports it well:

QUEUE_CONNECTION=redis

Dispatch work onto named queues from the request:

GenerateInvoicePdf::dispatch($invoice->id)
    ->onQueue('documents');

Split work by profile, such as default, emails, webhooks, documents, and imports, because each workload can need different worker counts and retry rules. Keep the names meaningful. During an incident, "the documents queue is 20 minutes behind" tells you far more than "default is slow."

How to Use Queue-Driven Architectures

Queues are one of Laravel's best scaling tools. They let the app accept work quickly and process it asynchronously with controlled concurrency. They also make the system more resilient: when a third-party API goes down, jobs retry on their own instead of tying up your PHP-FPM request workers.

Laravel Queues

A good job is small, idempotent, and safe to retry:

use App\Mail\OrderReceiptMail;
use App\Models\Order;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Support\Facades\Mail;

class SendOrderReceipt implements ShouldQueue
{
    use Queueable;

    public int $tries = 3;
    public int $backoff = 60;

    public function __construct(public int $orderId)
    {
    }

    public function handle(): void
    {
        \(order = Order::with('user')->findOrFail(\)this->orderId);

        Mail::to(\(order->user)->send(new OrderReceiptMail(\)order));
    }
}

Pass IDs into jobs rather than full Eloquent models. The model might change before the job runs, and serializing a whole model bloats the payload. For external APIs, add timeouts and guard against duplicate work:

use App\Models\Order;
use App\Services\CrmClient;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;

class SyncOrderToCrm implements ShouldQueue
{
    use Queueable;

    public int $tries = 3;
    public int $backoff = 60;

    public function __construct(public int $orderId)
    {
    }

    public function handle(CrmClient $crm): void
    {
        \(order = Order::findOrFail(\)this->orderId);

        if ($order->crm_synced_at) {
            return;
        }

        \(crm->upsertOrder(\)order->external_reference, [
            'total' => $order->total,
            'status' => $order->status,
        ]);

        $order->forceFill(['crm_synced_at' => now()])->save();
    }
}

The crm_synced_at check is the whole point. Jobs run more than once in real life, and idempotency is what keeps a retry from double-charging or double-syncing.

Horizon

Horizon gives you visibility and control over Redis queues. A typical setup runs different supervisors for different workloads:

'production' => [
    'supervisor-default' => [
        'connection' => 'redis',
        'queue' => ['default', 'emails'],
        'balance' => 'auto',
        'maxProcesses' => 20,
        'tries' => 3,
    ],

    'supervisor-documents' => [
        'connection' => 'redis',
        'queue' => ['documents'],
        'balance' => 'simple',
        'maxProcesses' => 5,
        'tries' => 2,
        'timeout' => 300,
    ],
],

The separation matters: a long-running document job shouldn't starve a quick password-reset email.

Failed Jobs and Retries

Retries only help when failures are temporary. Retrying a job that's permanently broken just burns capacity. For jobs with a business deadline, use retryUntil:

use DateTime;
use Throwable;

public function retryUntil(): DateTime
{
    return now()->addMinutes(30);
}

public function failed(Throwable $exception): void
{
    ImportBatch::whereKey($this->batchId)->update([
        'status' => 'failed',
        'failed_reason' => $exception->getMessage(),
    ]);
}

Use failed to flag the problem somewhere a human will see it. Whatever you do, don't set unlimited retries on jobs that hit a third-party service.

Queue Monitoring

Track queue depth, wait time, failure rate, and processing time together. Depth alone can mislead you. When depth starts climbing, walk through it methodically: are workers keeping pace with incoming jobs? If the queue keeps growing, check how long individual jobs take. If the slow part is the database, fix the query or dial back worker concurrency. If it's an external API, add backoff or a circuit breaker. If the work is CPU-bound, scale workers or break the jobs into smaller pieces.

Be careful with the "scale workers" instinct, though. Adding more workers without checking the database first can make an incident worse. More workers mean more concurrent queries, more locks, and more pressure on the primary exactly when it's already struggling.

How to Optimize API Performance

APIs earn special attention because clients call them repeatedly and payloads tend to grow quietly over months.

API Resources

Resources keep your response shape intentional:

class OrderResource extends JsonResource
{
    public function toArray($request): array
    {
        return [
            'id' => $this->id,
            'status' => $this->status,
            'total' => $this->total,
            'placed_at' => $this->created_at->toIso8601String(),
            'customer' => new CustomerSummaryResource($this->whenLoaded('customer')),
        ];
    }
}

whenLoaded is doing real work here. It stops the resource from quietly triggering a lazy query when the relationship wasn't eager loaded:

$orders = Order::query()
    ->with('customer:id,name')
    ->where('account_id', $accountId)
    ->latest()
    ->paginate(50);

return OrderResource::collection($orders);

Pagination

Returning unbounded collections is an easy way to create an API performance problem you won't notice until a client has a lot of data:

$perPage = min((int) request('per_page', 50), 100);

\(orders = Order::where('account_id', \)accountId)
    ->latest()
    ->paginate($perPage);

Cap the page size. If a client genuinely needs every record for an export, make that an async job rather than a giant synchronous response.

Response Optimization

Stop returning fields nobody reads. On read-heavy endpoints, selecting only the columns you need cuts both database I/O and serialization cost:

$products = Product::query()
    ->select(['id', 'name', 'slug', 'price', 'thumbnail_url'])
    ->where('is_visible', true)
    ->orderBy('name')
    ->paginate(40);

It's also worth turning on compression at the web server or load balancer. JSON compresses extremely well, and that's often a small config change with a real bandwidth payoff.

Rate Limiting

Design API rate limits around identity and endpoint cost:

Route::middleware(['auth:sanctum', 'throttle:api'])
    ->group(function () {
        Route::get('/orders', [OrderController::class, 'index']);
        Route::post('/exports/orders', [OrderExportController::class, 'store'])
            ->middleware('throttle:exports');
    });

This keeps casual browsing and expensive exports under separate policies, so one heavy user can't squeeze out everyone else.

Caching API Responses

Cache responses that are expensive to compute and can tolerate being a little stale:

public function index(Request $request)
{
    \(accountId = \)request->user()->account_id;
    \(page = \)request->integer('page', 1);

    \(cacheKey = "api:accounts:{\)accountId}:orders:v1:page:{$page}";

    return Cache::remember(\(cacheKey, now()->addSeconds(60), function () use (\)accountId) {
        return OrderResource::collection(
            Order::with('customer:id,name')
                ->where('account_id', $accountId)
                ->latest()
                ->paginate(50)
        )->response()->getData(true);
    });
}

Notice the v1 in the key. Bumping that version number lets you invalidate an entire response format at once when the shape changes. Always scope the key to the tenant or user for anything that's not truly global.

How to Monitor Laravel in Production

The teams that catch problems before customers do are the ones collecting signals from everywhere: Laravel, queues, the database, Redis, the infrastructure, and external services.

Laravel gives you several good starting points. Horizon shows queue throughput, failed jobs, wait times, and worker balancing. Telescope surfaces request details, queries, exceptions, jobs, mail, and cache events. Your logs capture slow operations, unexpected retries, and external failures. Your metrics track latency, error rate, queue depth, job runtime, database CPU, lock waits, cache hit ratio, and Redis memory. Your alerting ties all of it back to something a customer would actually feel.

That last part is where teams often make mistakes. The best alerts are about symptoms, not machines being busy: p95 API latency over 800ms for 10 minutes, checkout error rate above 1%, the emails queue waiting more than 5 minutes, database CPU over 85% with slow queries rising, Redis memory over 80%, or failed payment webhooks crossing a threshold.

A useful mental model is this: logs tell you what happened, metrics tell you whether the system is healthy, and traces tell you where the time went. In practice, wrapping your expensive business operations in a bit of instrumentation pays off quickly:

use Illuminate\Support\Facades\Log;

$startedAt = microtime(true);

\(report = \)builder->forAccount($account)->build();

Log::info('Billing report generated', [
    'account_id' => $account->id,
    'duration_ms' => (int) ((microtime(true) - $startedAt) * 1000),
    'invoice_count' => $report->invoiceCount(),
]);

When something is failing at 2am, a log line like that can tell you which account, import, or report is causing the pressure.

One more thing worth internalizing: monitor wait time, not just throughput. A queue can process thousands of jobs a minute and still be unhealthy if important jobs sit waiting too long before they start. Users feel the wait, not the throughput.

An Example High-Traffic Laravel Architecture

A high-traffic Laravel setup generally separates four things: stateless web requests, shared cache and session storage, asynchronous workers, and database roles.

Users hit a load balancer, which spreads traffic across a fleet of stateless Laravel app servers. Those servers use Redis for cache, sessions, rate limits, queues, and Horizon data. Queue workers handle slow or unreliable work off to the side. A MySQL primary takes all writes and any consistency-sensitive reads, while a read replica absorbs read-heavy endpoints that can tolerate some replication lag.

The flow looks like this:

Users
  -> Load balancer
  -> Stateless Laravel app servers
  -> Redis for cache, sessions, rate limits, queues, and Horizon data
  -> Primary database for writes and consistency-sensitive reads
  -> Read replica for safe read-heavy endpoints

Redis queue
  -> Queue workers
  -> Database, external APIs, mail providers, object storage, and other services

This isn't the only valid shape. PostgreSQL can stand in for MySQL, Amazon SQS can replace Redis queues, a CDN can serve static assets and cache public responses, and object storage should hold user uploads. The principle that matters is that each layer has one clear job and can be scaled or tuned on its own.

The flip side of stateless app servers is that anything a user needs after the request ends has to live in shared storage. Uploads, generated files, and session state shouldn't sit on a single server's local disk, or they may disappear from the user's point of view when the load balancer sends the next request somewhere else.

Lessons Learned the Hard Way

1. Premature Optimization

This usually shows up as elaborate infrastructure built before the app has any real visibility into itself.

The practical path works better: measure, rank the bottlenecks, fix the biggest one, repeat. For most Laravel apps, the first round of scaling is mostly indexes, N+1 fixes, queue separation, and trimming payloads.

2. Over-caching

Caching can make a system faster and harder to reason about at the same time. One team cached an account-settings response for 30 minutes, then later folded role changes into that same response. The result was that users who had just lost access could still see features until the cache expired.

The fix was splitting stable account metadata away from permission-sensitive state. The lesson is to avoid caching authorization data unless you have thought carefully about invalidation.

3. Missing Indexes

These hide until a table crosses a size threshold. A query that scanned 20,000 rows in development can scan 20 million in production. Bake index review into feature work, and plan big index migrations carefully so they don't lock a hot table at the worst possible time.

4. Queue Overload

Queues don't remove work, they move it. The classic failure is letting one noisy workload block everything else. A big CSV import floods the default queue, and password-reset emails get stuck behind it. Separate queues are cheap insurance against that entire class of incident.

5. Large Transactions

Long transactions hold locks longer and make failures more expensive. Dispatching a job inside a transaction is especially risky because a worker can grab it before the transaction commits:

DB::transaction(function () use ($request) {
    $order = Order::create([...]);
    \(order->items()->createMany(\)request->items);

    GenerateInvoicePdf::dispatch($order->id);
    SyncOrderToCrm::dispatch($order->id);
});

Use after-commit dispatching for any job that depends on committed data:

GenerateInvoicePdf::dispatch($order->id)->afterCommit();
SyncOrderToCrm::dispatch($order->id)->afterCommit();

Keep transactions scoped to the data that genuinely has to change atomically, and nothing more.

6. Treating Symptoms as Causes

This is the expensive one. If latency is high because an endpoint runs 300 queries, adding app servers adds database pressure. If jobs are slow because an external API is rate-limiting you, adding workers multiplies the failures.

Good scaling work keeps asking the same questions: What resource is saturated? Which endpoint, job, tenant, or query is causing it? Is this work necessary during the request? Can I reduce it, defer it, cache it, or isolate it? How will I know whether the change helped?

A Pre-Launch Scaling Checklist

Run through this before a big launch, a traffic campaign, or an enterprise rollout.

Application and runtime: Cache config, routes, and views during deploy. Set APP_DEBUG=false. Turn on OPcache. Keep web requests short and move slow work to queues. Store uploads in object storage, not on app-server disk. Keep servers stateless. Set timeouts on every external HTTP call.

Database: Review slow query logs first. Add indexes for your high-volume filters, joins, and ordering. Hunt for N+1 queries in controllers, resources, policies, and views. Paginate every list endpoint. Use chunkById or cursors for batch work. Avoid long transactions and external calls inside transactions. Confirm your backup and restore process works. Test stale-read behavior if you use replicas.

Redis and cache: Use Redis for cache, sessions, rate limiting, and queues where it fits. Set TTLs unless you have a clear reason not to. Include tenant, user, locale, and version in keys when relevant. Watch memory and the eviction policy. Avoid caching permission-sensitive responses without careful invalidation. Guard against cache stampedes on expensive recomputation.

Queues: Separate queues by workload. Configure Horizon supervisors per queue. Set timeouts, retries, and backoff on purpose. Make jobs idempotent where you can. Use afterCommit for jobs that depend on committed data. Monitor wait time, runtime, failures, and retries. Review failed jobs instead of ignoring them.

APIs: Use Resources to control response shape. Cap per_page. Use cursor pagination for big feeds and logs. Cache expensive reads with safe, versioned keys and short TTLs. Apply rate limits by endpoint cost. Don't return raw Eloquent models. Compress responses at the edge.

Observability: Track p50, p95, and p99 latency on the endpoints that matter. Track error rates by route and job class. Alert on queue wait time, not just size. Watch database CPU, connections, slow queries, and lock waits. Watch Redis memory, latency, and evictions. Log important business operations with durations and identifiers. Test your alerts before launch night because a silent alert is worse than no alert.

Conclusion

Laravel runs high-traffic production systems well when you design around the real costs of data, concurrency, and external dependencies. Just make sure you measure before you optimize, because guessing wastes time and tends to complicate the wrong layer.

Fix the database first: indexes, query shape, pagination, and eager loading usually deliver the biggest early wins. Lean on queues to keep requests fast and push slow work into controlled background workers. Cache deliberately, with clear keys, sane TTLs, and a plan for invalidation. Keep watching latency, errors, queue wait time, database health, Redis memory, and your external dependencies.

The best scaling work is practical and repeatable. You study the system you actually have, remove waste, isolate slow parts, and give yourself enough visibility to make the next change with confidence. Do that on a loop, and you rarely need the big rewrite.

References

]]> <![CDATA[ How to Start your Career in Tech with freeCodeCamp - Full Talk in Spanish ]]> https://www.freecodecamp.org/news/how-to-start-your-career-in-tech-with-freecodecamp-full-talk-in-spanish/ 6a2ace357fbd41b5facfabdf Thu, 11 Jun 2026 15:03:17 +0000 Technology is rapidly reshaping the world. Understanding how to use free learning resources and contribute to open source projects can be very helpful to start your career in this field.

We just published a talk on the freeCodeCamp Spanish YouTube channel about how to leverage freeCodeCamp's free learning resources and community to start your career in technology. You’ll learn how to find these resources and the core concepts that you need to know to start contributing to open source.

If you have Spanish-speaking friends, you're welcome to share the Spanish version of this article with them.

This talk was presented by Estefania (me!). I'm part of the freeCodeCamp team. I develop educational content and manage the freeCodeCamp Spanish YouTube channel. I love helping others learn and grow professionally. I gave this virtual talk for the Escuela Superior PolitΓ©cnica del Litoral, located in Guayaquil, Ecuador.

Why Consider a Career in Tech?

Before we dive into the content of the talk, let's see what careers in technology involve, and why you should start now if having a career in this field is your goal.

Technology careers can be very interesting because they give you the opportunity to solve real-world problems. Being part of the technology field means that you'll have the chance to help shape the future of humanity. By joining the freeCodeCamp community, you'll find the support and the practical tools you need to get started.

Learning to code and contributing to open source projects, like freeCodeCamp, are two fundamental steps for reaching your goal. Contributing to open source projects develops your technical skills and connects you with a global network of developers who share your interests and goals. By contributing to open source, you’ll gain hands-on experience, improve your portfolio, and increase your chances of landing your first job.

What You'll Learn During the Talk

Great. Let's see what you’ll learn during the talk:

  • The story and mission of freeCodeCamp.org.

  • How to use freeCodeCamp’s free learning resources to learn programming.

  • freeCodeCamp's certifications.

  • freeCodeCamp's daily coding challenges.

  • freeCodeCamp's catalog and forums.

  • freeCodeCamp's YouTube channels with full courses.

  • freeCodeCamp's publication.

  • Additional free learning resources provided by freeCodeCamp.

  • How to contribute to open source projects and why this is important.

  • Common terminology used in open source projects.

  • Personal tips for getting started in your first job.

  • How to join the freeCodeCamp community.

By the end of the talk, you’ll know how to find and leverage freeCodeCamp's free learning resources to learn programming and start your career.

Talk on YouTube

Check out the talk on the freeCodeCamp Spanish YouTube channel:

✍️ Talk presented by Estefania Cassingena Navone.

Collaborating with:

]]> <![CDATA[ Web Scraping for Beginners 2026 ]]> https://www.freecodecamp.org/news/web-scraping-for-beginners-2026/ 6a28c9110fd9e5ca15ede211 Wed, 10 Jun 2026 02:16:49 +0000 If you have ever wanted to collect product data, monitor competitors, track SEO rankings, or build AI tools that pull information from the internet, you have likely run into the common frustrations of web scraping: broken scripts, rate limits, bot detection, and tedious CAPTCHAs.

We just published a new tutorial on the freeCodeCamp.org YouTube channel, featuring software developer and course creator Ania Kubow.

In this comprehensive, beginner-friendly course, Ania teaches you a much simpler, more efficient approach. Instead of building scrapers from scratch, you will learn how to leverage an API to handle the heavy lifting for you.

Throughout this tutorial, you will master the following:

  • How to bypass web scraping obstacles like bot protection and rate limits using a powerful API.

  • How to extract structured JSON data directly from search engines like Google, Amazon, YouTube, and more.

  • How to use the Google Lens API to scrape images and visual matches.

  • How to build your own functional web application that searches for and downloads content locally to your computer.

By the end of this video, you will have the knowledge and the basic code necessary to turn internet data into actionable insights for your own projects.

Watch the full tutorial on the freeCodeCamp.org YouTube channel (1 -hour watch).

]]> <![CDATA[ How to Build a PostgreSQL-Backed Job Queue in Go ]]> https://www.freecodecamp.org/news/how-to-build-a-postgresql-backed-job-queue-in-go/ 6a28a0135ea1e6904efb11dc Tue, 09 Jun 2026 23:21:55 +0000 When you build a web application, not every task should happen inside a user's request.

Some work is slow. Some work can fail. Some work should happen later. Sending emails, resizing images, processing webhooks, generating reports, and retrying third-party APIs are all good examples.

These tasks are usually handled by a background job system.

In this article, you'll use an open source Go project called Swig as a practical example of how a PostgreSQL-backed job queue works in practice.

By the end, you'll understand how to build a background job queue with Go and PostgreSQL, and why PostgreSQL is more capable than most developers realize.

Table of Contents

  1. Prerequisites

  2. What You Will Learn

  3. What Is a Job Queue?

  4. Why Use PostgreSQL for a Queue?

  5. Swig's Architecture

  6. How to Represent Jobs in PostgreSQL

  7. How to Define a Worker in Go

  8. How to Register Workers Without Sharing State

  9. How to Add a Job

  10. How to Handle Multiple Workers Safely

  11. How to Use Goroutines for Concurrent Workers

  12. How to Wake Workers with LISTEN/NOTIFY

  13. How to Elect a Leader with Advisory Locks

  14. How to Handle Failed Jobs

  15. How to Abstract the Database Driver

  16. Conclusion

Prerequisites

To follow along, you should have:

  • Basic familiarity with Go (structs, interfaces, goroutines)

  • A working understanding of PostgreSQL and SQL

  • Go installed (1.21 or later)

  • A PostgreSQL instance available locally or remotely

What You Will Learn

  • How to represent and store jobs in PostgreSQL

  • How to claim jobs safely across concurrent workers using FOR UPDATE SKIP LOCKED

  • How to wake workers efficiently using LISTEN/NOTIFY

  • How to elect a leader across instances using advisory locks

  • How Go interfaces, goroutines, contexts, and transactions fit together in a real system

What Is a Job Queue?

A job queue is a system that stores work to be done later.

Your application adds a job to the queue. A worker takes a job from the queue and runs it.

For example, when a user signs up, your application might create the user immediately and then add a job like this:

{
  "kind": "send_welcome_email",
  "payload": {
    "to": "user@example.com",
    "subject": "Welcome!"
  }
}

A background worker later picks up that job and sends the email. This keeps the user request fast. The signup route doesn't need to wait for the email provider before returning a response.

A job queue usually needs to answer a few important questions:

  • Where are jobs stored?

  • How do workers find jobs?

  • How do you stop two workers from processing the same job?

  • How do you retry failed jobs?

  • How do you shut workers down safely?

  • How do you keep job creation consistent with application data?

Swig answers those questions with Go and PostgreSQL.

Why Use PostgreSQL for a Queue?

Many job queues use Redis, RabbitMQ, SQS, or Kafka. Those are all useful tools. But many applications already depend on PostgreSQL. If your app already has Postgres, you may not want to operate another service just to run background jobs.

PostgreSQL gives you several features that are surprisingly useful for queues:

  • Tables for durable job storage

  • Transactions for atomic writes

  • Row locks for safe concurrent processing

  • SKIP LOCKED for letting workers claim different jobs

  • LISTEN/NOTIFY for waking workers when new jobs arrive

  • Advisory locks for leader election

  • JSONB for flexible job payloads

The tradeoff is important. A PostgreSQL-backed queue isn't trying to replace Kafka for event streaming or RabbitMQ for complex routing. It makes common application background jobs simple, reliable, and easy to operate without adding infrastructure.

Swig's Architecture

At a high level, Swig has five parts:

  1. A swig_jobs table in PostgreSQL

  2. Go workers that process jobs

  3. A worker registry that maps job names to worker types

  4. A driver layer that supports both pgx and database/sql

  5. A leader loop for shared maintenance work

The basic flow looks like this:

  1. Your app calls AddJob

  2. Swig serializes the job payload to JSON

  3. Swig inserts a row into swig_jobs

  4. PostgreSQL sends a notification that a job was created

  5. A Go worker wakes up and tries to claim one pending job

  6. PostgreSQL row locks ensure only one worker claims that row

  7. The worker runs the job

  8. Swig marks the job as completed or failed

The hard parts are concurrency, failure, connection lifecycle, and shutdown. That's where Go and PostgreSQL work together.

How to Represent Jobs in PostgreSQL

A simplified version of Swig's job table looks like this:

CREATE TABLE swig_jobs (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  kind TEXT NOT NULL,
  queue TEXT NOT NULL,
  payload JSONB NOT NULL,
  status TEXT NOT NULL DEFAULT 'pending',
  priority INTEGER NOT NULL DEFAULT 0,
  attempts INTEGER NOT NULL DEFAULT 0,
  max_attempts INTEGER NOT NULL DEFAULT 3,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  scheduled_for TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  instance_id UUID,
  worker_id UUID,
  locked_at TIMESTAMPTZ,
  last_error TEXT,
  last_error_at TIMESTAMPTZ
);

Each row is one job. The important columns are:

  • kind: the type of job, such as send_email

  • payload: the JSON data needed to run the job

  • status: whether the job is pending, processing, completed, or failed

  • attempts: how many times the job has been tried

  • scheduled_for: when the job is allowed to run

  • locked_at: when the job was claimed

The table is the source of truth. PostgreSQL notifications can wake workers, but notifications aren't the durable queue. The rows in swig_jobs are.

How to Define a Worker in Go

In Swig, a worker is a Go type that knows how to process one kind of job.

Here's a simple email worker:

type EmailWorker struct {
    To      string `json:"to"`
    Subject string `json:"subject"`
    Body    string `json:"body"`
}

func (w *EmailWorker) JobName() string {
    return "send_email"
}

func (w *EmailWorker) Process(ctx context.Context) error {
    fmt.Printf("Sending email to %s with subject %s\n", w.To, w.Subject)
    return nil
}

There are two important methods:

  • JobName tells Swig what kind of job this worker handles

  • Process contains the actual work

The struct fields are also the job arguments. When you enqueue an EmailWorker, Swig serializes the struct into JSON and stores it in PostgreSQL. Later, a worker claims the row, unmarshals the JSON back into a fresh EmailWorker, and calls Process.

Go Interfaces

Go interfaces describe behavior. Swig doesn't need to know the exact concrete type of every worker. It only needs to know that a worker can provide a job name and process a job:

type Worker interface {
    JobName() string
    Process(context.Context) error
}

If a type has those methods, it satisfies the interface with no explicit declaration required. This is one of the reasons interfaces are so useful in Go. They let you design around behavior instead of inheritance.

How to Register Workers Without Sharing State

Swig has a worker registry that maps a job name to a worker type:

registry := workers.NewWorkerRegistry()
registry.RegisterWorker(&EmailWorker{})

Later, when a job row says kind = 'send_email', Swig looks up the registered worker and runs it.

There's a subtle concurrency issue here. If the registry stored the exact &EmailWorker{} pointer and reused it for every job, multiple goroutines could unmarshal payloads into the same Go value at the same time.

Swig avoids this with a factory approach internally. Registration captures the worker type, and each claimed job gets a fresh worker instance before JSON is unmarshaled. The API stays simple, but internally Swig creates a new EmailWorker for each job. This is a useful Go pattern: keep the public API simple while making the internal lifecycle safer.

How to Add a Job

Here's what adding a job looks like from the user side:

err := swigClient.AddJob(ctx, &EmailWorker{
    To:      "user@example.com",
    Subject: "Welcome!",
    Body:    "Thanks for signing up.",
})

Inside Swig, the process is roughly:

argsJSON, err := json.Marshal(workerWithArgs)
if err != nil {
    return err
}

_, err = db.ExecContext(ctx, `
    INSERT INTO swig_jobs (kind, queue, payload, priority, scheduled_for, status)
    VALUES (\(1, \)2, \(3, \)4, $5, 'pending')
`, jobName, queue, argsJSON, priority, runAt)

How to Enqueue Jobs Inside Transactions

One of the best reasons to use PostgreSQL for jobs is transactional enqueueing.

Imagine a user signs up. You want to insert the user and queue a welcome email. If those happen separately, you can get inconsistent states. With a transaction, both succeed or both fail:

tx, err := pool.Begin(ctx)
if err != nil {
    return err
}
defer tx.Rollback(ctx)

_, err = tx.Exec(ctx, `INSERT INTO users (email) VALUES ($1)`, email)
if err != nil {
    return err
}

err = swigClient.AddJobWithTx(ctx, tx, &EmailWorker{
    To:      email,
    Subject: "Welcome!",
    Body:    "Thanks for joining.",
})
if err != nil {
    return err
}

return tx.Commit(ctx)

If the transaction rolls back, the user isn't created and the job isn't queued. This is much harder to guarantee when your database and queue are separate systems.

How to Handle Multiple Workers Safely

A queue gets interesting when many workers run at the same time. Imagine three workers all asking PostgreSQL for the next pending job. You don't want all three to process the same job.

A naΓ―ve approach has a race condition. Two workers can select the same job before either one updates it.

PostgreSQL FOR UPDATE SKIP LOCKED

PostgreSQL can lock rows selected inside a transaction. FOR UPDATE means "lock this row because I plan to update it." SKIP LOCKED means "if another worker already locked a row, skip it and find another one."

This is perfect for a queue:

  • Worker A locks job 1

  • Worker B skips job 1 and locks job 2

  • Worker C skips jobs 1 and 2 and locks job 3

No central coordinator is needed. Swig uses an atomic update pattern:

UPDATE swig_jobs
SET status = 'processing',
    instance_id = $1,
    worker_id = $2,
    locked_at = NOW(),
    attempts = attempts + 1
WHERE id = (
  SELECT id
  FROM swig_jobs
  WHERE status = 'pending'
    AND scheduled_for <= NOW()
  ORDER BY priority DESC, created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING id, kind, payload;

This query finds a pending job, skips already-locked jobs, marks it as processing, records which worker claimed it, and returns the job data. All of this happens atomically. Workers never do a separate SELECT and hope the later UPDATE is still safe.

How to Use Goroutines for Concurrent Workers

Swig starts worker loops as goroutines:

for i := 0; i < maxWorkers; i++ {
    go s.startWorker(ctx, queueType)
}

Each worker runs independently. PostgreSQL coordinates which job each worker gets. Go handles concurrency with goroutines, while PostgreSQL handles safe job claiming with locks.

How to Handle Graceful Shutdown

When a service shuts down, it should wait for workers to finish cleanly. Go's sync.WaitGroup helps:

var wg sync.WaitGroup

wg.Add(1)
go func() {
    defer wg.Done()
    processJobs()
}()

wg.Wait()

Swig also uses sync.Once to make shutdown idempotent. Calling Stop more than once shouldn't panic because of a double channel close. Shutdown paths are often where production systems behave differently from happy-path demos.

How to Wake Workers with LISTEN/NOTIFY

If workers constantly poll the database for jobs, they waste resources when the queue is empty. PostgreSQL has LISTEN/NOTIFY to solve this.

A connection can listen on a channel:

LISTEN swig_jobs;

Another session can send a notification:

NOTIFY swig_jobs, '{"id":"job-id"}';

Swig creates a trigger so PostgreSQL sends a notification after a job is inserted. Workers sleep when there's no work and wake when a new job arrives.

There's an important PostgreSQL detail here: LISTEN is session-scoped. A worker must wait for notifications on the same database session that executed LISTEN. Swig handles this by creating a dedicated listener for each worker that owns one database session throughout its lifecycle.

This is a common backend engineering lesson: abstractions like connection pools are useful, but some database features depend on the lifecycle of a specific connection.

How to Elect a Leader with Advisory Locks

Some queue maintenance tasks should only run on one instance at a time, including retrying failed jobs, recovering stale jobs, and cleaning old history.

Swig uses PostgreSQL advisory locks for this:

SELECT pg_try_advisory_lock($1);

If the result is true, that Swig instance becomes the leader. Advisory locks are also session-scoped, so Swig uses a dedicated advisory-lock connection for leadership. If that session ends, PostgreSQL releases the lock and another instance can take over. Simple failover without ZooKeeper or etcd.

How to Handle Failed Jobs

When a worker returns an error, Swig records the error and either retries the job or marks it as failed:

UPDATE swig_jobs
SET status = CASE
    WHEN attempts >= max_attempts THEN 'failed'
    ELSE 'pending'
  END,
  last_error = $2,
  last_error_at = NOW()
WHERE id = $1;

A Note on Delivery Semantics

It's tempting to say a job queue processes jobs exactly once. In distributed systems, that's a dangerous claim.

Consider this scenario:

  1. A worker sends an email

  2. The worker crashes before marking the job completed

  3. The job is retried

  4. The email might be sent again

The accurate description is that Swig provides atomic claiming and at-least-once processing. Because jobs can be retried, workers should be idempotent. Running the same operation more than once should produce the same result as running it once.

How to Abstract the Database Driver

Swig supports both pgx and database/sql through a driver interface:

type Driver interface {
    Exec(ctx context.Context, sql string, args ...interface{}) error
    Query(ctx context.Context, sql string, args ...interface{}) (Rows, error)
    QueryRow(ctx context.Context, sql string, args ...interface{}) Row
    WithTx(ctx context.Context, fn func(tx Transaction) error) error
    NewListener(ctx context.Context, channel string) (Listener, error)
    TryAdvisoryLock(ctx context.Context, lockID int64) (AdvisoryLock, bool, error)
}

The core queue code only depends on behavior, not a specific library. This is a common Go design: define the behavior your core package needs, write small adapters for concrete dependencies, and keep the core logic independent.

Conclusion

A PostgreSQL-backed queue isn't the right answer for every system. If you need massive event streaming, Kafka may be a better fit. If you need complex routing, RabbitMQ may be better.

But for many Go applications, PostgreSQL is already there. Swig shows how far you can get with a small Go API and a few PostgreSQL features:

  • Store jobs in a table

  • Claim jobs atomically with FOR UPDATE SKIP LOCKED

  • Wake workers with dedicated LISTEN/NOTIFY sessions

  • Coordinate leadership with advisory locks

  • Keep app data and jobs consistent with transactions

  • Manage worker lifecycles with goroutines and contexts

That combination makes a solid foundation for background processing and a great project for learning how Go and PostgreSQL work together in production systems. You can explore the full source code at github.com/glamboyosa/swig.

]]> <![CDATA[ Open Source Tools Every STEM Student Should Know About ]]> https://www.freecodecamp.org/news/open-source-tools-every-stem-student-should-know-about/ 6a27af485df8cf4edcb24d9b Tue, 09 Jun 2026 06:14:32 +0000 Technology has changed the way students learn science, mathematics, engineering, and computer science.

A decade ago, most STEM students depended on textbooks, calculators, and expensive licensed software. Today, open source tools have made advanced learning resources available to anyone with an internet connection.

Many of these tools are powerful enough for professional researchers and software engineers, yet simple enough for students who are just getting started. They help with coding, data analysis, mathematics, technical writing, visualization, collaboration, and project management.

In this article, we'll look at seven open source tools that can help STEM students study more effectively, build projects faster, and develop industry-ready technical skills.

What We'll Cover:

Why Open Source Tools Matter for STEM Students

Open source software is more than just free software. It gives students access to the underlying code, community support, and the freedom to experiment without restrictions.

This matters because STEM education is becoming increasingly hands-on. Employers expect students to understand practical workflows, not just theory. Learning how to use modern tools early can make the transition into internships and engineering roles much easier.

Open source ecosystems also evolve quickly. Students can explore real-world technologies used in research labs, startups, and large engineering organizations. Many of these environments also rely on open-source automation tools to simplify development workflows and improve collaboration across technical teams.

Jupyter Notebook for Interactive Learning

One of the most important tools for STEM students is Jupyter Notebook.

Jupyter Notebook

Jupyter Notebook allows users to combine code, mathematical equations, visualizations, and notes inside a single interactive document. This makes it extremely useful for subjects like data science, physics, statistics, and machine learning.

A student can write Python code, run calculations, and immediately visualize the output using graphs or tables. Instead of switching between multiple applications, everything exists in one place.

For example, a physics student can simulate motion equations, while a statistics student can analyze datasets directly inside the notebook.

Jupyter is widely used in universities and research institutions because it supports experimentation and iterative learning.

VS Code for Programming and Technical Projects

Visual Studio Code has become one of the most popular development environments in the world. Although it is developed by Microsoft, it's built on open source technologies and supports a massive extension ecosystem.

VS Code

For STEM students, VS Code is valuable because it supports nearly every major programming language. Whether you're learning Python, JavaScript, C++, or Rust, the editor provides debugging, syntax highlighting, terminal integration, and Git support in one interface.

Engineering students often work across multiple disciplines. A robotics student might write Python scripts, configure embedded systems, and document experiments all in the same environment.

VS Code also integrates well with Jupyter Notebook, making it an excellent all-in-one workspace for technical learning.

GeoGebra for Mathematics Visualization

Mathematics becomes easier when students can visualize concepts instead of memorizing formulas.

GeoGebra is an open source mathematics platform that helps students explore algebra, geometry, calculus, and statistics through interactive graphs and simulations.

GeoGebra

Students can manipulate equations dynamically and observe how graphs change in real time. This creates a much deeper understanding of mathematical relationships.

Interactive visualisation tools are especially useful for students preparing for advanced mathematics courses. Popular teaching platforms like Brighterly who are known as a great precalculus tutor, use graphing platforms like GeoGebra to better understand trigonometric functions, transformations, and polynomial behaviour. The platform is also useful for individual teachers who want to create interactive lessons instead of relying entirely on static diagrams.

Git and GitHub for Collaboration

Version control is one of the most important technical skills students can learn.

Git is an open source version control system that helps developers track changes in code and collaborate efficiently. It is widely used across software engineering, data science, and research projects.

Github

Students often lose work because they overwrite files or create confusing project versions. Git solves this problem by maintaining a complete history of changes.

When paired with GitHub, students can collaborate on projects, contribute to open source repositories, and build a public portfolio of technical work.

This is especially valuable for computer science students applying for internships or engineering roles. Recruiters frequently review GitHub profiles to evaluate coding ability and project experience.

Even students outside traditional software engineering fields benefit from Git. Researchers use it for reproducible experiments, while engineering teams use it to manage technical documentation and simulation code.

Blender for Scientific and Engineering Visualization

Most people associate Blender with animation and game design, but it's also a powerful tool for STEM applications.

Blender is an open source 3D modeling and rendering platform used in industries ranging from architecture to scientific visualization.

Blender

Engineering students can use Blender to create product prototypes, mechanical visualizations, and simulation renders. Biology students can build anatomical models, while physics students can visualize complex systems in three dimensions.

Visualization plays a major role in technical understanding. A well-designed 3D model can explain concepts that are difficult to communicate through text alone.

Blender also teaches valuable spatial reasoning and design skills that are increasingly useful in fields like robotics, manufacturing, and augmented reality.

OBS Studio for Recording and Presentations

Modern STEM learning is becoming more collaborative and content-driven.

Students now create tutorials, record presentations, explain coding projects, and participate in online learning communities. OBS Studio is an open source tool that allows users to record screens, stream presentations, and create technical demonstrations.

OBS Studio

This is particularly useful for students building portfolios or preparing project walkthroughs.

For example, a software engineering student can record a demo of a web application, while a mathematics student can create video explanations of problem-solving methods.

OBS Studio is lightweight, flexible, and widely used by educators, developers, and technical creators.

How Open Source Tools Build Career Skills

One of the biggest advantages of open source tools is that they mirror real industry workflows.

Students aren't just learning academic concepts. They're learning systems used in professional engineering environments.

A student who understands Git, VS Code, Jupyter, and collaborative development practices already has exposure to modern software engineering workflows. Similarly, students using Blender or GeoGebra are developing visualization and analytical skills that transfer into technical careers.

Open source communities also encourage experimentation. Students can inspect source code, contribute fixes, participate in discussions, and learn directly from experienced developers around the world.

This creates a more active learning process than simply consuming tutorials.

The Future of STEM Education

STEM education is shifting toward project-based and interdisciplinary learning.

Students are expected to solve problems, communicate ideas clearly, and adapt to rapidly evolving technologies. Open source tools make this possible by lowering financial barriers and giving students access to professional-grade software.

The rise of artificial intelligence, data science, and remote collaboration has also increased the importance of technical self-learning. Students who can independently explore tools and build projects will have a significant advantage in both academics and industry.

The good news is that modern open source ecosystems make this easier than ever before. A student with a laptop and internet connection can now access tools that were once available only to large universities or research organizations.

Final Thoughts

The best STEM students aren't always the ones with the most expensive hardware or software. Often, they're the ones who learn how to use accessible tools creatively and consistently.

Platforms like Jupyter Notebook, VS Code, GeoGebra, LibreOffice, Git, Blender, and OBS Studio provide a strong foundation for technical learning across many disciplines.

More importantly, these tools encourage curiosity, experimentation, and practical problem-solving. Those skills matter far beyond the classroom.

As STEM education continues to evolve, students who embrace open source technology will be better prepared for research, engineering, software development, and the increasingly interdisciplinary future of technical work.

]]> <![CDATA[ Database Version Control with Liquibase and Spring Boot ]]> https://www.freecodecamp.org/news/database-version-control-with-liquibase-and-spring-boot/ 6a277a781aef44b8099e978b Tue, 09 Jun 2026 02:29:12 +0000 Picture this familiar scenario: you're working on a new feature that requires a new database column. You open your local database client, write an ALTER TABLE statement, and execute it. Your code works perfectly. You commit the Java code, push it to the repository, and go grab a coffee.

A few hours later, a teammate pulls your branch, runs the application, and everything crashes.

"Hey," they ask across the room (or in a Slack channel), "did you change the database?"

You quickly realize you forgot to share the SQL script. You paste it into the chat. They run it. Everything works. Then, a week later, the deployment to the staging environment fails for the exact same reason. By the time this code reaches production, everyone is asking a variation of the same terrified question: "Which SQL script should I run?"

This situation is called schema drift. It happens when the state of your database diverges across different environments. Staging has one schema, production has another, and every developer's local machine is a unique snowflake of untested database modifications.

Managing database changes manually is a recipe for deployment headaches and team collaboration challenges. Application code is stateless and easy to replace. Databases are stateful. Databases have surprisingly good memories, and they rarely forget a bad migration.

Liquibase solves this problem by bringing version-control discipline to your database changes. Instead of passing around SQL files and hoping people remember to run them, you define your database changes in code. These changes travel with your application repository and execute automatically.

Here is a high-level look at how this architecture works:

Architecture diagram showing code flowing from a developer to a database via Git, Spring Boot, and Liquibase.

Think about the journey of a single database change. A developer commits their database migration alongside their Java code into Git. When the CI/CD pipeline (or a teammate) pulls that code, the Spring Boot application starts. But before the app fully boots up and accepts web traffic, Liquibase intercepts the process. It acts as a gatekeeper, connecting to the database and applying the required schema changes. This ensures the database exactly matches the code's expectations before a single user makes a request.

Why Database Version Control Matters

If you've spent any time working on team-based applications, you've probably seen a folder structure that looks exactly like this:

project-sql-scripts/
β”œβ”€β”€ create_employee_table.sql
β”œβ”€β”€ create_employee_table_final.sql
β”œβ”€β”€ create_employee_table_final_v2.sql
β”œβ”€β”€ add_email_column.sql
β”œβ”€β”€ latest.sql
└── definitely_latest_use_this_one.sql

The phrase "just run this SQL script manually" has launched many memorable incidents.

When you rely on manual database updates, you guarantee failure at scale. Onboarding a new developer becomes an archeological expedition to figure out how to build the local schema. Deployments become stressful events requiring a checklist of manual queries that must be run in a highly specific order.

Version-controlled database changes treat your schema as code. When your database changes live alongside your application logic, you gain several immediate benefits:

  • Consistency: Every environment (local, staging, production) applies the exact same changes in the exact same order.

  • Safety: You eliminate the human error of skipping a script or running an outdated query.

  • Visibility: You can look at a Git commit and see exactly how the Java code and the database schema changed together to support a new feature.

Git solved version control for code. Liquibase helps prevent databases from becoming the rebellious sibling.

What is Liquibase?

At its core, Liquibase is a database migration tool that tracks and applies schema changes in a predictable and repeatable way.

Instead of writing loose SQL scripts, you write "migrations" (also called changeSets). Liquibase reads these files, compares them against a tracking table inside your actual database, and figures out exactly what needs to be executed to bring the database up to date.

To use Liquibase effectively, you only need to understand a few conceptual terms:

  • changeLog: The master file. This is essentially a list that tells Liquibase which migration files to execute and in what order.

  • changeSet: A single, atomic change to your database. Creating a table is one changeSet. Adding a column is another.

  • Migration History: A table Liquibase automatically creates in your database (called DATABASECHANGELOG) to remember which changeSets have already been executed.

  • Checksums: A unique hash generated for every changeSet. Liquibase uses this to detect if someone secretly modified a file after it was already executed.

When you integrate Liquibase with Spring Boot, the migration process happens completely automatically during the application startup phase.

Sequence diagram of Spring Boot startup where Liquibase checks the tracking table, locks the database, runs migrations, and releases the lock before allowing HTTP traffic.

During startup, Liquibase takes control before your web server is allowed to receive HTTP traffic. It reaches into the database and checks the tracking table to see which migrations have already run. If it finds new migrations in your local files, it locks the database to prevent concurrent updates, executes the changes, records the new history, and finally releases the lock. Only after this entire process completes does Spring Boot finish booting up.

Because Liquibase runs before Spring Boot fully initializes the web server, your application will never serve traffic with an outdated database schema. If a migration fails, the application fails to start, protecting your system from entering a broken state.

Project Setup

Now that you understand the theory, let's build something real. We're going to build the database layer for an Employee Management API.

For this project we'll use:

  • Java 17+

  • Spring Boot 3.x

  • Maven

  • Liquibase

  • H2 Database

We're using H2 because it's an in-memory database that requires zero installation. You can run this project immediately without configuring Docker containers or installing database servers. But everything you learn here applies exactly the same way to PostgreSQL, MySQL, SQL Server, or Oracle.

If you're generating this project via Spring Initializr, select the following dependencies: Spring Web, Spring Data JPA, Liquibase Migration, and H2 Database.

In your pom.xml, you'll see the critical dependencies that make this work:

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>

    <dependency>
        <groupId>com.h2database</groupId>
        <artifactId>h2</artifactId>
        <scope>runtime</scope>
    </dependency>

    <dependency>
        <groupId>org.liquibase</groupId>
        <artifactId>liquibase-core</artifactId>
    </dependency>
</dependencies>

Next, configure Spring Boot to talk to H2 and find your Liquibase files. Open your src/main/resources/application.properties file and add the following:

# H2 Database Configuration
spring.datasource.url=jdbc:h2:file:./data/employeedb;DB_CLOSE_DELAY=-1
spring.datasource.driverClassName=org.h2.Driver
spring.datasource.username=sa
spring.datasource.password=

# Enable H2 Console to inspect the database in your browser
spring.h2.console.enabled=true
spring.h2.console.path=/h2-console

# Liquibase Configuration
spring.liquibase.change-log=classpath:db/changelog/db.changelog-master.xml

That last line is the most important. It tells Spring Boot exactly where to find the "master list" of your database changes.

Note: We're using a file-based H2 database instead of an in-memory database. The problem with an in-memory database is that it completely wipes itself clean every time you restart Spring Boot.

While Liquibase will happily rebuild the schema from scratch on every boot, a file-based database is much better for this tutorial (and for real-world local development). With a file-based database, your data, and more importantly, your Liquibase history, will actually persist between application restarts.

Understanding Core Liquibase Concepts

Before we write our first table, we need to understand how Liquibase organizes files. Liquibase uses a hierarchical structure.

Think of it like a book. The changeLog is the table of contents, and the changeSets are the actual chapters.

  1. The Master ChangeLog: This is the entry point. It rarely contains actual database changes. Instead, its only job is to include other files in a specific order.

  2. Child ChangeLogs: These group related changes together.

  3. ChangeSets: These are the actual, atomic database commands (like creating a table or adding a column).

Here's a visual breakdown of how this hierarchy works in a real Spring Boot project:

File structure diagram showing a master changelog XML file pointing to three child migration files in strict chronological order.

Liquibase organizes migrations hierarchically. You maintain a single master file that acts as a table of contents. This master file rarely holds actual SQL commands. Instead, it explicitly includes child XML files in a strict execution order. Each of those child files (like 01-create-employees.xml) contains one or more individual database commands, which Liquibase calls changeSets.

A changeSet is uniquely identified by three things:

  • id: A unique string (often a number or a Jira ticket ID).

  • author: The person who wrote the migration.

  • file path: Where the file is located.

When Liquibase runs, it looks at a changeSet, calculates a cryptographic hash of its contents (a checksum), and records the id, author, and checksum in the database. If it sees that exact combination of id, author, and file path in the database again on the next startup, it skips it.

Create the Initial Employee Schema (Version 1)

Let's write our first version. We need a table to store employees.

First, create the master file at src/main/resources/db/changelog/db.changelog-master.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.20.xsd">

    <include file="db/changelog/changes/01-create-employees.xml"/>

</databaseChangeLog>

Next, create the actual migration file at src/main/resources/db/changelog/changes/01-create-employees.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.20.xsd">

    <changeSet id="1" author="ashutoshkrris">
        <createTable tableName="employees">
            <column name="id" type="BIGINT" autoIncrement="true">
                <constraints primaryKey="true" nullable="false"/>
            </column>
            <column name="first_name" type="VARCHAR(50)">
                <constraints nullable="false"/>
            </column>
            <column name="last_name" type="VARCHAR(50)">
                <constraints nullable="false"/>
            </column>
        </createTable>
    </changeSet>

</databaseChangeLog>

Let's look at what we just did. We defined a changeSet with an id of "1" and an author of "ashutoshkrris". Inside, we used Liquibase's XML syntax to define a table.

Why use XML instead of plain SQL? Because Liquibase is database-agnostic. This exact XML will generate the correct auto-increment syntax for PostgreSQL (SERIAL), MySQL (AUTO_INCREMENT), or Oracle (IDENTITY). You define the structure, and Liquibase translates it to the specific database dialect.

Now, run your Spring Boot application. Watch your terminal output. You'll see logs similar to this:

Terminal Logs for Liquibase Startup

Liquibase realized the database was empty. It automatically created its tracking table (DATABASECHANGELOG), read our changeSet, executed the table creation, and recorded the event.

If you restart the application right now, Liquibase will run again. But this time, it'll check the DATABASECHANGELOG table, see that id="1" and author="ashutoshkrris" has already been executed, and silently skip it. Your database is now safely version-controlled.

What Just Happened?

Up to this point, Liquibase might feel a bit like magic. You dropped an XML file into a folder, started Spring Boot, and your database schema transformed.

But understanding how Liquibase actually works under the hood is critical. If you understand the startup sequence, you'll know exactly how to debug deployments when things eventually go wrong.

When your Spring Boot application starts, it doesn't immediately begin accepting web requests. First, it initializes its internal components. When it creates the Liquibase component, the migration process begins.

Here's exactly what happens during that startup phase:

Detailed sequence diagram showing Liquibase checking the lock table, acquiring the lock, running unexecuted migrations, and releasing the lock before Tomcat starts.

Let's trace the exact sequence. When Spring Boot initializes Liquibase, the very first thing the tool does is query the lock table to ensure no other application instance is currently migrating the database. If the coast is clear, it claims the lock. It then calculates cryptographic checksums for your local XML files, compares them against the database history, executes any missing changes, and logs them. Finally, it releases the lock so the Tomcat web server can safely start.

This sequence guarantees that your application will never serve a user request before the database schema is completely ready to handle it.

Inspecting the Database: Liquibase Metadata Tables

Let's look at what this history and locking actually looks like inside the database itself. Since we configured the H2 Console earlier, we can inspect the raw tables.

While your Spring Boot application is running, open your browser and navigate to http://localhost:8080/h2-console. Connect using the JDBC URL jdbc:h2:file:./data/employeedb with the username sa and a blank password.

Inside, you'll see your employees table. You'll also see two extra tables created automatically by Liquibase: DATABASECHANGELOG and DATABASECHANGELOGLOCK.

The DATABASECHANGELOG Table

This table is the brain of your migration strategy. It acts as the permanent ledger of every database change ever applied to this environment.

If you run SELECT * FROM DATABASECHANGELOG;, you'll see output that looks like this:

ID AUTHOR FILENAME DATEEXECUTED ORDEREXECUTED EXECTYPE MD5SUM DESCRIPTION COMMENTS TAG LIQUIBASE CONTEXTS LABELS DEPLOYMENT_ID
1 ashutoshkrris db/changelog/changes/01-create-employees.xml 2026-05-30 13:11:35.937919 1 EXECUTED 9:66e7dcffb2b1902a4e9f01670cb5f192 createTable tableName=employees null 4.31.1 null null 0126894849

Let's break down the most important columns:

  • ID, AUTHOR, FILENAME: These three columns form a composite key. Together, they uniquely identify a single migration.

  • DATEEXECUTED & ORDEREXECUTED: Tells you exactly when a script ran and in what sequence.

  • MD5SUM: This is the cryptographic hash of your XML file. When Liquibase starts, it hashes your local XML file and compares it to this column. If you secretly edit a file after it's been executed, this hash won't match, and Liquibase will crash the startup to protect your database.

  • EXECTYPE: Most of the time, this simply says EXECUTED. But it provides a crucial audit trail: if you use Liquibase commands to intentionally skip a migration but record it as finished, you'll see MARK_RAN. If a migration was skipped because its preconditions failed, you'll see SKIPPED.

  • TAG: Think of this as a Git tag for your database schema. Before a major, high-risk deployment, you can configure Liquibase to "tag" the current state of the database (for example, v1.4.0). If the deployment fails catastrophically, you can trigger a rollback command telling Liquibase to undo every change applied after the v1.4.0 tag.

  • CONTEXTS: This is how you manage environment-specific changes. By adding a context attribute to your changeSet (for example, <changeSet id="7" author="ashutoshkrris" context="dev, qa">), that migration will only execute if Spring Boot passes "dev" or "qa" to Liquibase on startup. Production will safely ignore it.

  • LABELS: While Contexts target environments, Labels target categories of work. You can label a changeSet with a Jira ticket number (issue-842) or a release train (Q3-release). This allows advanced teams to selectively execute or roll back specific subsets of features without affecting the rest of the database.

The DATABASECHANGELOGLOCK Table

This table is tiny, but it plays a massive role in modern deployments.

If you run SELECT * FROM DATABASECHANGELOGLOCK;, you'll see a single row:

ID LOCKED LOCKGRANTED LOCKEDBY
1 FALSE null null

Imagine you're deploying your Spring Boot application to a Kubernetes cluster. You tell Kubernetes to spin up three identical instances simultaneously. All three instances connect to the exact same database.

If all three instances try to run the CREATE TABLE migration at the exact same millisecond, your database will throw concurrency errors. The lock table prevents this. The very first instance to reach the database sets LOCKED to TRUE. The other two instances check the table, see the lock, and politely wait.

Practical Troubleshooting Tip: Sometimes, a deployment fails catastrophically mid-migration (perhaps the server lost power). When this happens, Liquibase might die before it can set LOCKED back to FALSE.

The next time you start the application, the logs will hang indefinitely, repeating: Waiting for changelog lock....

If you're absolutely certain no other applications are currently running migrations, you can manually fix this by running a simple SQL command in your database client:

UPDATE DATABASECHANGELOGLOCK SET LOCKED = FALSE;

This forces the lock open, allowing your application to resume.

Evolving the Employee API

Software is never finished. Two weeks after your successful Version 1 deployment, the business team comes back with new requirements.

Because you now understand how Liquibase tracks history, evolving the database is simple. You just append new files to your master list.

Version 2: Adding an Email Field

The HR team needs to contact employees. You need an email column.

Create a new file at src/main/resources/db/changelog/changes/02-add-employee-email.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.20.xsd">

    <changeSet id="2" author="ashutoshkrris">
        <addColumn tableName="employees">
            <column name="email" type="VARCHAR(100)">
                <constraints nullable="false" unique="true"/>
            </column>
        </addColumn>
    </changeSet>

</databaseChangeLog>

Add this to your db.changelog-master.xml file immediately below your first include:

<include file="db/changelog/changes/02-add-employee-email.xml"/>

When you restart the application, Liquibase checks the DATABASECHANGELOG table. It sees that id="1" is already there, so it skips it. It sees id="2" is missing, so it executes it and adds a new row to the tracking table.

Version 3: Adding Departments Support

The company is growing. Employees now belong to departments. You need a departments table and a foreign key constraint linking the two.

Create 03-add-departments.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.20.xsd">

    <changeSet id="3" author="ashutoshkrris">
        <createTable tableName="departments">
            <column name="id" type="BIGINT" autoIncrement="true">
                <constraints primaryKey="true" nullable="false"/>
            </column>
            <column name="name" type="VARCHAR(50)">
                <constraints nullable="false" unique="true"/>
            </column>
        </createTable>
    </changeSet>

    <changeSet id="4" author="ashutoshkrris">
        <addColumn tableName="employees">
            <column name="department_id" type="BIGINT"/>
        </addColumn>
        <addForeignKeyConstraint baseTableName="employees"
                                 baseColumnNames="department_id"
                                 constraintName="fk_employee_department"
                                 referencedTableName="departments"
                                 referencedColumnNames="id"/>
    </changeSet>

</databaseChangeLog>

Notice that we used two separate changeSets in one file. This is a best practice. Each changeSet represents one logical operation. If the foreign key creation (id="4") fails, the department table creation (id="3") will still be recorded as successful, and only id="4" will roll back.

Version 4 & 5: Employee Status and Performance Indexes

Finally, HR wants to track active versus inactive staff, and the database team noticed that searching by last name is getting slow.

Create 04-status-and-indexes.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.20.xsd">

    <changeSet id="3" author="ashutoshkrris">
        <createTable tableName="departments">
            <column name="id" type="BIGINT" autoIncrement="true">
                <constraints primaryKey="true" nullable="false"/>
            </column>
            <column name="name" type="VARCHAR(50)">
                <constraints nullable="false" unique="true"/>
            </column>
        </createTable>
    </changeSet>

    <changeSet id="4" author="ashutoshkrris">
        <addColumn tableName="employees">
            <column name="department_id" type="BIGINT"/>
        </addColumn>
        <addForeignKeyConstraint baseTableName="employees"
                                 baseColumnNames="department_id"
                                 constraintName="fk_employee_department"
                                 referencedTableName="departments"
                                 referencedColumnNames="id"/>
    </changeSet>

</databaseChangeLog>

Remember to add all new files to your db.changelog-master.xml. The order of your include statements is the exact order Liquibase will execute them.

The Golden Rule: Never Modify Executed ChangeSets

Eventually, a developer on your team will look at your 01-create-employees.xml file and notice a mistake. Perhaps they spot a typo in a column name, or perhaps they realize a column is missing a strict non-null constraint.

Their instinct, based on years of writing standard Java code, will be to open that XML file, fix the mistake, save the file, and restart the application.

Let's actually do this and see what happens.

Open your src/main/resources/db/changelog/changes/01-create-employees.xml file. Change the first_name column to given_name:

<column name="given_name" type="VARCHAR(50)">
    <constraints nullable="false"/>
</column>

Save the file and restart your Spring Boot application.

Instead of a smooth startup, your application will instantly crash, and your terminal will vomit a massive stack trace. Look closely at the top of the error logs. You should see this exact message:

Caused by: liquibase.exception.ValidationFailedException: Validation Failed:
     1 changesets check sum
          db/changelog/changes/01-create-employees.xml::1::ashutoshkrris was: 9:66e7dcffb2b1902a4e9f01670cb5f192 but is now: 9:2bd3ef21343d3b5c9448cc50bc35deef

Here's why this happens. Once a changeSet runs against an environment, it becomes immutable history. You can't change the past.

When Liquibase starts up, it calculates a cryptographic hash (an MD5 checksum) of your local XML file. It then queries the DATABASECHANGELOG table and compares the freshly calculated hash against the hash that was recorded when the file originally executed.

If you change even a single character in a file that has already been executed, the hash changes. Liquibase detects the tampering and refuses to start. It does this to protect your data. If your XML code says a column is named first_name but the database was originally built using fist_name, your Spring Data JPA repositories are going to fail anyway.

How to Fix It (The Right Way)

If you made this mistake locally, you might be tempted to go into your database, delete the row from the DATABASECHANGELOG table, and try again. Don't do this. If this code reaches staging or production, you can't manually delete rows on production servers.

The correct way to fix a schema mistake is to roll forward.

First, undo your change in 01-create-employees.xml so the hash matches the database again. Then, write a brand new changeSet to apply the fix:

<changeSet id="7" author="ashutosh">
    <renameColumn tableName="employees" 
                  oldColumnName="first_name" 
                  newColumnName="given_name" 
                  columnDataType="VARCHAR(50)"/>
</changeSet>

Include it in your master changelog, restart the application, and the database will safely evolve to the correct state.

Working with Seed Data

Sometimes, a schema change requires initial data to be useful.

For example, in Version 3, we created a departments table. Right now, that table is completely empty. When a new developer clones the repository and spins up the project locally, they have to manually write SQL INSERT statements just to test the API.

We can automate this by making baseline data insertion part of our migration strategy.

Create a new file at src/main/resources/db/changelog/changes/05-seed-departments.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.20.xsd">

    <changeSet id="8" author="ashutoshkrris">
        <insert tableName="departments">
            <column name="name" value="Engineering"/>
        </insert>
        <insert tableName="departments">
            <column name="name" value="Human Resources"/>
        </insert>
        <insert tableName="departments">
            <column name="name" value="Finance"/>
        </insert>
    </changeSet>

</databaseChangeLog>

Add the include statement to your db.changelog-master.xml file. When you restart the application, Liquibase will insert these rows. Your API is now instantly usable out of the box.

The Danger of Data Migrations

While seeding data is powerful, it requires discipline. Here is a practical engineering rule of thumb:

Do use Liquibase for:

  • Static lookup tables (status codes, country lists, default departments).

  • System configuration flags required for the application to boot.

Do NOT use Liquibase for:

  • Generating thousands of fake users for testing.

  • Migrating massive amounts of transactional data (for example, moving 5 million records from one table to another).

Large data migrations can lock up database tables for hours. If you lock a core table during a deployment, your application will experience a massive outage. Keep your changeSets focused on schema structure and essential baseline data. Use dedicated scripts or background jobs for heavy data manipulation.

Rollbacks

In a perfect world, code always works. In reality, you'll eventually deploy a database change that breaks a critical production query or corrupts data. When this happens, you need a way to hit the undo button.

Liquibase supports rollbacks, but you have to understand how it interprets them.

Automatic vs. Explicit Rollbacks

Many Liquibase commands are automatically reversible. For example, if you write a changeSet to <createTable> or <addColumn>, Liquibase implicitly knows that the opposite of adding a column is dropping a column. You don't have to tell it how to undo these actions.

But some operations are inherently destructive or ambiguous. If you use custom <sql> tags, or if you use <dropTable>, Liquibase has no idea how to put the data back. In these cases, you must provide explicit rollback instructions.

Let's simulate a scenario where we add a temporary access code column, but we want to ensure we know exactly how to remove it safely.

Create 06-temporary-access.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.20.xsd">

    <changeSet id="9" author="ashutosh">
        <addColumn tableName="employees">
            <column name="temp_access_code" type="VARCHAR(10)"/>
        </addColumn>
        
        <rollback>
            <dropColumn tableName="employees" columnName="temp_access_code"/>
        </rollback>
    </changeSet>

</databaseChangeLog>

Add this to your master file and run the application. The column is added.

If you were deploying this via a CI/CD pipeline and the deployment failed, you could trigger a Liquibase Maven command to roll back by a specific number of steps (for example, mvn liquibase:rollback -Dliquibase.rollbackCount=1), or roll back to a specific tag we discussed earlier.

The Reality Check on Rollbacks

While it's important to know how rollbacks work, here's a practical reality from the trenches of backend engineering: Rollbacks are often discussed but rarely executed cleanly in production.

Dropping a column is mathematically easy. Recovering the customer data that was written to that column during the 15 minutes the bad code was live is incredibly difficult.

Because of this, modern engineering teams often prefer a "roll forward" strategy. If a migration causes an issue, instead of running a scary database rollback command, they quickly write a new changeSet that fixes the issue (for example, adding a missing index or relaxing a constraint) and deploy the application again.

It's highly recommended to design your database changes to be additive and non-destructive to avoid needing complex rollbacks in the first place.

Common Beginner Mistakes

Adopting database version control is a massive step forward for any engineering team, but it comes with a learning curve. When developers transition from writing loose SQL scripts to using Liquibase, they tend to fall into a few predictable traps.

Here are the most common beginner mistakes and exactly how to avoid them.

1. The "Mega" ChangeSet

When starting out, it's tempting to dump your entire initial schema into a single XML file under a single changeSet. You might put 15 createTable statements and 20 addForeignKeyConstraint statements into id="1".

This is a terrible idea for one simple reason: transaction failure.

If your database engine fails on table number 14 (perhaps due to a syntax error), what happens to the first 13 tables? Some database engines support transactional DDL (Data Definition Language), meaning it will roll back all 13 tables automatically. But many databases do not.

If it fails halfway through, your database is now in a fractured state. Liquibase didn't record id="1" as successful, so the next time you start the app, it will try to create all 15 tables again. It will immediately crash because table 1 already exists.

The Fix: Stick to the rule of "one logical operation per changeSet." If you're creating three tables, write three separate changeSets. If one fails, the successful ones are permanently recorded, and you only have to fix the broken one.

2. Manual Database Tweaking (The Phantom Menace)

This is the most dangerous habit to break. A developer spots a missing index in production. Instead of writing a Liquibase migration, going through code review, and deploying, they log directly into the production database and run CREATE INDEX manually to save time.

A week later, another developer writes a proper Liquibase migration to create that exact same index and deploys it. The application crashes on startup. Liquibase tries to execute the CREATE INDEX command, but the database throws an error saying the index already exists.

When you adopt Liquibase, you must accept a fundamental rule: Liquibase is the absolute source of truth for your schema. Human hands should never touch the database structure directly.

The Fix: If someone accidentally does this, you have two options to fix the deployment pipeline. You can manually drop the index from the database so Liquibase can recreate it properly, or you can use the <preConditions> tag in Liquibase to check if the index exists before trying to create it.

3. Ignoring the "From Scratch" Build

When you work on a project for months, your local database accumulates a lot of history. You write migrations assuming certain tables or test data already exist.

Then, a new developer joins the team. They pull the code, spin up an empty database, start Spring Boot, and the migrations crash halfway through.

This happens because the migrations rely on an assumed state (like expecting a specific row to exist before creating a foreign key) rather than a guaranteed state.

The Fix: You should regularly test your migrations against a completely blank database. If you're using Docker, tear down your database container and rebuild it. If you're using a file-based H2 database like we set up earlier, simply delete the ./data/employeedb.mv.db file from your project folder and restart Spring Boot. If the application can't boot successfully from a completely empty state, your migration history is broken.

4. Hardcoding Environment Details

Beginners sometimes hardcode environment-specific details directly into their XML files. For example, they might hardcode a specific schema name (schemaName="dev_schema") or grant permissions to a specific local user (GRANT ALL ON employees TO my_local_user).

When this code goes to staging, the staging database uses a different schema name, and the deployment fails.

The Fix: Keep your migrations abstract. Let Spring Boot handle the connection details via application.properties. If you absolutely must use dynamic values inside your Liquibase files, use property substitution. You can define variables in Liquibase and pass them in from Spring Boot during startup.

5. Messing Up Migration Ordering

Liquibase executes files in the exact order they're listed in your db.changelog-master.xml file.

If developer A creates the departments table in a branch, and developer B creates a foreign key linking to departments in another branch, whoever merges their code first dictates the order. If developer B's code gets included in the master file before developer A's code, Liquibase will try to create the foreign key before the target table exists.

The Fix: The master changelog is the ultimate chokepoint for database changes. During code reviews, always verify that the <include> statements are ordered chronologically and that dependencies make sense.

Liquibase vs Flyway vs Manual SQL Scripts

When you decide to implement database version control, you'll immediately face a choice. Liquibase isn't the only tool in the Java ecosystem. The three most common approaches to managing schema evolution are Liquibase, Flyway, and manual SQL scripts.

You should understand the practical tradeoffs of each so you can choose the right tool for your specific team and project.

1. Manual SQL Scripts (The Baseline)

This is the default approach for most beginners. You write a script.sql file and execute it directly against the database using a tool like DBeaver, pgAdmin, or DataGrip.

  • Strengths: There is zero setup required. You have total control over the exact syntax, and every backend developer already knows how to write SQL.

  • Weaknesses: There's absolutely no execution tracking. This approach practically guarantees schema drift across environments. Deployments become stressful because they rely on humans remembering to execute the right scripts in the exact right order.

  • The Verdict: Manual scripts are perfectly fine for solo weekend projects or rapid prototyping where you don't care if the database gets destroyed. But they become a massive liability the moment a second developer joins the team or a staging environment is created.

2. Flyway (The SQL Purist)

Flyway is the most popular alternative to Liquibase. Instead of using XML or YAML abstractions, Flyway embraces raw SQL. You write pure SQL files with a strict naming convention (for example, V1__Create_employee_table.sql).

  • Strengths: There's no new syntax to learn. If you know SQL, you already know how to use Flyway. It's incredibly fast to set up, highly opinionated, and integrates flawlessly with Spring Boot.

  • Weaknesses: Because you write raw SQL, your migrations are intimately tied to your specific database dialect. If you write Flyway scripts for MySQL and later decide to migrate the project to PostgreSQL, you have to manually rewrite your migration history. Furthermore, seamless automated rollbacks are a paid feature in Flyway's commercial tier.

  • The Verdict: Flyway is excellent for teams that are highly skilled in SQL, are permanently committed to a single database vendor, and prefer strict conventions over flexible configurations.

3. Liquibase (The Abstraction Layer)

As we have seen throughout this tutorial, Liquibase takes a different approach by abstracting database changes into XML, YAML, or JSON.

  1. Strengths: It's truly database-agnostic. You define the logical structure, and Liquibase automatically translates that into the correct SQL dialect for H2, PostgreSQL, or Oracle. It supports powerful automatic rollbacks, preconditions, contexts, and deployment labels out of the box for free.

  2. Weaknesses: It has a steeper learning curve than Flyway. The XML syntax is undeniably verbose and can feel heavy for very simple, single-table applications.

  3. The Verdict: Liquibase shines in complex applications, multi-tenant systems, projects that support multiple database vendors, and enterprise environments that require fine-grained control over CI/CD deployment pipelines.

Liquibase Best Practices

Now that you understand the mechanics of Liquibase, you need to know how to use it in a professional environment. Writing a migration that works on your local machine is only half the battle. Writing a migration that your entire team can safely deploy to production requires discipline.

Here are the engineering best practices you should adopt when managing database changes.

1. One Logical Change Per ChangeSet (The Atomic Rule)

We discussed this in the common mistakes section, but it's important enough to repeat. Never bundle a table creation, an index creation, and a data insertion into a single changeSet.

If you're adding a salary column and an idx_employee_salary index, put them in two separate changeSets within the same file. This ensures that if the index creation fails, the column creation is still safely recorded, and you don't end up in a fractured database state.

2. Meaningful File Organization and Naming

Don't name your files update1.xml or new_changes.xml. Your file names should tell a story about how your database evolved.

Adopt a strict prefix system. In our project, we used 01-create-employees.xml and 02-add-employee-email.xml. In a real team, you might use Jira ticket numbers or release versions (for example, v1.2.0_ticket-482_add_email.xml). Whatever convention you choose, enforce it rigorously during code reviews.

3. Treat Database Changes Like Application Code

Database migrations belong in source control right next to your Java code. They should be reviewed with the exact same level of scrutiny.

When reviewing a pull request that includes a Liquibase file, engineers should ask:

  • Does this column need an index?

  • Is this a destructive change (like renaming a column) that will break the currently running application?

  • Did the author include explicit rollback instructions for custom SQL?

4. Integrate Migrations into CI/CD

Human hands should never run database migrations against a production server. Your deployment pipeline should handle this automatically.

When you merge code into your main branch, your CI/CD pipeline (like GitHub Actions or GitLab CI) should build your Spring Boot application and deploy it. Because we bundled Liquibase into our Spring Boot startup sequence, the application will automatically migrate the production database before it starts accepting web traffic.

Here's what a safe, automated deployment pipeline looks like:

CI/CD pipeline architecture showing code moving from Git to testing, deploying to a staging environment where Liquibase runs, and then promoting to production.

In a mature deployment pipeline, human hands never touch the production database. When you merge a pull request, the CI/CD pipeline builds the code and runs unit tests. It deploys the Spring Boot application to a staging environment, where Liquibase automatically acquires a lock and runs the migrations during startup. Once validated, that exact same artifact is promoted to production, triggering the identical automated migration process.

5. Never Fix Forward by Deleting History

If a migration fails in an upper environment (like staging or production), never log into the database to delete the DATABASECHANGELOG row so you can try again.

You must respect the immutability of the changelog. If you made a mistake, write a new changeSet that drops the broken table or fixes the data type, and push it through your Git workflow just like you would a Java bug fix.

Final Thoughts

Managing database schema changes doesn't have to be a source of anxiety.

By treating your database schema as code, you eliminate the chaos of manual SQL scripts. You prevent the dreaded "schema drift" where every developer's local machine behaves differently. Most importantly, you make your deployments predictable and boring (which is exactly what you want deployments to be).

In this tutorial, you built a practical Spring Boot application from scratch. You learned how Liquibase intercepts the application startup, locks the database, calculates cryptographic checksums, and safely applies incremental changes. You evolved a single table into a relational schema, added seed data, and learned how to avoid the most common traps beginners fall into.

The next time you start a Spring Boot project, don't reach for a manual SQL client. Add the Liquibase dependency, create your master changelog, and start version controlling your database from day one. Your future self (and your team) will thank you.

]]> <![CDATA[ How to Optimize Enterprise Knowledge Graphs for Scalable Digital Product Platforms ]]> https://www.freecodecamp.org/news/how-to-optimize-enterprise-knowledge-graphs-for-scalable-digital-product-platforms/ 6a26427ed198e572e0517866 Mon, 08 Jun 2026 04:18:06 +0000 Enterprises are building more and more digital products that depend on real time intelligence. This means that being able to connect, contextualize, and reason over data has become a core capability.

Recommendation systems, fraud detection engines, personalization platforms, and enterprise search solutions all rely on integrating data from multiple systems while preserving context and relationships.

Enterprise Knowledge Graphs (EKGs) have emerged as a foundational architecture for addressing this challenge. By modeling enterprise data as entities and relationships, EKGs enable richer semantics, improved data discoverability, and more intelligent downstream decision making.

While the conceptual benefits of knowledge graphs are well understood, scaling them to production grade digital platforms remains complex. Graph systems that perform well at small or medium scale often struggle under high ingestion rates, complex traversal queries, and strict latency requirements.

This article outlines some practical, field tested strategies for optimizing enterprise knowledge graphs for real world scalability. Rather than presenting purely theoretical models, we'll focus on architectural patterns, operational lessons, and performance insights from large scale enterprise deployments.

What We'll Cover:

Prerequisites

This is an architectural guide intended for data engineers, platform architects, and developers managing production-grade graph systems. To get the most out of this article, you should have the following:

Conceptual Knowledge

  • A solid understanding of Enterprise Knowledge Graphs (EKGs) and the fundamental differences between RDF triple stores and Labeled Property Graphs (LPGs).

  • Familiarity with distributed systems concepts, including data partitioning, semantic inference, and event-driven architectures.

Technical Background

  • Experience working with real-time data integration pipelines (such as CDC, Kafka, or Pulsar).

  • Familiarity with database observability, query execution planning, and general performance optimization techniques at scale.

Understanding the Enterprise Knowledge Graph (EKG)

Before exploring how to scale these systems, it's helpful to understand exactly what a knowledge graph is and how it organizes information.

At its core, a knowledge graph is a data model that represents real-world entities and the complex relationships between them. Unlike traditional relational databases that lock data into rigid, disconnected tables, knowledge graphs store data as a flexible, interconnected network.

A knowledge graph is built on three fundamental components:

  • Nodes (Entities): The distinct objects, concepts, or people in your data ecosystem (for example a Customer, a Product, a Location).

  • Edges (Relationships): The lines connecting the nodes that define how they interact (for example "PURCHASED," "LOCATED_IN," "MANUFACTURED_BY").

Properties: The descriptive metadata attached to nodes or edges (for example, a customer's signup date, or the price of a product).

Our Running Example: The Global Electronics Supply Chain Graph

To ground these concepts, we'll use a unified example throughout this article: an enterprise graph for a global electronics manufacturer managing product data, suppliers, and manufacturing compliance.

816a8985-93c2-4e0e-a085-87d3dd4e6fc7

  • Nodes (Entities): Customer (Alice), Product (NeoPhone 15), Component (MX-200 Chip), Supplier (MaxSemi), and Region (EU).

  • Edges (Relationships): PURCHASED, PART_OF, SUPPLIES, and LOCATED_IN.

  • Properties: The NeoPhone 15 node has properties like price: 999 and sku: "NP15-01". The PURCHASED edge has a property of timestamp: 2026-06-03.

Imagine you're building the data foundation for a retail recommendation engine. To build the graph, you move through a few distinct phases:

  1. Establish ontology: First, you define the blueprint – the rules dictating what kinds of entities exist and how they are allowed to interact.

  2. Define the nodes: You integrate data to generate specific entity nodes, such as a Customer node for "Alice," a Product node for "Noise-Canceling Headphones," and a Brand node for "TechAudio."

  3. Map the edges: You connect these nodes based on user actions and inventory data. Alice VIEWED the Headphones. The Headphones are MANUFACTURED_BY TechAudio.

Why does this matter? Because the data is natively structured as a relationship network, the system can rapidly execute context-rich queries.

If you want to know what else Alice might buy, you don't need to write a heavy, expensive SQL query that joins millions of rows across five different tables. Instead, the graph simply "walks" the pathways you've already built. It traverses from Alice, across the VIEWED edge to the Headphones, across the MANUFACTURED_BY edge to TechAudio, and can instantly return other products connected to that same brand.

By prioritizing the relationships between data points as much as the data points themselves, EKGs provide the contextual intelligence required for modern digital products.

Why Scalability Becomes the Core Challenge

Most enterprise knowledge graph initiatives begin with a limited scope, integrating a small number of datasets, enabling semantic search, or improving reporting accuracy. Early-stage deployments often succeed using a single graph database or RDF store.

Scalability challenges emerge when EKGs become production critical infrastructure, particularly when supporting customer facing or latency-sensitive applications. At this stage, multiple pressures converge:

  1. Rapid data growth as more systems and entities are integrated

  2. Continuous ingestion from streaming pipelines and transactional systems

  3. Increasing query complexity, including multi hop traversals

  4. Strict response time requirements, often under tens of milliseconds

  5. Inference overhead introduced by ontologies and reasoning engines

Simply adding hardware or scaling nodes horizontally rarely resolves these issues. Performance degradation often results from architectural mismatches between graph workloads and system design.

Moving Beyond a Single Graph Store: Hybrid Architectures

The Limits of Monolithic Graph Deployments

RDF triple stores offer strong semantic expressiveness and standards compliance but may struggle with high volume transactional updates or deep real time traversals. Conversely, labeled property graph (LPG) databases often provide efficient traversal performance but lack native semantic reasoning capabilities.

Attempting to consolidate semantic modeling, inference, operational queries, and analytics into a single system frequently results in trade offs that affect performance, cost, or maintainability.

A Pragmatic Hybrid Model

A hybrid or polyglot architecture distributes responsibilities across systems optimized for specific workloads:

  1. Semantic layer (RDF / OWL): Ontology management, schema governance, reasoning workflows.

  2. Operational graph layer (LPG): Real time traversals, recommendation engines, application queries.

  3. Analytical stores: Aggregations, reporting, and historical analysis.

To maintain consistency between the semantic layer (RDF/OWL) and the operational graph layer (LPG), many teams implement synchronization strategies like Change Data Capture (CDC) and event driven pipelines.

In this approach, updates in one layer are captured as events and propagated to the other layer in near real time using streaming platforms such as Kafka or Pulsar. For example, updates in the operational graph can trigger semantic updates, ensuring that ontologies and relationships remain aligned.

Some systems also use dual write patterns or scheduled reconciliation jobs to detect and resolve inconsistencies. In practice, event-driven synchronization combined with periodic validation provides a balance between real time accuracy and system reliability.

This separation isolates performance critical paths while preserving semantic richness where it adds value.

In production environments, hybrid architectures consistently demonstrate improved query latency and operational flexibility compared to monolithic graph deployments, particularly for traversal-heavy workloads. Some teams have also reported latency reductions of 30–60% when separating traversal-heavy workloads into LPG layers, compared to monolithic graph deployments.

This improvement is primarily due to reduced query complexity and optimized storage for specific access patterns.

In Practice: Splitting the Supply Chain Graph

In a production-grade digital platform, a single database engine struggles to handle both semantic governance and high-speed operational queries on this data simultaneously.

Here is how the hybrid model divides the labor:

  • The Semantic layer (RDF/OWL): Manages strict ontological classification and compliance rules. For example, it defines the rule: β€œIf a Component is supplied by an entity in a country under a trade embargo, the final Product inherits a 'High Risk' compliance flag.”

  • The Operational Layer (LPG): Optimized for fast, multi-hop traversals required by customer-facing apps. When Alice views the NeoPhone 15 on a mobile app, the system queries a Labeled Property Graph (like Neo4j) using a language like Cypher to instantly traverse from the product to its components for a real-time availability check:

MATCH (p:Product {id: 'NeoPhone15'})-[:HAS_COMPONENT]->(c:Component)
RETURN c.name, c.stock_level

Partitioning for Scale: Reducing Distributed Traversal Costs

As enterprise knowledge graphs outgrow single node capacity, distributed execution becomes necessary. Partitioning strategy then becomes a critical performance factor.

Why Default Partitioning Often Fails

Many graph systems use hash-based or random partitioning to distribute data evenly across nodes. While this approach balances storage, it often fragments highly connected subgraphs. Even moderately complex traversals may then require excessive cross-node communication, increasing latency and reducing throughput.

Topology-Aware Partitioning

Topology-aware partitioning colocates frequently connected entities to minimize network hops during traversal. Common approaches include:

  1. Partitioning by business domain (for example, customers, products, organizations).

  2. Community detection based clustering.

  3. Partitioning informed by observed query patterns.

In practice, teams can achieve topology-aware partitioning by first analyzing query patterns and identifying frequently traversed relationships. Based on this analysis, related entities are co-located within the same partition to minimize cross-partition queries.

Graph processing frameworks and database tools often provide built-in algorithms for community detection, which help group highly connected nodes. Teams can also monitor query performance over time and iteratively refine partitioning strategies to align with evolving workloads.

By combining domain driven design with continuous performance monitoring, teams can incrementally optimize graph layouts without requiring major architectural changes.

In production-inspired environments, topology-aware strategies significantly reduce traversal fan out and improve both median and tail latency under concurrent load.

Though repartitioning introduces operational complexity, the performance gains justify the effort once the knowledge graph becomes central to digital product delivery.

In Practice: Partitioning by Product Domain

Let’s look at what happens when our supply chain graph scales across multiple database nodes.

If we use Default Hash Partitioning, the graph is split randomly by node IDs. Alice might end up on Machine 1, the NeoPhone 15 on Machine 2, and the MX-200 Chip on Machine 3. A query tracking whether a component shortage affects Alice's order requires a slow, expensive network hop across three separate physical servers.

Using Topology-Aware Partitioning, we can configure the cluster to use the Region or Product_Line as a partitioning key.

  • Partition A (Europe Hub): Co-locates Region: EU, Product: NeoPhone 15, its internal MX-200 Chip, and local customer orders.

Result: A multi-hop traversal checking component supply chains for European customers happens entirely within local memory on a single machine, reducing query latency.

Managing Semantic Inference Without Sacrificing Performance

Semantic inference is a defining strength of EKGs but also a frequent source of scalability challenges.

The Inference Cost Problem

Applying full ontology reasoning at query time can dramatically increase computational overhead. In some systems, inference effectively multiplies graph size, increasing memory and CPU consumption. Not all inferred relationships are equally valuable for every workload.

Strategies for Selective Inference and Materialization

Scalable EKG platforms typically adopt a selective strategy:

  1. Precompute and materialize frequently accessed inferences

  2. Offload complex reasoning to batch or asynchronous pipelines

  3. Disable low value inference paths in latency-sensitive workloads

Hierarchical classifications and role-based relationships are often materialized ahead of time, while complex rule based reasoning is reserved for offline processing. This approach stabilizes query latency and reduces peak CPU utilization in enterprise deployments.

In Practice: Materializing the Compliance Path

Recall our semantic rule: If a component has a supply risk, the final product inherits that risk.

  • The Scalability Bottleneck (Query-Time Inference): Every time an enterprise dashboard loads a product catalog of 10,000 items, the engine must recursively calculate: Product -> Has Component -> Supplied By -> Supplier Country -> Embargo List. Under high concurrent load, this calculation crashes performance.

  • The Optimization (Materialization): We run an asynchronous batch job or Kafka consumer that listens for supplier updates. When a supplier's status changes, it computes the inference once and writes a direct property is_high_risk: true directly onto the Product node in the operational LPG.

Now, the customer-facing application reads a simple, static property without running an expensive multi-hop recursive inference query during runtime.

Improving Query Performance with Smarter Planning

As query complexity increases, query planning becomes a decisive performance lever.

Limitations of Static Planning

Traditional graph engines often rely on static heuristics or limited statistics for execution planning. In dynamic enterprise environments where data distributions evolve, these heuristics frequently produce suboptimal execution plans, leading to unpredictable performance.

ML-Assisted Query Optimization

Machine learning techniques are increasingly being applied to query optimization, particularly for cardinality estimation. By learning from historical query execution data, ML models can predict plan costs more accurately than rule-based systems.

In controlled experiments and production pilots, ML-assisted planning has demonstrated substantial reductions in execution time for complex traversals, as well as improved consistency in response times.

While implementation requires operational maturity, this represents a promising direction for large scale graph optimization.

In Practice: Optimizing Traversal Direction

Consider this query on our data: "Find all customers who purchased a product containing the MX-200 Chip."

There are two ways the graph execution planner can execute this:

  1. Plan A: Start at Component: MX-200, find the products it belongs to, and then find the customers who bought those products.

  2. Plan B: Scan all Customer nodes in the database, look at their purchases, and filter for the ones containing the chip.

If the MX-200 is a rare chip used in only one niche product, Plan A is incredibly fast. If it is a generic resistor used in millions of products, Plan B or a modified hybrid plan might be more efficient.

An ML-assisted query planner analyzes the real-time cardinality (the actual count) of the PART_OF and PURCHASED relationships in your specific database instance. It prevents the graph engine from choosing a disastrously slow traversal path when data distributions shift unexpectedly.

Observability as a First Class Requirement

Scalability can't be managed without deep observability.

Beyond Infrastructure Metrics

Monitoring CPU and memory alone provides limited insight into graph-specific performance issues. Effective EKG observability includes:

  1. Query level latency metrics

  2. Traversal depth and fan-out tracking

  3. Inference cost monitoring

  4. Partition imbalance detection

Closing the Optimization Loop

By continuously analyzing these signals, teams can iteratively refine partitioning strategies, caching policies, and materialization decisions. This feedback loop improves predictability and reduces production incidents.

In practice, strong observability often distinguishes proactive optimization from reactive firefighting.

Impact on Digital Product Platforms

When applied collectively, these optimization strategies materially enhance scalability and reliability. Across enterprise deployments, teams commonly observe:

  1. Reduced latency in real time workloads

  2. Improved ingestion throughput under sustained load

  3. Linear or near linear scaling as datasets grow

  4. Greater stability during traffic spikes

These technical improvements translate directly into business outcomes: faster recommendations, more relevant search results, and increased confidence in deploying EKGs as mission critical infrastructure.

Conclusion

Enterprise knowledge graphs are no longer experimental. They're becoming the backbone of intelligent, data driven systems. As teams move toward AI-powered decision making, the role of knowledge graphs is expanding beyond storage into enabling context-aware reasoning and automation.

An optimized EKG isn't just a database – it acts as the connective tissue between data, models, and real world applications. It provides the structured context that modern AI systems, including agentic workflows and autonomous decision engines, rely on to operate effectively.

By adopting hybrid architectures, topology-aware partitioning, and intelligent query strategies, teams can build scalable and resilient graph systems that support both operational and analytical workloads.

Ultimately, organizations that invest in well-designed knowledge graph infrastructure will be better positioned to power the next generation of AI systems where retrieval, reasoning, and action are seamlessly integrated.

]]> <![CDATA[ How to Build a Browser-Based PDF Metadata Editor Using JavaScript – A Step-by-Step Guide ]]> https://www.freecodecamp.org/news/how-to-build-a-browser-based-pdf-metadata-editor-using-javascript/ 6a24b9ff67572e709df5342b Sun, 07 Jun 2026 00:23:27 +0000 PDF files contain more information than what appears on the page.

Behind every PDF document is metadata that stores information such as the document title, author, subject, keywords, creator application, creation date, and modification date.

Metadata helps organize documents, improve searchability, and provide useful information when files are shared between users or systems.

In this tutorial, you'll build a browser-based PDF Metadata Editor using JavaScript.

Users will be able to upload a PDF, preview the document, view existing metadata, update metadata fields, add custom metadata entries, and download the updated PDF directly from the browser.

The entire process runs locally without requiring a backend server

Table of Contents

  1. Why PDF Metadata Is Important

  2. How PDF Metadata Editing Works

  3. Project Setup

  4. What Library Are We Using?

  5. Creating the Upload Interface

  6. Previewing Uploaded PDF Files

  7. Reading PDF Metadata

  8. Editing PDF Metadata

  9. Updating and Saving Metadata

  10. Generating the Updated PDF

  11. Why PDF Metadata Editing Is Useful

  12. Demo: How the PDF Metadata Tool Works

  13. Important Notes from Real-World Use

  14. Common Mistakes to Avoid

  15. Conclusion

Why PDF Metadata Is Important

PDF metadata is commonly used in business documents, contracts, reports, invoices, ebooks, academic papers, legal documents, and archived files.

When a PDF contains proper metadata, document management systems can organize files more effectively.

Search engines, enterprise search tools, and document indexing systems can also identify documents more accurately.

Metadata becomes especially useful when managing large collections of files because users can quickly locate documents based on title, author, subject, keywords, or custom information.

Updating metadata also helps keep documents organized after modifications, ownership changes, or publishing updates.

How PDF Metadata Editing Works

A PDF metadata editor loads the document inside the browser and reads information stored within the PDF file properties.

Users can review existing metadata, update values, add custom metadata fields, and save the changes into a new PDF document.

Everything happens locally inside the browser.

This means uploaded documents never leave the user's device, which improves privacy and security while eliminating the need for server-side processing.

Project Setup

This project is intentionally simple.

You'll only need:

  • An HTML file

  • A JavaScript file

  • A PDF processing library

No backend server or database is required. Everything runs right inside the browser.

What Library Are We Using?

We'll use PDF-lib to read and update PDF metadata.

PDF-lib provides functions for loading PDF documents, accessing metadata properties, modifying document information, and exporting updated files.

Add the library using a CDN:

<script src="https://unpkg.com/pdf-lib/dist/pdf-lib.min.js"></script>

Once loaded, JavaScript can access PDF metadata directly from the browser.

Creating the Upload Interface

Users first need a way to upload PDF files.

A simple file input is enough:

<input type="file" id="pdfInput" accept=".pdf">

JavaScript can then detect when a PDF file is selected:

const input = document.getElementById("pdfInput");

input.addEventListener("change", (event) => {
  const file = event.target.files[0];
  console.log(file.name);
});

Here's what the upload section looks like:

PDF upload interface for browser-based metadata editor

Previewing Uploaded PDF Files

After uploading a PDF, users should be able to preview the document before making metadata changes.

The browser can render PDF pages using PDF.js:

const loadingTask = pdfjsLib.getDocument(url);

loadingTask.promise.then((pdf) => {
  console.log(pdf.numPages);
});

The preview area also includes page navigation buttons so users can move between pages.

This helps verify the correct document was uploaded before editing metadata.

Here's what the preview section looks like:

Uploaded PDF preview with page navigation controls

Reading PDF Metadata

Once the PDF is loaded, metadata can be extracted from the document.

For example:

const pdfDoc = await PDFLib.PDFDocument.load(arrayBuffer);

const title = pdfDoc.getTitle();
const author = pdfDoc.getAuthor();

console.log(title);
console.log(author);

This information can then be displayed inside editable form fields.

Editing PDF Metadata

Users can update common document properties such as title, author, subject, keywords, creator information, and modification dates.

Custom metadata fields can also be added when additional document information is required.

For example:

pdfDoc.setTitle("Project Report");
pdfDoc.setAuthor("John Doe");
pdfDoc.setSubject("Monthly Review");

Here's what the metadata editor looks like:

PDF metadata editor with title author keywords and custom metadata fields

Updating and Saving Metadata

Once the metadata fields have been updated, JavaScript can apply the changes to the PDF document.

For example:

pdfDoc.setTitle("Updated Document");
pdfDoc.setAuthor("John Doe");
pdfDoc.setSubject("PDF Metadata Tutorial");

Custom metadata values can also be inserted before exporting the document.

After all changes are complete, users click the Update Metadata button to generate the modified PDF.

Generating the Updated PDF

After updating metadata, the browser creates a new PDF document containing the revised information.

The original document remains unchanged while the updated version is generated locally.

const pdfBytes = await pdfDoc.save();

The updated file can then be prepared for download.

Why PDF Metadata Editing Is Useful

Metadata is often overlooked, but it plays an important role in document management.

Organizations use metadata to organize thousands of PDF files across internal systems.

When documents contain proper titles, keywords, subjects, and author information, they become easier to search, categorize, and manage.

For example, legal teams may store contracts with custom metadata fields for clients or case numbers.

Businesses often use metadata to organize invoices, reports, proposals, and project documents.

Publishers frequently update document properties before distributing ebooks, manuals, and guides.

Metadata can also improve indexing in document management systems and make archived files easier to locate months or years later.

Updating metadata before sharing documents creates a cleaner and more professional final file while improving long-term document organization.

Demo: How the PDF Metadata Tool Works

Step 1: Upload a PDF File

Users begin by uploading a PDF document into the browser.

The upload area supports drag-and-drop functionality as well as manual file selection.

Upload PDF file for metadata editing

Step 2: Preview the Uploaded Document

After uploading the PDF, the tool displays a document preview.

Users can navigate between pages using the left and right navigation buttons.

This allows quick verification that the correct document has been loaded.

Uploaded PDF preview with page navigation

Step 3: Edit PDF Metadata

The metadata editor loads existing document properties automatically.

Users can update fields such as title, author, subject, keywords, creator information, dates, and custom metadata values.

Custom fields can be added or removed as needed.

Edit PDF metadata including custom metadata fields

Step 4: Update Metadata

After making changes, users click the Update Metadata button.

The browser processes the document and applies all metadata updates locally.

allinonetools pdf toolskit pdf meata dat update

Step 5: Download the Updated PDF

Once processing is complete, the updated PDF becomes available for download.

The output section displays the updated filename, total page count, file size information, and download controls as well as rename option before download.

A Start Over button is also available for processing another document.

Updated PDF ready for download with file details

Important Notes from Real-World Use

When working with PDF metadata, it's important to validate uploaded files before processing them.

For example:

if (!file.name.endsWith(".pdf")) {
  alert("Please upload a PDF file");
  return;
}

Large PDF files may require additional processing time.

Always verify metadata values before generating the updated document.

Sensitive information stored inside metadata should be reviewed carefully before sharing documents publicly.

Common Mistakes to Avoid

One common mistake is assuming that all PDFs contain metadata. Many documents may have empty metadata fields that need to be populated manually.

For example:

const title = pdfDoc.getTitle() || "Untitled Document";

Another mistake is forgetting to update the modification date after changing document properties.

Always review metadata values before exporting the final file.

Previewing the document and checking file details before download can help prevent mistakes.

Conclusion

In this tutorial, you built a browser-based PDF Metadata Editor using JavaScript.

You learned how to upload PDF files, preview document pages, read existing metadata, update document properties, add custom metadata fields, and generate updated PDF files directly inside the browser.

More importantly, you saw how modern browsers can handle PDF property management locally without requiring a backend server.

This approach keeps document processing fast, private, and easy to use.

If you'd like to see a working example, you can try out this free PDF Metadata Tool and explore how metadata can be viewed and updated directly in the browser.

Once you understand this workflow, you can extend it further with features like PDF encryption, document signing, watermarking, page organization, annotations, and advanced PDF editing tools.

]]>