Building Scalable APIs with Dart and Shelf - Part 1: A Step-by-Step Guide

Scalable APIs are the backbone of modern applications, enabling seamless communication between services and delivering reliable performance under varying loads. Dart, primarily known for Flutter development, has emerged as a compelling choice for building server-side applications. With its strong type system, excellent performance and modern syntax, when paired with Shelf, a lightweight web server framework, developers can build powerful, scalable APIs with ease.

In this guide, we'll explore how to build scalable APIs using Dart and Shelf. We'll cover everything from setting up your development environment to deploying a production-ready API.

Table of Contents

• Setting Up the Development Environment
• Creating Your First API Endpoint
• Building and Organizing Your API
• Integrating with a Database
• Enhancing API Performance and Scalability
• Testing and Deploying Your API
• Best Practices
• Conclusion

Setting Up the Development Environment

It’s important to have a well-configured development environment before getting into building scalable APIs with Dart and Shelf. We'll detail the process of setting up your environment and getting started.

1. Installing the Dart SDK

The first requirement is to install the Dart SDK on your local machine if you have not already done so. The Dart SDK provides the tools necessary for developing and running Dart applications.

For Linux:
Install Dart SDK with the following command:

sudo apt update
sudo apt install dart

For macOS:
Use Homebrew to install the Dart SDK as follows:

brew tap dart-lang/dart
brew install dart

For Windows:

• Download the Dart SDK from the official Dart website.
• Run the installer and follow the on-screen instructions.
• Add the Dart SDK to your system’s PATH variable to make it accessible from the command line.

2. Set Up a Project Directory

After installation, create a new Dart project using the CLI using your terminal.

dart create my_api && cd my_api

This command creates a new Dart project with a default structure and a simple "Hello world: 42!" program.

3. Add Project Dependencies

Adding Shelf to our project because it is needed for building APIs.

• Open the pubspec.yaml file in your project directory.
• Add Shelf, Shelf Router and Postgres as a dependencies:

dependencies:
  postgres: ^2.4.1
  shelf: ^1.4.2
  shelf_router: ^1.1.4

• Save the file and run the following command to install the dependency:

dart pub get

4. Start the PostgreSQL server

To complete this tutorial, it's expected that you have the PostgreSQL server installed in your computer. Check out this guide to install the PostgreSQL if you have not.

Starting a PostgreSQL server differs slightly depending on the operating system you're using. Below are the instructions for starting PostgreSQL on Windows, macOS, and Linux.

Windows: Using Command Prompt

• Open the Command Prompt as an Administrator.
• Navigate to the PostgreSQL installation directory, typically C:\Program Files\PostgreSQL\<version>\bin.
• Start the server using:

pg_ctl -D "C:\Program Files\PostgreSQL\<version>\data" start

• Replace <version> with your installed PostgreSQL version.

macOS: Using Terminal

• If installed via Homebrew, start the PostgreSQL server using:

brew services start postgresql

Linux: Using Systemd

• Open a terminal.
• Use the following command to start the PostgreSQL service:

sudo systemctl start postgresql

• To check the status:

sudo systemctl status postgresql

5. Create a Server File in the bin Directory

With Shelf installed, you can set up your project for API development:

• Create a new Dart file, server.dart, in the bin directory:

touch bin/server.dart

• Set up a basic server in the server.dart file as seen:

import 'dart:io';
import 'package:shelf/shelf.dart';
import 'package:shelf/shelf_io.dart' as io;

void main() {
  var handler = const Pipeline()
      .addMiddleware(logRequests())
      .addHandler((Request request) {
    return Response.ok('Hello, World!');
  });

  io.serve(handler, InternetAddress.anyIPv4, 8080).then((server) {
    print('Serving at http://${server.address.host}:${server.port}');
  });
}

• Run server with the command:

dart run bin/server.dart

Open a web browser and navigate to http://localhost:8080. You should see "Hello, World!" displayed.

With your development environment set up, you're now ready to build and scale your API using Dart and Shelf. In the next section, we'll dive into creating your first API endpoint.

Creating Your First API Endpoint

Now that your development environment is set up, it’s time to create your first API endpoint using Dart and Shelf. This section will guide you through setting up a basic Shelf server and handling a simple GET request.

1. Setting Up a Basic Shelf Server

To start, you need to create a basic server that listens for incoming HTTP requests.

• Open the server.dart file in the bin directory if you haven’t already.
• Update the file to include a basic Shelf server setup:

import 'dart:io';
import 'package:shelf/shelf.dart';
import 'package:shelf/shelf_io.dart' as io;

void main() async {
  // Define a handler function that responds to requests
  final handler = const Pipeline()
      .addMiddleware(logRequests())
      .addHandler((Request request) {
    return Response.ok('Welcome to your first API!');
  });

  // Start the server on port 8080
  final server = await io.serve(handler, InternetAddress.anyIPv4, 8080);
  print('Server listening on port ${server.port}');
}

This script sets up a basic Shelf server that logs incoming requests and responds with "Welcome to your first API!" for every request.

2. Handling Basic GET Requests

create a more dynamic API, you’ll want to handle different types of requests and routes.

• Modify the handler function to respond differently based on the request path:

import 'dart:io';
import 'package:shelf/shelf.dart';
import 'package:shelf/shelf_io.dart' as io;

void main() async {
  final handler = const Pipeline()
      .addMiddleware(logRequests())
      .addHandler((Request request) {
    if (request.requestedUri.path == '/hello') {
      return Response.ok('Hello, Dart!');
    } else {
      return Response.notFound('Page not found');
    }
  });

  final server = await io.serve(handler, InternetAddress.anyIPv4, 8080);
  print('Server listening on port ${server.port}');
}

In this example, when a request is made to /hello, the server responds with "Hello, Dart!". Any other path returns a 404 "Page not found" response.

3. Returning JSON Responses

APIs often return data in JSON format. Let’s modify the handler to return a JSON response:

• Add the dart:convert package to handle JSON encoding:

import 'dart:convert';

• Update the handler to return a JSON response:

import 'dart:io';
import 'dart:convert';
import 'package:shelf/shelf.dart';
import 'package:shelf/shelf_io.dart' as io;

void main() async {
  final handler = const Pipeline()
      .addMiddleware(logRequests())
      .addHandler((Request request) {
    if (request.requestedUri.path == '/user') {
      final user = {
        'id': 1,
        'name': 'John Doe',
        'email': 'john.doe@example.com'
      };
      final jsonResponse = jsonEncode(user);
      return Response.ok(jsonResponse, headers: {'Content-Type': 'application/json'});
    } else {
      return Response.notFound('Page not found');
    }
  });

  final server = await io.serve(handler, InternetAddress.anyIPv4, 8080);
  print('Server listening on port ${server.port}');
}

When a request is made to /user, the server responds with a JSON object containing user details.

4. Testing Your Endpoint

• Start the server by running:

dart run bin/server.dart

• Open your browser or a tool like Postman and navigate to http://localhost:8080/user. You should see the JSON response with the user details.

Building and Organizing Your API

Creating a scalable and maintainable API involves more than just writing code; it requires thoughtful organization and structure. This section will guide you through best practices for building and organizing your API using Dart and Shelf, ensuring your codebase remains clean, modular, and easy to manage.

1. Structuring your API

A well-structured API makes it easier to navigate, debug, and extend. Here’s a recommended structure for your Dart project:

my_api/
├── bin/
│   └── server.dart
├── lib/
│   ├── handlers/
│   │   └── user_handler.dart
│   ├── middleware/
│   │   └── auth_middleware.dart
│   └── routers/
│       └── user_router.dart
└── pubspec.yaml

• bin/server.dart: The entry point for your application where the server is initialized.
• lib/handlers/: Contains request handlers, each handling specific endpoints or sets of endpoints.
• lib/middleware/: Contains middleware functions for processing requests and responses (e.g., authentication, logging).
• lib/routers/: Contains routers that define and organize routes for different parts of your API.

The structure of the project will continue to change as we continue optimizing our codebase for scalability and modularity.

2. Implementing Routes and Handlers

To keep your code modular, you should separate the logic for handling requests into different handler files. For example, a handler for user-related routes:

lib/handlers/user_handler.dart

import 'package:shelf/shelf.dart';

Response userHandler(Request request) {
  return Response.ok('User Handler Response');
}

You can then use this handler in your router:

lib/routers/user_router.dart

import 'package:shelf/shelf.dart';
import 'package:shelf_router/shelf_router.dart';
import '../handlers/user_handler.dart';

Router get userRouter {
  final router = Router();

  router.get('/user', userHandler);

  return router;
}

In your server.dart, use this router to organize your API routes:

bin/server.dart

import 'dart:io';
import 'package:shelf/shelf.dart';
import 'package:shelf/shelf_io.dart' as io;
import '../lib/routers/user_router.dart';

void main() async {
  final handler = const Pipeline()
      .addMiddleware(logRequests())
      .addHandler(userRouter);

  final server = await io.serve(handler, InternetAddress.anyIPv4, 8080);
  print('Server listening on port ${server.port}');
}

3. Using Middleware

Middleware in Shelf allows you to intercept and process requests and responses. For example, an authentication middleware can be used to check if a request is authenticated:

lib/middleware/auth_middleware.dart

import 'package:shelf/shelf.dart';

Middleware authMiddleware() {
  return (Handler innerHandler) {
    return (Request request) async {
      // Example authentication check
      if (request.headers['Authorization'] == null) {
        return Response.forbidden('Unauthorized');
      }
      return innerHandler(request);
    };
  };
}

Add this middleware to your pipeline:

bin/server.dart

final handler = const Pipeline()
    .addMiddleware(logRequests())
    .addMiddleware(authMiddleware())
    .addHandler(userRouter);

4. Organizing Routes for Scalability

As your API grows, organizing routes in a scalable manner becomes crucial. Group related routes into separate routers, and combine them in your main server file. This modular approach ensures each part of your API is independently manageable.

Example of combining multiple routers:

bin/server.dart

import 'dart:io';
import 'package:shelf/shelf.dart';
import 'package:shelf/shelf_io.dart' as io;
import 'package:my_api/routers/user_router.dart';
import 'package:my_api/middleware/auth_middleware.dart';
import 'package:my_api/routers/product_router.dart';


void main() async {
  final handler = const Pipeline()
      .addMiddleware(logRequests())
      .addMiddleware(authMiddleware())
      .addHandler(Cascade()
          .add(userRouter)
          .add(productRouter)
          .handler);

  // Example: Internal request with Authorization header for testing
  final request = Request('GET', Uri.parse('http://localhost:8080/user'), headers: {
    'Authorization': 'Bearer test-token',
  });
  final response = await handler(request);
  print('Internal Request Response: ${await response.readAsString()}');
}

Integrating with a Database

Integrating a database into your Dart and Shelf project is a crucial step for building a full-featured API. This section will guide you through setting up a connection to a database, executing queries, and organizing your database interaction code for scalability and maintainability.

1. Choose a Database

For this guide, we'll use PostgreSQL, a popular relational database. You'll need the postgres package for Dart to interact with a PostgreSQL database.

2. Installing the PostgreSQL Package

Add the postgres package to your pubspec.yaml:

dependencies:
  postgres: ^2.4.1

Run dart pub get to install the package.

3. Setting Up the Database Connection

Create a new file to manage your database connection:

lib/database/connection.dart:

import 'package:postgres/postgres.dart';

class DatabaseConnection {
  final PostgreSQLConnection connection;

  DatabaseConnection._privateConstructor()
      : connection = PostgreSQLConnection(
          'localhost', // Host
          5432,        // Port
          'my_database', // Database name
          username: 'user', // Database username
          password: 'password', // Database password
        );

  static final DatabaseConnection _instance = DatabaseConnection._privateConstructor();

  static DatabaseConnection get instance => _instance;

  Future<void> open() async {
    await connection.open();
  }

  Future<void> close() async {
    await connection.close();
  }
}

N.B.: Replace the connection credentials to match your local PostgreSQL server.

4. Creating a Data Access Layer (DAL)

Organize your data access logic in a dedicated folder. Here's an example for handling user data:

lib/data/user_dal.dart:

import '../database/connection.dart';

class UserDAL {
  final db = DatabaseConnection.instance.connection;

  Future<List<Map<String, Map<String, dynamic>>>> getAllUsers() async {
    return await db.mappedResultsQuery('SELECT * FROM users');
  }
}

5. Using the DAL in Your API Handlers

Update your user handler to fetch data from the database:

lib/handlers/user_handler.dart:

import 'package:shelf/shelf.dart';
import '../data/user_dal.dart';

Future<Response> userHandler(Request request) async {
  final userDAL = UserDAL();
  final users = await userDAL.getAllUsers();
  return Response.ok(users.toString());
}

6. Updating the Server Initialization

Ensure the database connection is opened when the server starts and closed when it shuts down:

bin/server.dart:

import 'dart:io';
import 'package:shelf/shelf.dart';
import 'package:shelf/shelf_io.dart' as io;
import 'package:my_api/routers/user_router.dart';
import 'package:my_api/middleware/auth_middleware.dart';
import 'package:my_api/routers/product_router.dart';
import 'package:my_api/database/connection.dart';



void main() async {

  final db = DatabaseConnection.instance;
  await db.open();

  final handler = const Pipeline()
      .addMiddleware(logRequests())
      .addMiddleware(authMiddleware())
      .addHandler(Cascade()
          .add(userRouter)
          .add(productRouter)
          .handler);

  // Start the server on all IPv4 addresses, port 8080
  final server = await io.serve(handler, InternetAddress.anyIPv4, 8080);
  print('Server listening on port ${server.port}');

  // Graceful shutdown: Close database and server on SIGINT (Ctrl+C)
  ProcessSignal.sigint.watch().listen((_) async {
    print('Shutting down...');
    await db.close();
    await server.close(force: true);
    exit(0);
  });

  // Example internal request with Authorization header for testing purposes
  final request = Request('GET', Uri.parse('http://localhost:8080/user'), headers: {
    'Authorization': 'Bearer test-token',
  });
  final response = await handler(request);
  print('Internal Request Response: ${response.statusCode}');
}

Here is what an updated project structure should look like:

my_api/
├── bin/
│   ├── migrate.dart
│   └── server.dart
├── lib/
│   ├── data/
│   │   └── user_dal.dart       
│   ├── database/
│   │   └── connection.dart
│   ├── middleware/
│   │   └── auth_middleware.dart
│   ├── routers/
│   │   ├── product_router.dart
│   │   └── user_router.dart
│   └── scripts/
│       └── insert_users.dart
├── pubspec.yaml
└── README.md

By integrating a database into your Dart project, you can dynamically manage and retrieve data for your API. Structuring the database connection and access logic into separate files ensures maintainability and scalability, setting the stage for more complex operations and features in your application.

In the next part of this tutorial, we will cover more on API scalability, testing and deployment and best practices. A link to the entire project on GitHub will also be shared.