ShadowDB
ShadowDB is a comprehensive, enterprise-grade PostgreSQL database management platform that combines high availability clustering, containerized deployments, and a modern web interface to provide developers and organizations with a robust database-as-a-service solution.
For developers and businesses building scalable applications, managing the underlying database infrastructure is a complex, time-consuming, and resource-intensive process. This often involves manually configuring, scaling, and maintaining multiple disparate technologies for load balancing, caching, and connection pooling, which detracts from the primary focus of application development and can lead to performance bottlenecks, reliability issues, and security vulnerabilities.
ShadowDb is designed to be a fully managed, production-ready database service that combines the power of PostgreSQL with Haproxy load balancing, built-in PgPool caching and Connection Pooling. We focus on performance, scalability, and ease of use, allowing developers to focus on building applications without worrying about database management.
Key Features
Architecture
A multi-tenant, containerized infrastructure managed via a Next.js web application. It uses Docker Compose for instance orchestration, with HAProxy for write-routing and Pgpool for connection pooling and read-routing. ETCD is used for service discovery and configuration management, ensuring high availability.
Database Design
A high-availability PostgreSQL cluster managed by Patroni. The cluster consists of a Primary DB for write operations and multiple Replica DBs for read operations, with data synchronized via real-time streaming replication. This setup ensures both load balancing and automatic failover.
Real-time Features
Real-time data integrity is maintained through PostgreSQL's streaming replication, which continuously pushes data from the primary to the replica databases. The Patroni managers monitor the cluster's health in real-time to trigger failovers instantly upon detecting a failure.
Frontend
Backend
Database
Orchestration
Deployment
Tiered Resource Management
Allocating specific computational resources, like CPU and RAM, to each database cluster based on the customer's subscription tier to ensure premium users receive more powerful instances.
Solution:
Enhanced the orchestration service to use different Docker Compose templates with predefined resource limits. When provisioning a cluster, the service selects the appropriate template based on the user's tier, automatically enforcing resource allocation.
Secure Role-Based Management
Ensuring that user roles within PostgreSQL are configured securely, preventing any role from having excessive or risky permissions that could compromise data security.
Solution:
Developed a permission management system with predefined, least-privilege roles. Created an automated process within the orchestration layer to manage role assignments and audit permissions.
Dynamic Cluster Orchestration
The need to dynamically create, configure, and manage entire high-availability PostgreSQL clusters on-demand to support a multi-tenant architecture.
Solution:
Built a custom orchestration service that uses Docker Compose and Patroni templates to programmatically provision new, isolated PostgreSQL clusters, using ETCD for configuration and automatic failover.
Timeline
3 months
2025
Team
Team of 1