Unifying Internal and External APIs: Lessons Learned from Harbour
Caio Pizzol
Engenheiro de Software & Criador do usepolvo | Resolvendo Desafios de Integra??o @ Harbour
Introduction
In software development, a common practice has long been to maintain separate APIs for internal and external use. The reasoning often goes that internal APIs can be more flexible and feature-rich while external APIs need to be more stable and carefully designed. This separation, while logical on the surface, can lead to a host of challenges, including inconsistency, duplicated effort, and slower improvement cycles.
At Harbour, we've taken a different approach. We've implemented a unified API architecture where our customer-facing API is also used internally. This might seem counterintuitive at first, but it's a strategy that has yielded significant benefits.
The concept of a unified API architecture is simple: instead of maintaining two separate sets of APIs, you create a single, robust API that serves both internal and external needs. This approach ensures that every endpoint, every feature, and every improvement benefits both your internal teams and your external customers.
Through our experience at Harbour, we've found that this unified approach leads to three key outcomes: better quality, improved consistency, and faster improvements. By using our API internally, we subject it to rigorous real-world testing before it reaches our customers. This unified structure naturally keeps our internal and external interfaces in sync, reducing inconsistencies and maintenance overhead. Perhaps most importantly, it accelerates improvements based on actual use cases and pain points experienced by our own teams.
In this article, we'll explore the lessons we've learned from implementing this unified API architecture at Harbour. We'll discuss the benefits, address the challenges, and provide practical strategies for those considering a similar approach. Whether you're a startup looking to streamline your development process or an established company aiming to improve API quality, the insights from our journey at Harbour offer valuable perspectives on the power of unified APIs.
The Case for Unified APIs:
Real-world Testing
When your internal teams use the same API as your external customers, you create a powerful testing environment. At Harbour, we've found that this approach stress-tests our API in ways that traditional QA processes often miss. Our developers, product managers, and other internal users interact with the API in diverse, often unpredictable ways. They push the boundaries of what the API can do, use it in combination with other systems, and often find edge cases that we hadn't considered. This real-world usage uncovers potential issues, performance bottlenecks, and usability problems before they ever reach our external customers. The benefits of this approach are significant. We catch and fix issues earlier in the development cycle, reducing the cost and complexity of repairs. More importantly, it dramatically improves the reliability and robustness of our API for our customers, enhancing their experience and building trust in our platform.
Consistency by Design
One of the most challenging aspects of maintaining separate internal and external APIs is ensuring consistency between the two. Discrepancies can easily creep in, leading to confusion, bugs, and increased maintenance overhead. By unifying our API, we've eliminated this problem by design. There's no possibility of inconsistency because there's only one API to maintain. This approach ensures that our internal teams are always working with the same interface as our customers, reducing misunderstandings and aligning everyone's expectations. Moreover, this consistency significantly reduces our maintenance overhead. Instead of updating and testing two separate systems, we focus our efforts on a single, high-quality API. This not only saves time and resources but also allows us to invest more in improving and expanding our API's capabilities.
Rapid Iteration and Improvement
Perhaps the most powerful benefit of a unified API is how it accelerates our improvement cycle. By using the API internally, we create a tight feedback loop that drives rapid enhancements. Our internal teams become the first line of users, quickly identifying pain points, usability issues, and areas for improvement. This immediate, hands-on experience leads to faster identification and resolution of issues. Often, the people using the API are the same ones who can implement changes, further speeding up the iteration process. This rapid iteration extends to feature development as well. As our internal teams use the API to build new products or services, they naturally discover new requirements or opportunities for enhancement. These insights drive our API's evolution, ensuring that it continually improves and expands to meet real-world needs. The result is an API that evolves more quickly and in more relevant ways than would be possible with a separate internal/external approach. This not only benefits our internal efficiency but also ensures that our external customers are always working with an API that's been battle-tested and refined based on practical usage.
By embracing a unified API architecture, we've transformed our API development process into a dynamic, self-improving system that benefits both our internal operations and our external customers.
Challenges and Solutions:
Security Concerns
One of the primary hesitations in adopting a unified API approach is the fear of exposing sensitive internal operations to the outside world. At Harbour, we took these concerns seriously and implemented robust solutions. To address fears about exposing internal operations, we adopted a principle of "secure by design." This means that every endpoint, regardless of its intended use, is built with the assumption that it could be accessed externally. We implemented fine-grained access controls, allowing us to restrict certain operations or data to specific user roles or internal systems. For authentication and authorization, we implemented a multi-layered approach tailored to different use cases. For our customers, we use API keys, providing a simple yet secure method for accessing our API. Our internal web application employs Relationship-Based Access Control (ReBAC), allowing for nuanced, context-aware permissions that adapt to the complex relationships within our system. For internal web services, we utilize OAuth 2.0, enabling secure, token-based authentication between services. This diverse approach allows us to maintain high-security standards while catering to the specific needs of different user types. It ensures that all users, whether external customers or internal systems, only have access to the resources and operations they need, maintaining the integrity of our unified API across all use cases.
Performance Considerations
Balancing the performance needs of internal systems with the scalability requirements of external users was another significant challenge. Internal operations often prioritize low latency, while external APIs need to handle variable loads and potential traffic spikes. To address this, we implemented a caching strategy that benefits both use cases. For frequently accessed data, we use a distributed caching system to reduce database load and improve response times. This helps maintain efficiency for internal operations while also supporting external scalability. We also adopted an asynchronous processing model for certain operations. This approach prevents slow operations from blocking other requests, improving overall API responsiveness for both internal and external users.
Versioning and Backwards Compatibility
Managing API changes without disrupting users is crucial, especially when both internal systems and external customers rely on the same API. At Harbour, we implemented a comprehensive versioning strategy to address this challenge. We adopted semantic versioning for our API, clearly communicating the nature of changes (major, minor, or patch) to all users. For major versions that include breaking changes, we maintain multiple active versions of the API concurrently. This allows both internal teams and external customers to migrate at their own pace. To manage backward compatibility effectively, we implemented a deprecation policy. When introducing new features or changing existing ones, we clearly communicate these changes and provide a deprecation timeline. This typically includes:
领英推荐
By addressing these challenges head-on, we've been able to realize the benefits of a unified API while maintaining security, performance, and stability for all users. These solutions have been key to the success of our unified API strategy at Harbour.
Implementation Strategies:
Designing with Both Audiences in Mind
Creating an API that serves both internal and external needs requires careful consideration in the design phase. At Harbour, we found success by focusing on flexibility and comprehensive documentation. For flexible endpoints, we implemented a query parameter-based approach that allows for different levels of detail in responses. For example, a GET request to /users might return basic information by default, but adding ?expand=details , permissions would provide more comprehensive data typically needed for internal operations. This allows a single endpoint to serve various use cases without compromising simplicity for external users. For documentation, we implemented a dual-layer approach. We maintain a public-facing API reference that covers all endpoints and basic usage. Additionally, we have an internal wiki with more detailed information about each endpoint's implementation, potential gotchas, and internal-only features. This ensures that both external developers and our internal teams have the information they need to use the API effectively.
Gradual Migration Approach
Transitioning from separate internal and external APIs to a unified structure is a significant undertaking. At Harbour, we adopted a gradual migration approach to minimize disruption. Our migration process followed these steps:
To manage this process without disrupting operations, we maintained both old and new endpoints simultaneously during the transition. We used feature flags to control access to new endpoints, allowing us to roll out changes gradually and roll back quickly if issues arose. We also invested heavily in monitoring during this process, setting up alerts to quickly identify any problems with the new unified endpoints. This allowed us to address issues promptly, often before they impacted users.
Monitoring and Analytics
Our unified API monitoring strategy centers on HealthCheck endpoints and a public status page. HealthCheck endpoints provide real-time information on API components, including database connections and external service dependencies.
Our public status page aggregates this data, offering: Real-time API status, Historical uptime, Maintenance schedules and Incident reports.
This approach provides a unified view of our API's health, serving both internal needs and offering transparency to users. It enables quick issue detection and response, maintaining high availability for all API users.
By closely monitoring both internal and external usage, we're able to continuously refine our API, ensuring it meets the evolving needs of all users. This data-driven approach has been key to maintaining a high-quality, efficient API that serves both our internal teams and our customers effectively.
Conclusion:
Our journey with unified API architecture at Harbour has yielded significant benefits:
While challenges exist in security, performance, and versioning, addressing these has resulted in a stronger, more versatile API.
We encourage developers and organizations to explore unified API approaches. Start by assessing your current structure for inconsistencies and duplication. Begin small, perhaps with a single endpoint, and expand based on feedback.
Remember, this is a mindset shift: your API is a product serving both internal and external users equally.
Looking ahead, we're excited to announce that Harbour will soon launch API v2. This new version builds on our unified API principles, introducing enhanced performance, expanded capabilities, and even greater flexibility for our users. Stay tuned for more details on how API v2 will revolutionize your integration experience with Harbour.
By adopting a unified API architecture, you position yourself at the forefront of efficient, consistent, and high-quality API development. At Harbour, it's become the cornerstone of our technical strategy. We believe it can do the same for you.