These are the news items I've curated in my monitoring of the API space that have some relevance to the API design conversation and I wanted to include in my research. I'm using all of these links to better understand how the space is designing their APis and the underlying schema used as part of their requests and responses.06 Mar 2017
This is a repeat story of one I wrote two years ago, but things haven't changed so I'm going to rant about again, 2017 style. We need someone to develop an open source, visual API design editor. There is the Swagger Editor, but that is more a web IDE, and I'm looking for a well designed, intuitive, visual editor for managing your OpenAPI definitions, that is embeddable and easily integrated into any web or mobile system.
There are examples of nice visual API design editors with some of the leading API service providers, but I'm looking for something any API service provider could put to work. I included these examples in my last story, except I'm including Stoplight this round instead of Gelato, because they gave me an upgrade my account message, and wouldn't let me see the interface. These are currently my favorite three API design editors:
- Restlet - A design, deployment, management service provider.
- APIMATIC - An SDK, API definition transformer, and continous integration provider.
- Stoplight - An API design, definition, documentation, and orchestration provider.
I enjoy using all three of their API design interfaces. They are well thought out and provide what you need to design new APIs, as well manage the design of your existing APIs. While they are doing a good job at this, it is something I'd like to see standardized, and open sourced, so that other service providers do not have to reinvent the wheel here. We shouldn't have to use a different API design interface for each service we depend on, we need a common editor, making things more familiar.
Ok, rant over. This is something I'll keep talking about until there is a solution. Oh, I have to say, I am impressed with what the Materia team has done for API design on the desktop. Between them and Stoplight, there has been positive motion forward. I'm just looking to give things another nudge and help folks understand the opportunity. I'll explore possible business models for such a solution in another post, and keep beating this drum until I get what I want.
Everyone developing APIs struggles with API design. Ok, maybe a few of the gurus out there don't, but the rest of us need education, practice, and ideally someone or something to help guide us through the best practices when it comes to API design. I try to track what different companies are doing when it comes to API design, and maintain a list of any API design guides I come across in my work.
If you are looking to start getting a grasp on API design at your company, organization, institution or agency, I recommend starting on the journey to define your own API design guide--you will be surprised what you can learn along the way. You can kick things off by visiting my API design research, but I strongly recommend you head over to the API Stylebook, and build on this important work.
My friend Arnaud has aggregated 12 of the leading API design guides out there and is currently processing Google's API design guide, adding more knowledge to the repository. He takes each API design guide and breaks them down into common API design topics that you can consider as part of your own process, and resulting guide. His work is all machine readable and forkable so that you can easily assemble your own guide, and get to work learning, defining, and refining--then you can submit your guide to the API Stylebook when you are ready.
API design and API definitions are the top two most trafficked areas of my website in 2017, and most of 2016. API design is top of mind for API providers who are further along in their journey, and the ones who are seriously rocking it usually have some sort of guide, helping keep things consistent. If you need any help, feel free to reach out. I am not an API design expert, but I know my way around the finer details, and concepts at play, and I'm happy to help where I can. I know Arnaud is happy to answer questions as well, and if nothing else, let me help tell the story of your API design journey so that others can learn from it.
<p>I’m working on the next evolution in my API research, and I’m investing more time and energy into the design of the guides I produce as a result of each area of my research. I’ve long produced a 20+ page PDF dumps of the leading areas of my research like API design, definitions, deployment, and management, but with the next waveÂ of industryÂ guides, I want to polish my approach a little more.Â </p>
The biggest critique I get from folks about the API industry guides I produce is that they provide too much information, aren't always polished enough, and sometimes contain some obvious errors. I'm getting better at editing, but this only goes so far, and I'm bringing in a second pair of eyes to review things before they go out. Another thing I'm introducing into the process is the use am of professional design software (Adobe InDesign) rather than just relying on PDF's generated from my internal system with a little HTML spit shine.
While it is taking me longer to dust off my Adobe skills than I anticipated, I am finding the design process to be extremely valuable. I've often dismissed the fact that my API research needed to look good, and that it is more important that it is just available publicly on my websites. This is fineÂ and is something that will continue, but I'm finding a more formal design process is helping me think through all of the material, better understand what is valuable, what is important, and hopefully better tell a story about why it is relevant to the API space. It is helping me take my messy backend data and content, and present it in a more coherent and useful way.
As I'm formalizing my approach using my API definition guide, I'm moving on to my API design guide, and I can't help but see the ironyÂ in learning the value of designÂ while publishing a guide on API design, where I highlight the benefits of API design to make sense of our growing digital assets. Once I've applied this new approach to my definition, design, deployment, DNS, and management guides I am going to go back to my API design for these resources, and take what I've learned and applied to the way I share the raw data and content. The lesson that stands out most at the moment, is that less is more, and simple speaks volumes.
I get the argument from hypermedia and linked data practitioners that we need to model our web API behavior on the web. It makes sense, and I agree that we need to be baking hypermedia into our API design practices. What I have trouble with is the fact that the web is a cornerstone that we should be modeling it after. I do not know what web y'all use every day, but the one I use, and harvest regularly is quite often is a pretty broken thing.
It just feels like we overlooking so much to support this one story. I'm not saying that hypermedia principles don't apply because the web is shit, I'm just saying maybe it isn't as convincing of an anchor to build a story that currently web APIs are shit. I understand that you want to sell your case, and trust me...I want you to sell your case, but using this argument just does not pencil out for me.
There is another aspect of this that I find difficult. That the web was developed and took root in a very different environment than web APIs are. We had more time and space to be more thoughtful about the web, and I do not think we have that luxury with web APIs. The stakes are higher, the competition is greater, and the incentives for doing it thoughtfully really do not exist in the startup environment that has taken hold. We can't be condemning API designers and architects for serving their current master (or can we?).
While I will keep using core web concepts and specs to help guide my views on designing, defining, and deploying my web APIs, I'm going to explore other ways to articulate why we should be putting them to use. I'm going to also be considering the success or failure of these elements based on the shortcomings of the web, and web APIs, while I work to better polish the existing stories we tell, as well as hopefully evolve new ones that help folks understand what the best practices for web APIs are.
I recently carved out some time to read A Web API ecosystem through feature-based reuse by Ruben Verborgh (@RubenVerborgh) and Michel Dumontier. It is a lengthy, very academic proposal on how we can address the fact that "the current Web API landscape does not scale well: every API requires its own hardcoded clients in an unusually short-lived, tightly coupled relationship of highly subjective quality."
I highly recommend reading their proposal, as there are a lot of very useful patterns and suggestions in there that you can put to use in your operations. The paper centers around the notion that the web has succeeded because we were able to better consider interface reuse, and were able to identify the most effective patterns using analytics, and pointing out that there really is no equivalent to web analytics for measuring an APIs effectiveness.
In order to evolve Web API design from an art into a discipline with measurable outcomes, we propose an ecosystem of reusable interaction patterns similar to those on the human Web, and a task-driven method of measuring those.
To help address these challenges in the world of web APIs, Verborgh and Dumontier propose that we work to build web interfaces, similar to what we do with the web, employing a bottom-up to composing reusable features such as full-text search, auto-complete, file uploads, etc.--in order to unlock the benefits of bottom-up interfaces, they propose 5 interface design principles:
- Web APIs consist of features that implement a common interface
- Web APIs partition their interface to maximize feature reuse.
- Web API responses advertise the presence of each relevant feature
- Each feature describes its own invocation and functionality.
- The impact a feature on a Web API should be measured across implementations.
They provide us with a pretty well thought out vision involving implementations and frameworks, and the sharing of documentation, while universally applying metrics for being able to identify the successful patterns. It provides us with a compelling, "feature-based method to construct the interface of Web APIs, favoring reuse overreinvention, analogous to component-driven interaction design on the human Web."
I support everything they propose. I cannot provide any critique on the technical merits of their vision. However, I find it lacks an awareness of the current business and political landscape that I find regularly present in the hypermedia, and linked data material I consume.
Here are a few of the business and political considerations that contribute to the situation we find ourselves in that Verborgh and Dumontier are focused on, which will also work to slow the adoption of their proposed vision:
- Venture Capital - The current venture capital driven climate does not incentivize sharing and reuse, and their startups investing time and energy into web technologies.
- Intellectual Property - Modern views of the intellectual property, partially fueled by VC investment, but further exacerbated by legal cases like Oracle v Google force developers and designers to hold patterns close to their chest, limiting sharing and reuse again.
- Lazy Developers - Not all developers are knowledge seekers like the authors of this paper, and myself, many are just looking to get the job done and get home. There are few rewards for contributing back to the community, and once I have mine, I'm done.
- The Web Is Shit - One area that linked data and hypermedia folks tend to lose me is their focus on modeling things after the web. I agree the web is "working", but I don't know which one you use, but the one I use is shit, and only getting worse--have you scraped web content lately?
- Metrics & Analytics - Google Analytics started out providing us with a set of tools to measure what works and doesn't work when it comes to the parts and pieces of our websites, but now it just does that for advertising. Also we do have analytics in the API space, but due to the other areas cited above, there is no sharing of this wisdom across the space.
These are just a handful of areas I regularly see working against the API design, definition, and hypermedia areas of the space, and will flood in slow the progress of their web API ecosystem vision. It doesn't mean I'm not supportive. I see the essence of a number of positive things present in their proposal, like reuse, sharing, and measurement. I feel the essence of existing currents in the world of APIs like microservices, DevOps, and continuous integration (aka orchestration).
My mission, as it has been since 2010, is make sure really smart folks like Ruben and Michel at institutions, startups, and the enterprise better understand the business and political currents that are flowing around them. It can be very easy to miss significant signals around the currents influencing what is working, or not working with APIs when you are heads down working on a product, or razor focused on getting your degree within an institution. The human aspects of this conversation are always well cited, but I'm thinking we aren't always honest about the human elements present on the API side of the equation. Web != API & API !=Web.
APIs can help make technology better serve us humans when you execute them thoughtfully. This is one of the main reasons I kicked off API Evangelist in 2010. I know that many of my technologist friends like to dismiss me in this area, but this is more about their refusal to give up the power they possess than it is ever about APIs.
I have been working professionally with databases since the 1980s, and have seen the many ways in which data and power go together, and how technology is used as smoke and mirrors as opposed to serving human beings. One of the ways people keep data for themselves is to make it seem big, complicated, and only something a specific group of people (white men with beards (wizards)) can possibly make work.
The dominant industrial metaphors for data do not privilege the position of the individual. Instead, they take power away from the person to which the data refers and give it to those who have the tools to analyze and interpret data. Data then becomes obscured, specialized, and distanced.
We need a new framing of a personal, embodied relationship to data. Embodied metaphors have the potential to bring big data back down to a human scale and ground data in lived experience, which in turn, will help to advance the public’s investment, interpretation, and understanding of our relationship to our data.
DATA IS A MIRROR portrays data as something to reflect on and as a technology for seeing ourselves as others see us. But, like mirrors, data can be distorted, and can drive dysmorphic thought.
This is API for me. The desire to invest, interpret, and understand our relationship to our data is API design. This is why I believe in the potential of APIs, even if the reality of it all often leaves me underwhelmed. There is no reason that the databases have to be obscured, specialized, and distant. If we want to craft meaningful interfaces for our data we can. If we want to craft useful interfaces for our data, that anyone can understand and put to work without specialized skills--we can.
The problem in this process is often complicated through our legacy practices, the quest for profits, or there are vendor-driven objectives in the way of properly defining and opening up frictionless access to our data. Our relationships with our data are out of alignment because it is serving business and technological masters, and do not actually benefit the humans whom it should be serving.
One of the reasons I enjoy profiling APIs is that they give an honest view of what a company does, absent of all the marketing fluff, and the promises that I see from each wave of startups. If designed right, APIs can provide a very functional, distilled down representation of data, content, and algorithmic resources of any company. Some APIs can be very fluffy and verbose, but the good ones are simple, concise, and straight to the point.
As I'm profiling the APIs for the companies included in my API monitoring research, what API Science, Apica, API Metrics, BMC Software, DataDog, New Relic, and Runscope offer quickly become pretty clear. A simple list of valuable resources you can put to use when monitoring your APIs. Crafting an OpenAPI Spec allows me to define each of these companies APIs, and easily articulate what it is that they do--minus all the bullshit that often comes with the businesses side of all of this.
I feel like the detail I include for each company in an APIs.json file provides a nice view of the intent behind an API, while the details I put into the OpenAPI Spec provide insight into whether or not a company actually has any value behind this intent. It can be frustrating to wade through the amount of information some providers feel they need to publish as API documentation, but it all becomes worth it once I have the distilled down OpenAPI Spec, giving an honest view of what each company does.
I was playing with a new API design and deployment solution, from some of my favorite developers out there this weekend called Materia, which bills itself as "a modern development environment to build advanced mobile and web applications"--I would add, "with an API heart".
Materia is slick. it is modern. While very simple, it is also very complete--allowing you to define your underlying data model or entities, design and deploy APIs, and then publish a single page applications (SPA) for use on the web, or mobile devices. Even though I'm one of those back to land, hand-crafted API folks, I could see myself using Materia to quickly design and deploy APIs.
I say this in the most positive light imaginable, but Materia reminds me of the Microsoft Access for APIs. Partly its the diagramming interface for the entities, but it is also the fact that it bridges the backend to the frontend, allowing you to not just design and deploy the database and APIs, but also the resulting user interface that will put them to work.
I know they are just getting going with developing Materia, but I can't help but share a couple things I'd like to see, that would make it continue to be the modern API driven application it is striving to be:
- OpenAPI Specs or Blueprints - Allow users to import, export, and manage my APIs in the popular API definition format my choice.
- Schema.org - Provide users with a wealth of existing entity models to choose from, so they do not reinvent the wheel.
- Github - Allow for the publishing of projects, and importing of them to and from Github, allowing for the sharing of server design and deployment patterns.
There are a number of other things I'd like to see, but I'm sensitive to the fact that they are just getting started. These three areas would significantly widen the initial audience for Materia beyond the developer class, which is who the application solution should be targeting. Like I said, it has the potential to be the Microsoft Access of APIs for small businesses, which isn't quite the Microsoft Excel of APIs, but a close second. ;-)
Nice work guys! It is the another positive advancement in the world of API design alongside Restlet launched their API design studio, and Apiary setting this modern era of API design into motion with Apiary. I'll be tuning into Materia's evolution on Twitter, and play more with the server and designer editions available on Github.
My friend Arnaud Lauret (@arno_di_loreto), the API Handyman, has released a very cool new project called the API Stylebook--a collections of resources for API designers. It is a brilliant aggregation of twelve API design guides from Atlassian, Cisco, Cloud Foundry, Haufe, Heroku, Microsoft, PayPal, Red Hat, The White House, Zalando.
I think the API Stylebook purpose describes it well:
The API Stylebook aims to help API Designers to solve API design matters and build their API design guidelines by providing quick and easy access to selected and categorized resources.
In this first version, the API Stylebook provides direct links to specific topics within publicly available API Design Guidelines. Instead of reinventing the wheel or searching google for hours, API Designers quickly can find solutions and inspiration from these existing guidelines.
The API Stylebook isn't just a list of API design guidelines from these companies, it is a machine readable distillation of the twelve API design guides into a master list of API design topics you can consider when crafting your own API design guide. It is slick. I like Arnaud's approach to analyzing the existing API design patterns across API platforms who have shared their guides. I also really like the approach to breaking things down as YAML, and publish as a very good looking website using Github, and Github Pages.
I'm hoping it is something that will stimulate other API providers to craft and share their own API design guide. If you do, I'm sure Arnaud would add to his work, especially if you use his work as a blueprint for your design guide--keeping things as standardized as possible. Projects like this are critical to the growth of the space, and a lack of resources like API Stylebook, and Webconcepts.info have seriously slowed the consistency and interoperability across the world of APIs.
In the last couple of years, I've seen the concept of API design go from being something the API elite discuss, to something that involves business users, and something that has spawned a whole ecosystem of service and tooling providers.
This API design research covers the concepts people often associate with API design, like REST, and Hypermedia, but also touches on other media types, API definitions, and many of the services, tooling and other solutions that have emerged to serve the space.
I fund my research through consulting, selling my guides, and the generous support of my sponsors. I appreciate your support in helping me continue with my work and hope you find it relevant in your API journey.
I am working on one possible API definition for the Human Services Definition Specification (HSDS), and the next phase of this work involves bringing in a small group of technical, and non-technical folks to have discussions around their API designs, in context of the master specification I am looking to pull together.
To help drive the discussion I am wanted to use the OpenAPI Specification that I created for HSDS, and I also knew I wanted to use Github issue management to help keep track of the synchronous, and asynchronous conversation that would occur around it. However Github tends to have a perceived barrier to entry for many non-developers (which I don't fully get), so I wanted to leverage the platform, but also soften the way everyone discussed the overall specification, as well as each individual endpoint.
The HSDS specification is pretty verbose, and I needed a way to have conversations at the high level, but also down to the endpoint level. To help facilitate this, I got to work on a prototype micro tool which enables a conversation around any API(s) that are indexed within an APIs.json file, producing a human readable list of endpoints, parameters, etc., but then uses Github issue management as the core of the conversation.
Resulting in my APIs.json Annotation tool. It is just a first draft, so there will be a lot of changes that need to occur. I'm going to test it against 20+ APIs.json collections I have defined as part of my API Stack work to try and harden it a little bit. My APIs.json Annotation tool runs in my single repo app style, leveraging Jekyll + Github Pages + Github API to operate--Github is the front and backend.
Anyone can view the conversation, but if you want to participate you have to be added to the Github repository, and pass in a Github personal token. This is something I often automate with a simple Github login, where I use OAuth.io to obtain token, but I kind of see the token as a minimum viable investment to understanding Github for using each tool.
It is really important to me that each app stands on its own feet. Not all of my micro tools that I develop in this way will enjoy zero outside dependencies, but most of them can be easily forked, and ran in any Github user account or org (with little or no setup). Conversations around API is just one area I am looking to simulate with this approach to delivering tooling, and specifically APIs.json tooling, that can be used throughout an API life cycle.
You are welcome to play with the APIs.json Annotation, or any of the other tools I have listed on my home page. I will keep adding them here, so that they can be found, but like all my work they are all a work in progress. Each tool has its own dedicated repo and issue management, where you are welcome to join in the conversation around the road map for each one. I am just looking to develop a robust toolbox of small micro tools that will help be more successful across the life cycle of the APIs I am working on, but maybe you can benefit using them too.
Over the last five years many of us have been pushing forward our API design skills to deliver valuable resources to mobile apps. The multi channel opportunity for delivering data, content, and other resources to not just websites, and web applications, but also to iPhone, Android, Windows, and other mobile platform is clear. Alongside delivering web apps to multiple browsers (IE, Chrome, and Firefox), we had to learn to deliver to an increasingly diverse mobile ecosystem (iOS, Android, Windows).
When you have an API core, crafting new endpoints that speak to new channels is not as daunting of a task. I'm thinking through how to deliver API Evangelist knowledge into a Slack channel, Facebook Messenger discussions, and via Alex Voice for use at our APIStrat Conference in Boston this fall. Whether I am creating bots, or delivering new skills to voice enablement, I'm just adding new design constraints to my already existing stack of things I consider when I'm crafting my API stack(s).
I can't help but be in tune with the frenzy around bots, and the interesting things AWS is doing within the Alexa ecosystem. I do not feel like it is a distraction from my core business, as I'm just applying some new design constraints to what I am already doing. I'm not chasing entirely new things, I'm just responding, and participating in what the latest shifts in what technology are.
I don't feel like any of the 50 areas of the API space that I'm keeping an eye on are distractions, as long as I stay true to my API core, and they help me critically think through, and apply new design constraints to the core business value I am already bringing to the table.
It always makes me smile, when I talk to someone about one or many areas of my API research, sharing how I conduct my work, and they are surprised to find how many areas I track on. My home page has always been a doorway to my research, and I try to keep this front door as open as possible, providing easy access to my more mature areas like API management, all the way to my newer areas like how bots are using APIs.
From time to time, I like to publish my API life cycle research to an individual blog post, which I guess puts my home page, the doorway to my research into my readers Twitter stream, and feed reader. Here is a list of my current research for April 2016, from design to deprecation.
I am constantly working to improve my research, organizing more of the organizations who are doing interesting things, the tooling that is being deployed, and relevant news from across the area. I use all this research, to fuel my analysis, and drive my lists of common building blocks, which I include in my guides, blueprints, and other white papers and tutorials that I produce.
I am currently reworking all the PDF guides for each research area, updating the content, as well as the layout to use my newer minimalist guide format. As each one comes off the assembly line, I will add to its respective research area, and publish an icon + link on the home page of API Evangelist--so check back regularly. If there is any specific area you'd like to see get more attention, always let me know, and I'll see what I can do.
As I made my way through nine of the leading SMS providers, profiling the details of their API plans, trying to bring it all together into a single, machine readable definition, Messente's pricing API stood out for me. Alongside their SMS API, and their number lookup API, they have a pricing API. As I take a moment to look at the simplicity of the endpoint, I am also thinking about the potential, and the details of their API docs really stand out for me.
In addition to a pricing API being a valuable utility for any API consumer, I can't help but think the process of constructing a pricing API would force API providers to think another layer deeper in their API plan(s). Similar to how you have to think through the details of any resource you make available via an API, considering how it will be put to use, crafting a pricing API would force you to do the same for one of the most valuable, and overlooked resources you offer, that is core to all of your API operations.
I've talked about the importance of API providers having management level APIs before, and I think they should also follow the lead of API providers like Messente who exist in a pretty mature sector, and have a handle on their API pricing. I've said it before, and will keeping harping on this issue...that all API providers and their service providers should have pricing APIs, as well as APIs for the other essential building blocks of their API operations.
Another thing to note here, that I will be writing about more in the future, is the presence of two-letter country code as a parameter for an API. While not all APIs will have country specific pricing like SMS providers do, the API providers who are looking to truly operate globally, you will need to continue the automation of your operations, making pricing available on demand, in any currency that consumers will need.
API design and definitions are the number one area when it comes to talks submitted for APIStrat 2015 in Austin, and when it comes to traffic across the API Evangelist network in 2015. After diving into the Amazon API Gateway a little more over the weekend, I was reminded of the opportunity out there when it comes to API design tooling.
Amazon did a good job, providing a GUI interface for crafting the methods, resources, and underlying data models for APIs you are deploying using the gateway. However when you compare to some of the GUI API design editors I mentioned in my last post on this subject, from Restlet, APIMATIC, and Gelato, the Amazon interface clearly needs evolve a little more.
AWS is just getting started with their solution, so I'm not knocking what they have done. I just wanted to keep comparing of all of the solutions as they emerge, and highlight the opportunity for some standardization in this layer of API tooling. I see a pretty big opportunity for some player to step up and provide an open source API design editor that provides a seamless experience across API service providers.
This post is nothing new. I am just trumpeting the call for open API design tooling each time I see another new interface introduced for crafting APIs, their paths, resources, parameters, headers, authentication, and underlying data models. At some point, a new player will emerge with the open source API design editor I am looking for, or one of the existing players open sources their current offering, and evolve it in context of the wider API design space, providing an abstracted layer that supports all API definition formats.
With the growth in the number of service providers I see stepping up to server the API space, the need for common, open tooling when it comes to API design is only going to grow. It took almost four years of waiting for the API management space to figure this out, I'm hoping I don't have to wait as long in the API design side of things.
There was a pretty interesting conversation around API design going on in one of my API slack channels over the last couple days, about what is API design, and what is needed to get developers, and even non-developers more aware of the best practices that exist. It is a private group, so I won't broadcast it as part of this post, but I did want to extract a narrative from it, and help me set a bar for other API design discussions I am having.
The Restafarian, hypermedia, and linked data folk have long been frustrated by the adoption of sensible API design practices across the sector, and the lack of basic HTTP literacy among developers, and non-developers is at dangerously high levels. The good news, is some of this is beginning to change, but we still have so much work to do, something that won't be easy, and unfortunately it won't always have the clean outcomes leaders in the space are looking for.
APIs returning JSON is just the next step in the evolution of the web, and when you consider how much work it took to get everyone on-board with HTML, and building web pages, then web apps, and recently mobile apps, you can begin to understand the work that still lies ahead. We have to help the next generation of developers be more HTTP literate (something the previous generations of developers aren't), and possess a baseline knowledge of common API design best practices. This needs to be done in a world where many of these developers really aren't going to care about the specifics of good API design, like us API architects and early pioneers have.
The average API designer in the future will not always be willing to argue about the nuance of how to craft URLs, whether to use a header or parameter, caching, and how to version. They just want to get the outcome they seek, accomplish their project, and get paid for their work. Consider the solar industry as a quick comparison. The first wave of installers were domain experts, while the following waves of installers who will be focused on scaling and maintaining the industry, will only need to be trained on only what is required to get the job done in a stable, profitable way.
Ok. So how do we do this right? I feel like we are already on the good path. We just need you to publish your own API design guide somewhere that we can all learn from, like other leading API providers already present in my API design research. As we build a critical mass of these, we need to also work to aggregate the best practices across them, so that instructors and textbook publishers can incorporate into their curriculum. If you have an API platform, and have ever wished that there were more highly skilled API designers out there, make sure you have your API design practices documented, and shared with the world.
This will get healthy API design practices out of the trenches of startups, SMBs, enterprise, and government agencies, and get it into the educational institutions around the world. Then we can start equipping the next generation of programmers with the knowledge they will need to be successful in delivering the resources need for the next generation of Internet powered apps, networks, and devices.
I want to add on emore thing. API service companies who are looking to provide tooling that API providers can use to do deploy APIs, will have share in the load here. This is core to my criticism of the AWS API Gateway, in that I applaud their use HAL, but please make sure you also provide a healthy dose of hypermedia literacy along the way, don't just hide it behind the curtain. I really do not want to see the another FrontPage for APIs, so if you are building an API editor, let me know so I can provide you with some ideas. (1) (2).
We all have a lot of work to do, in preparing the next generation of developers, and business users when it comes to a baseline of HTTP literacy, as well as a healthy dose of API awareness. We are going to need an army of API designers to help us deliver on the API economy we are all seeing in our heads--so let's get to work. If you do not have a formal API design strategy get to work on one (let me know if you need help). If you have one, please share it, so I can add it to my API design research, for others reference.
I walked around San Francisco with Jakub Nesetril (@jakubnesetril), the CEO of Apiary, Wednesday evening, talking about the API space. Eventually we sat down in Union Square, continuing our conversation, which is something I wanted to further process, and blend with some existing thoughts I'm working through. Much of our conversation centered around the need for an open abstraction layer for API design, which would reduce the focus on Swagger vs. API Blueprint vs. WADL vs. RAML vs. any other API definition, and make it just about defining and designing our APIs.
Jakub is right, I'm sure he'd love everyone to use API Blueprint (which thousands are), but it is more important to him that people just use API definitions, and commit to a healthy API design strategy. This line of thinking is in alignment with other thoughts I'm having around there being a common open source API design editor, which I'd like to use as vehicle to get us closer to my vision of a perfect API design editor.
I see an abstraction layer consisting of the following elements:
- Import - We need to be able to import ANY API definition format we desire.
- Export - We need to be able to export ANY API definition format we desire.
- Viewer - We a code view for ANY API definition format we want to look at while working.
- Editor - We a visual, GUI editor that is all about visual API design, WYSIWG editor for APIs
API definition format owners should be able to maintain importing, exporting, and viewing layers via some sort of plug-in architecture--meaning the platform is about API definition, the WSDL, WADL, Swagger, API Blueprint, RADL, RAML, Postman Collection, and other formats should be owned by each respective owner. The Open API Abstraction project could provide a single architecture for everyone to plug into, with an emphasis on building the best possible API design editing experience possible.
The Open API Abstraction tooling could be baked into any API service or tooling in the space, allowing the next generation API service providers to focus on what they do best, and not having to build out their own API design editor, while also baking in compatibility for all API definition formats by default. Such a layer would allow API architects and designers to craft APIs in a consistent way, no matter what API definition format designers and developers might be speaking, opening up a wider world for communicating and collaborating through the API design process.
I'm processing all of my thoughts from some travel to the big city of San Francisco. I was providing feedback on Microsoft's API design guide, as part of the OneAPI Technical Advisory Group. As I was thinking about the journey Microsoft is on, the role of the API design guide, and how many other companies like Paypal, and Cisco, are on the same journey.
In parallel to this, I am on my own journey with my own API stack, I've been looking at everything from a slightly different perspective than many other analysts and providers in the space. When I started in 2010, it was all about API management, then after folks kept asking me about options for deploying APIs I expanded my monitoring to API deployment. Then Jakub the CEO of Apiary, moved the dial back ruth on the life-cycle, and got me paying attention to the concept of API design.
Fast-forward to 2015, I am paying attention to almost 20 separate areas, as part of my core API research. I tune into a number of other areas, but these research projects make up the heart of my API storytelling. On the trip back from San Francisco today I had a few thoughts, and needed to organize them in context of my core research.
- A1 - Tech
- Do you use Swagger to define APIs?
- Do you use API Blueprint to define APIs?
- Are there other API definition formats you use?
- Do you employ NO API definitions?
- A2 - Business
- Do you support multiple API definition formats?
- Do you use API definition formats to engage with other services or tools?
- Are your service tiers represented in your aPI definitions?
- A3 - Politics
- Do you share your API definitions publicly?
- How do you license your API definitions? Are they open?
- B1 - Tech
- Do you use Restlet Studio's API design tools?
- Do you use APIMATIC API design tools?
- Do you use Apigee API Studio API design tools?
- Do you use Gelato API design tools?
- B2 - Business
- Do you use a service provider for API design?
- Do you use a consultant(s) for API design?
- Do partners and other stakeholders participate in the API design process?
- B3 - Politics
- Do you JSON:API?
- Do you hypermedia?
- Do you JSON-LD?
- C1 - Tech
- What gateway tech do you use?
- What API frameworks do you use?
- What cloud solutions do you use?
- C2 - Business
- Are you monetizing APIs?
- Are you covering costs of deployment?
- Do you hand API deployment internally?.
- Do you depend on outside resource to deploy APIs?
- C3 - Politics
- Are your API deployment solutions open source?
- Do Your API Deployment resources scale?
- D1 - Tech
- Are you rolling your own API management solution?
- Which API management solution are you using?
- D2 - Business
- Do you require registration to use an API?
- Do you have service tiers?
- Do you have partner levels of access?
- Do all areas of your companies operation apply same approach?
- D3 - Politics
- Is your API management solution open source?
- Do you provide support for your API developers?
- E1 - Tech
- Do you require SSL?
- Do you require API keys?
- Are you employing OAuth?
- E2 - Business
- Are you investing in security?
- What would a security breach cost you?
- Do all areas of your companies operation apply consistent template?
- E3 - Politics
- Are you more secure than your competition?
- Do you share your security practices publicly?
- F1 - Tech
- Which management solution do you use to handle monetization?
- Is there an API for your API monetization layer?
- Is there a developer dashboard for pricing, billing, and revenue sharing?
- F2 - Business
- Do you apply consistent monetization strategy across all APIs?
- Is your billing real-time?
- Is API revenue, your only revenue stream?
- Do you have a credit based system, beyond just API call based?
- F3 - Politics
- Do you share your pricing publicly?
- Is your partner program transparent?
- Do developers have dashboard for managing billing?
- G1 - Tech
- What services to you use for monitoring APIs?
- What open source tooling do you use for your monitoring?
- Does your monitoring include testing, performance, and security?
- G2 - Business
- What do you spend on API monitoring each month?
- Is there a dedicated person(s) to monitoring APIs?
- What has outages cost you in the past?
- G3 - Politics
- Do you publish your monitoring reports publicly?
- Do you keep your ecosystem in tune with monitoring via messaging system(s)?
- H1 - Tech
- What services to you use for testing your APIs?
- What open source tooling do you use for your testing?
- What are your benchmarks?
- H2 - Business
- Do you provide testing tools to your developers?
- Can developers request specific types of testing for APIs?
- H3 - Politics
- Do you publish your monitoring reports publicly?
- I1 - Tech
- What services to you use for testing your APIs?
- What open source tooling do you use for your testing?
- What are your benchmarks?
- I2 - Business
- Do you have SLAs for any tiers of operation?
- Do you generate any revenue from SLAs in place?
- I3 - Politics
- Do you publish your monitoring reports publicly?
- Do you consistently meet your SLAs?
- J1 - Tech
- What services to you use for virtualizing your APIs?
- What open source tooling do you use for your API virtualization?
- J2 - Business
- Is virtualization part of your QA process?
- Do you virtualized instances of your API as an added service?
- Do you provide a sandbox or simulators for developers by default?
- J3 - Politics
- Do you provide virtualization for developers?
- Are your virtualization images openly sourced?
- K1 - Tech
- What services to you use for virtualizing your APIs?
- What open source tooling do you use for your API virtualization?
- K2 - Business
- Do you have dedicated people to managing your API architecture?
- Do you have dedicated services for managing your API architecture?
- K3 - Politics
- Can you migrate between infrastructure providers? (ie. AWS to Google)
- Is your server side API code open source?
- Are you virtualization images openly licensed and available? (ie. Docker Images)
- L1 - Tech
- Do you use oEmbed?
- Do you have bookmarklets?
- L2 - Business
- Do you have a standardize strategy for allowing users to embed API driven resources?
- Is your embeddable strategy integrated with your overall marketing and branding efforts?
- Do you offer embeddable tool builder?
- L3 - Politics
- Are your embeddable tools non-invasive? Protect privacy?
- M1 - Tech
- Do you have robot evangelists? Just sounded cool, and couldn't think of anything to put here.
- M2 - Business
- Do you have dedicated evangelist resources?
- Do you contract with 3rd parties for any evangelist resources?
- Is your evangelism coupled with your marketing?
- M3 - Politics
- Are there opportunities for developers to get involved with evangelism?
- Do you have healthy feedback loop present with your API operations?
- N1 - Tech
- Do you have an APIs.json for your API operations?
- Do you employ API definitions?
- Are you using hypermedia?
- N2 - Business
- Are your APIs plug and play with other platforms?
- Are your APIs public or private?
- N3 - Politics
- Does discovery feed every other layer of API life-cycle?
- Does discover play into your security strategy?
Sorry, I don't mean to be a downer. But...we are just getting started with stabilizing how we do APIs. I feel like we are beginning to formalize how we manage or APIs (thanks Mashery, 3Scale, and APIgee), and we are getting a handle on API deploy (thanks Restlet, Amazon), and we are deep into understanding how we define (thanks Swagger and API Blueprint), and ultimately design (thanks Apiary, Restlet, and Apigee).
We are moving fast into testing, monitoring, performance, and virtualization (thanks Runscope, APITools, API Science, and SmartBear), but our security sucks, embeddability has stagnated, and evangelism and discovery really isn't improving. I'm working on APIs.json, and bringing together the Swagger and API Blueprint communities, but there is so much work left when it comes to discovery, and automating each of the areas listed above.
There really is no point to this post. It is my mental vomit, after a trip to San Francisco, and working on Microsoft's API design guide. Stay tuned for how any of this applies to anything.
I spent all day Wednesday, at Microsoft offices in San Francisco, providing feedback on the Microsoft API design guide, as part of the OneAPI Technical Advisory Group. The OneAPI team had already done most of the hard work in hammering out the API design guide, by working with the API leadership from groups across Microsoft--we were just brought in to provide outside perspectives.
A group of about 20 of us, spent the entire day walk through the high levels of the Microsoft API Design philosophy. The Microsoft OneAPI design guide is a draft, so they aren't ready for us to share it publicly, something we'll see in the near future. However, document being ready or not, the process showed me that Microsoft is really working hard to get their API design strategy house in order. They had worked hard to iterate through all the common areas of API design, with their various teams--even some of the more controversial areas like versioning.
I did not feel like I had a lot to contribute to the process. They have some really good API designers, and there was plenty of high quality API design talent in the room, to provide the feedback they wanted. When Microsoft is ready for more of the management, evangelism, and other areas more in the business and politics of APIs, I will have much more to bring to the table. I did however, provide some insight that I think could help the overall process, and will continue to provide feedback--which is why I'm gathering my thoughts in this series of posts.
Prior to me participating in the OneAPI Technical Advisory Group, I had just published the API design guides for Cisco and Paypal, as part of my API design research. Bringing the number of API design guides to 9, which is a good sign the space is getting more serious about standardizing how we do API design. For me, there are two important things going on here:
- Company Journey - The companies own API journey, allowing them to collaborate and craft a single API design philosophy, with the intent of making it a company-wide initiative.
- Publicly Sharing - Declaring to the public, this is how we design APIs. Something that allows others to learn from, and merge with their own API design philosophy--benefiting the entire space, while also establishing leaders and followers.
Crafting a document that defines how a company builds software is nothing new. However, APIs aren't just about building software, they are about defining your companies resources, and when you work to standardize how you define your resources as APIs, you are focusing on the core of your business. Next comes the value when building websites, web and mobile applications, and potentially devices as well. Getting your API design house in order, is all about standardizing how you speak API across your company, something that touches every product and service, and is the foundation for your corporate digital strategy.
My participation in the OneAPI Technical Advisory Group gave me a unique glimpse at the API strategy unfolding at Microsoft. While there is still a lot of work to be done, the fact that they are working on a central API design strategy demonstrates to me they are further along in their API journey than I thought they were. Where are you in your own API journey? Does your organization have an API design guide? How do you share this knowledge across your organization, and with the public?
As I monitor the API space, I am always looking for the signals of a healthy API strategy, and I think, when I see a formal API design guide present for any company that I track on, I am going to tack on a couple extra points. A polished, publicly shared API design guide shows organizations are a further along in their overall API journey, than other companies in the space, who are still trying to figure things out.
I love the API design editors that have emerged from Restlet as part of their design studio, and from APIMATIC as part of their SDK management service, and if you haven't checked out Gelato.io, I highly recommend it. All of these solutions provide me with a well designed user interface for working with common API definitions, but even with these offerings, I feel like the overall API universe is very deficient in this area.
It is good that these providers are using common API definitions like Swagger and API Blueprint, but the UI itself is a one-off, and for API definitions to take their place, front-and-center throughout the API lifecycle, we are going to need some common API design tooling. Any successful solution needs to go well beyond just the JSON, YAML, and Markdown, and give us a powerful user interface to craft exactly the APIs we need.
There are several other API definition editors that let you directly edit JSON, YAML, and Markdown, but to move things forward, we need an easy to use, standardized, open source, API interface interface that will work across common API definition formats. Think of what the WYSIWYG editors did for web design--we need a similar, elegant and modern solution for API design.
I have had this conversation with several API service providers, but I wanted to expand the conversation, and see if I can stimulate the development (or release) of an open source API design editor. I am looking for a solution that any API service provider can deploy within their management solutions, encouraging them to not just speak in common API definition formats, but do it in a way that developers, and non-developers alike, can participate at every stage of the API design life-cycle.
As I migrate the version 1.0 of my internal systems to an API-first version 2.0, one of the challenges I face is the optimization of some of the features in my own system. I have numerous systems I crafted to solve a specific need, at a specific point in time, with very little consideration about the most efficient design.
About 50% of my version 1.0 systems employed APIs, many systems relied on directly connecting to my MySQL databases. One example of this is my main news curation screen on my dashboard. This was the replacement for Google Reader about 6 months before they killed it off. My reader system is very chatty, every action I take against a single post (archive, curate, tag, etc.) initiates a round-trip to the database--something when you translate to API calls, becomes a very huge blip on my API analytics.
I treat all my internal apps, just like I would one of my API consumers, and make them sign up for key, and exist within my API service tiers. There are numerous reasons for this, but in the end, it helps me see things through an external lens. When I got my report on my usage, I noticed my curated API usage was through the roof. This is good right? No! It means I have super chatty features in my application, something I need to refactor.
My curation may involve 1000 to 3000 news articles which needs to be curated or archived, resulting in potentially 3000 API calls, and when you add on applying tags, and managing other details, I can be in the 5,000 to 10,000 range pretty quick. Its obvious I need a better approach, so I'm developing a new queue based system that lets me work with 25 news stories at once, and submit the changes as a single bulk API call--significantly reducing the number of calls I make.
In the end, my API rate limits are forcing me to think through my app design, and how I make API calls. It also is helping me rethink how I design my APIs. I want each API call to be as efficient as it can from app to server. To quote my friend Tyler Singletary (@harmophone), "rate limits on APIs pushes developers to be more creative". Something I agree with 100%, if it truly is a transparent conversation between API provider and consumer--which in my private, schizophrenic ecosystem is pretty easy to achieve.
I was wallowing in the exhaust of my previous story on Amazon API Gateway, and talking with an API development team about their preferred way to launch an API today. Leaving me thinking...what is the better way to design, and deploy an API? Gateway or hand-crafted? Ok, cut to the chase, its a bullshit question, but for the sake of working through my own anxiety, I'm going to explore.
For my audience, I'm going to oversimplify what an API gateway is by stating it is a hardware and / or software appliance you can you use to connect to any existing, internal system like a database, CRM, or other, and then quickly and efficiently design, deploy and manage APIs driven by these resources.
Personally I am from the artisan, farm to table guild of API architects--I prefer to hand craft my APIs. Well OK, that is bullshit--I use an API framework, but I wire it up by hand to whatever is the original resource is (database, file-store, other API). This is just my style, and probably most aligned with the fact that I cannot afford any API gateway--something the release of Amazon API gateway has me thinking more about.
My overall opinion of the gateways I've used is that they lack the amount of granular level control I desire--maybe this is just me. However this is something that is being re-examined for me because of the emergence latest cloud-based API service providers like Restlet APISpark, Wavemaker, DreamFactory, Apitite--and AWS. Honestly I think Amazon will skew the concept of what is an API gateway with their new offering, they just took the name to sell it to the enterprise.
I guess I didn't even have a real point to this story. I just wanted to talk to my API therapist about my own enterprise API gateway baggage and PTSD, and justify my hippie dippe, artisan approach to APIs. Sucka...I got you to listen!! In the end I think I support your API offering, by any name, as long as you help move the conversation forward, which I have to admit I think AWS is doing with their new AWS API Gateway offering--even though they used that name!!!
I told a contact of mine at the Environmental Protection Agency I would provide some feedback for the next iteration of the Envirofacts Data Service API, and as I prepare my formal response, there is a huge opportunity for me to carve off some other smaller stories. These stories are what make API Evangelist go around--pulling off these nuggets of learningz, as I'm working with API various providers.
A concept I work hard to help API designers and architect adopt is around experience-based API design over resource based API design. Something I learned from my friend Daniel Jacobson (@daniel_jacobson) at Netflix, but have been taking it in different directions to help designers and architects from smaller API groups, adopt this way of life--I feel like my work today on the Envirofacts Data Service API provide a great opportunity to move this conversation forward.
When you land on the Envirofacts Data Service API page, you are given a single base URL to employ when "constructing a search" for the API:
Ok, not the best API design, a little confusing, but where do I find the list of possible values for every potential path variable? I scroll down and see some examples, but I have to read the opening paragraph again, before I find a link to the data model. Once I click on the text link, I'm faced with 20 separate "tables"--which in reality are 20 separate APIs, all housed under the Envirofacts Data Service API. (who named this??)
This is resource based API design. You craft an API, that either resembles, or in this case is exactly named just like your database, and its fields. This type of API design, while very appealing to database administrators (DBA), but will leave any new API developer and potential "customer", pretty lost, left with having to work overtime just to understand what is possible. (you can do more work on my behalf)
I had no idea the Envirofacts Data Service API had such a wealth of resources. The overall API landing page, and API design does not tell the story of the value that lies within. Anyways, I'm just embarking on the profiling of this API, and providing a formal review that the team can put to use in their efforts. I'm not trying to bash their work, just help demonstrate to other folks, what I mean when I say "resource based API design", furthering my work to help folks design APIs more focused how they'll be used, not just where they come from, only ticking the box on the task you were given--launch API.
I recently went through my API design research, updating and evolving it, to help me better understand changes in the API design space, while also sharing as much of the information as I can with my readers. One of the resources I include under my tools section are the increasing number of API design guides being published.
Currently I'm showcasing the API design guides for 18F in the federal government, Heroku, and PayPal, then the other day my friend Holger Reinhardt (@hlgr360), formerly of CA Technologies, now the CTO at the Haufe Group in Germany, shared his teams API design guidie with me. The executive summary for the guide says it well:
Purpose of this style guide is to gather a list of rules, best practices, resources and our way of creating REST APIs in Haufe Group. The style guide addresses API Designers, mostly developers and architects, who want to design an API. Intention is to ease the design process by providing helpful rules to create a successful API that your customers will love.
One thing I really like, is they also added "please follow the guidelines but don't follow blindly! you can break the rules with justification". Overall the API design guide is full of very pragmatic API design theory, much of what I've heard before, but also emphasizes some very import, forward thinking aspects of modern web API as well.
The design guide leads with some very sound API wisdom:
- You should think of an API like a product. Indeed a Business API is a product.
- Design your API with a customer, or at least with a customer in mind and Get Feedback.
- Do not implement API Management (use existing services)
- Every API MUST be described using a formal API description language
- API clients MUST use a self-service API portal to get access to the API via an API Key
After that, they do not miss a beat going into REST principles stating: "All our REST APIs MUST support hypermedia." The way they lay out their API design best practices makes everything very accessible, leaving hypermedia seeming a very sensible practice, not a bleeding edge design theory. I'm going to try and go through each section and apply to my own internal API design practices, there are several things here I could do so much better, by following.
I may even begin work on my own API design guide, based upon what I've learned from the Haufe Group, 18F, Heroku, Paypal, and others. Not that my version would be any better than the sources, but I'm feeling the process of parsing the work of others, and assembling into my own document will be a healthy journey all by itself. An API design guide is quickly becoming an essential building block for API providers of all shapes and sizes, and publishing it publicly via Github is becoming a badge of leadership and transparency for the rest of the API community.
When API Evangelist began five years ago it was a single research into the business of APIs, which ultimately became a research project which I called API management. Over the last four years, I have spliced off other areas of research including the elements of my core research into API design, deployment, management, integration, evangelism, and monetization.
My goal with each research project, is to evolve my understanding of each area of the API space by identifying the key companies and organizations involved, the open tooling being developed and put to work, and some of the common building blocks that make up each area. For each of my research projects, you can visit its Github repository to get regular updates, but I find many people seem to prefer browsing a single, portable paper of my research.
Each year I usually published a PDF version of each research project, something I have always intended to update regularly. However it seems, this is usually is an annual thing that I find time for in the summer. To helpl change this, I've evolved my own tooling for publishing for what I considers guides to my research areas, allowing me to more easily publish the information and content I regularly curate about the space.
I just finished v1 of my paper API, which allows me to aggregate pages, organizations, building blocks, tools, and news that I curate, into field guides for my research. Using this new API, i intend to update each of my core research areas each month, getting more disciplined about distilling my research down to a single, coherent story.
Here is a copy of my API design research as a single PDF guide, you can also find it on the main navigation for my API design research. Now that I have my paper API stabilized, I will going through API deployment, and management, publishing a research guide for each. This process couldn't have come at a better time, as I'm carving out a couple of new research areas for API performance, testing, and monitoring, and I would like to be able to quickly publish guides for each of those areas as well.
If you think there is a link I should have listed here feel free to tweet it at me, or submit as a Github issue. Even though I do this full time, I'm still a one person show, and I miss quite a bit, and depend on my network to help me know what is going on.