These are the news items I've curated in my monitoring of the API space that have some relevance to the API design conversation and I wanted to include in my research. I'm using all of these links to better understand how the space is designing their APis and the underlying schema used as part of their requests and responses.11 Apr 2017
Establishing a feedback loop with your API community is one of the most valuable aspects of doing APIs, opening up your organization to ideas from outside your firewall. When you are designing new APIs or the next generation of your APIs, make sure you are tapping into the feedback loop you have already created within your community, by providing access to the alpha, beta, and prototype versions of your APIs.
The Oxford Dictionaries API is doing this with their latest additions to their stack of word related APIs, by providing early access for their community with two of their new API prototypes that are currently in development:
- The Oxford English Dictionary (OED) is the definitive authority on the English language containing the meaning, history, and pronunciation of more than 280,000 entries – past and present – from across the English-speaking world. Its historical record of the English language is traced through more than 3.5 million quotations ranging from classic literature and specialist periodicals to film scripts and cookery books.
- bab.la offers quick and easy translations and answers to everyday language questions. As part of the Oxford Dictionaries family, it provides practical support to people using a language that is not their mother tongue.
To get access to the new prototypes, all you have to do is fill out a short questionnaire, and they will consider giving you access to the prototype APIs. It is interesting to review the questions they ask developers, which help qualify users but also asks some questions that could potentially impact the design of the API. The Oxford Dictionaries API team is smart to solicit some external feedback from developers before getting too far down the road developing your API and making it available in any production environment.
I do not think all companies, organizations, and government agencies have it in their DNA to design APIs in this way. There are also some concerns when you are doing this in highly competitive environments, but there are also some competitive advantages in doing this regularly, and developing a strong R&D group within your API ecosystem--even if your competitors get a look at things. I'm going to be flagging API providers who approach API development in this way and start developing a list of best practices to consider when it comes to including your API community in the design and development process, and leveraging their feedback loop in this way.
I'm keeping an eye on the different approaches by API service providers when it comes to providing API editors within their services and tooling. While I wish there was an open source GUI API editor out there, the closest thing we have is from these API service providers, and I am trying to track on what the best practices are so that when someone does step up and begin working on an open, embeddable solution, they can learn from my stories about what is working or not working across the space.
One example I think has characteristics that should be emulated is the API Spec Editor from Stoplight. The GUI editor lets you manage all the core elements of an OpenAPI like the general info, host, paths, and even the shared responses and parameters. They even provide what they call a CRUD builder where you paste the JSON schema, and they'll generate the common paths you will need to create, read, update, and delete your resources. Along the way you can also make calls to API endpoints using their interactive interface, helping ensure your API definition is actually in alignment with your API.
The Stoplight API Spec Editor bridges the process of defining your OpenAPI for your operations, with actually documenting and engaging with an API through an interactive client interface. I like this approach to coming at API design from multiple directions. Apiary first taught us that API definitions were about more than just documentation, and I think our API editors should keeping evolving on this concept, and allowing us to engage with any stops along the API life cycle like we are seeing from API service providers like Restlet.
I'm already keeping an eye on Restlet and APIMATIC's approach to providing a GUI API design editor within their solutions and will keep an eye out on other providers as I have time. Like other areas of the API sector, I'm hoping I can develop a list of best practices that any service provider can follow when developing their tools and services.
I'm endlessly fascinated by APIs and enjoy studying their evolution. One of the challenges in helping evangelize APIs that I come across regularly is the many different views of what is or isn't an API amongst people who are API literate, as well as helping bring APIs into focus for the API newcomers because there are so many possibilities. Out of the two, I'd say that dealing with API dogma is by far a bigger challenge, than explaining APIs to newbies--dogma can be very poisonous to productive conversations and end up working against everyone involved in my opinion.
I'm enjoying reading about the evolution in the API space when it comes to GraphQL and gRPC. There are a number of very interesting implementations, services, tooling, and implementations emerging in both these areas. However, I do see similar mistakes being made regarding dogmatic behavior, aggressive marketing tactics, and shaming folks for doing things differently, as I've seen with REST, Hypermedia, and linked data efforts. I know folks are passionate about what they are doing, and truly believe their way is right, but I'm concerned you will all suffer from the same deficiencies in adoption I've seen with previous implementations.
I started API Evangelist with the mission of counteracting the aggressive approach of the RESTafarians. I've spent a great deal of time thinking about how I can turn average developers and even business folks on to the concept of APIs--no not just REST or just hypermedia, but web APIs in general. Something that I now feel includes GraphQL and gRPC. I've seen many hardworking folks invest a lot into their APIs, only to have them torn apart by API techbros (TM) who think they've done it wrong--not giving a rats ass regarding the need to actually help someone understand the pros and cons of each approach.
I'm confident that GraphQL will find its place in the API toolbox, and enjoy significant adoption when it comes to data-intensive API implementations. However, I'd say 75% of the posts I read are pitting GraphQL against REST, stating it is a better solution. Period. No mention of its limitations or use cases where it might not be a good idea. Leaving us to only find out about these from the GraphQL haters--playing out the exact same production we've seen over the last five years with REST v Hypermedia. Hypermedia is finding its place in some very useful API implementations like FoxyCart, and AWS API Gateway (to name just a few), but its growth has definitely suffered from this type of storytelling, and I fear that GraphQL will face a similar fate.
This problem is not a technical challenge. It is a storytelling and communication challenge, bundled with some very narrow incentive models fueled by a male-dominated startup culture, where folks really, really like being right and making others feel bad for not being right. Stop it. You aren't helping your cause. Even if you do get all your techbros thinking like you do, your tactics will fail in the mainstream business world, and you will only give more ammo to your haters, and further confuse your would be consumers, adopters, and practitioners. You will have a lot more success if you are empathetic towards your readers, and produce content that educates, and empowers, over shames and tearing down.
I'm writing this because I want my readers to understand the benefits of GraphQL, and I don't want gRPC evangelists to make the same mistake. It has taken waaaay too long for linked data efforts to recover, and before you say it isn't a thing, it has made a significant comeback in SEO circles, because of Google's adoption of JSON-LD, and a handful of SEO evangelists spreading the gospel in a friendly and accessible way--not because of linked data people (they tend to be dicks in my experience). As I've said before, we should be investing in a robust API toolbox, and we should be helping people understand that benefits of different approaches, and learn about the successful implementations. Please learn from others mistakes in the sector, and help see meaningful growth across all viable approaches to doing API--thanks.
I wrote about how gRPC API implements deliver a tighter API contract, but I wanted to also explore more thought from that same conversation, about how hypermedia APIs can help deliver a more evolvable API contract. The conversation where these thoughts were born was focused on the differences between REST and gRPC, in which hypermedia and GraphQL also came up. Leaving me thinking about how our API design and deployment decisions can impact the API contract we are putting forth to our consumers.
In contrast to gRPC, going with a hypermedia design for your API, your client relationship can change and evolve, providing an environment for things to flex and change. Some APIs, especially internal API, and trusted partners might be better suited for gRPC performance, but when you need to manage volatility and change across a distributed client base, hypermedia might be a better approach. I'm not advocating one over the other, I am just trying to understand the different types of API contracts brought to the table with each approach, so I can better articulate in my storytelling.
I'd say that hypermedia and gRPC approaches give API providers a different type of control over API clients that are consuming resources. gRPC enables dictating a high performance tighter coupling by generating clients, and hypermedia allows for shifts in what resources are available, what schema are being applied, and changes that might occur with each version, potentially without changes to the client. The API contract can evolve (within logical boundaries), without autogeneration of clients, and interference at this level.
As I learn about this stuff and consider the API contract implications, I feel like hypermedia helps API provides navigation change, evolve and shift to deliver resources to a more unknown, and distributed client base. gRPC seems like it provides a better contract for use in your known, trusted, and higher performance environments. Next, I will be diving into what API discovery looks like in a gRPC world, and coming back to this comparison with hypermedia, and delivering a contract. I am feeling like API discovery is another area that hypermedia will make sense, further helping API providers and API consumers conduct business in a fast changing environment.
I was learning more about gRPC from the Google team last week, while at the Google Community Summit, as well as the API Craft SF Meetup. I'm still learning about gRPC, and how it contributes to the API conversation, so I am trying to share what I learn as I go, keeping a record for others to learn from along the way. One thing I wanted to better understand was something I kept hearing regarding gRPC delivering more of a tighter API contract between API provider and consumer.
In contrast to more RESTful APIs, a gRPC client has to be generated by the provider. First, you define a service in a .proto file (aka Protocol Buffer), then you generate client code using the protocol buffer compiler. Where client SDKs are up for debate in the world of RESTful APIs, and client generation might even be frowned upon in some circles, when it comes to gRPC APIs, client generation is a requirement--dictating a much tighter coupling and contract, between API provider and consumer.
I do not have the first-hand experience with this process yet, I am just learning from my discussions last week, and trying to understand how gRPC is different from the way we've been doing APIs using a RESTful approach. So far it seems like you might want to consider gRPC if you are looking for significant levels of performance from your APIs, in situations where you have a tighter relationship with your consumers, such as internal, or partner scenarios. gRPC requires a tighter API contract between provider and consumer, something that might not always be possible, depending on the situation.
While I'm still getting up to speed, it seems to me that the .proto file, or the protocol buffer definition acts as the language for this API contract. Similar to how OpenAPI is quickly becoming a contract for more RESTful APIs, although it is often times much looser contract. I'll keep investing time into learning about gRP, but. I wanted to make sure and process what I've learned leading up to, and while at Google this last week. I'm not convinced yet that gRPC is the future of APIs, but I am getting more convinced that it is another important tool in our API toolbox.
I'm learning a lot about HTTP/2 and gRPC this week, so I have been thinking about how we isolate ourselves by focusing in on individual toolsets, where we should really be expanding our horizons, helping ensure we have the most robust and diverse API toolbox we possibly can. Depending on what part of the tech universe an engineer comes from they'll have a different view on just exactly what I mean when I say API to them.
The most common perspective that people respond with is REST. Folks automatically think this is what I mean when I say API--it isn't. That is your dogma or your gullibility to other people's dogma--please expand your horizons. When I say APIs, I mean an application programming interface that leverages web technology (aka HTTP, HTTP/2). Sure REST is the dominating philosophy in this discussion, but I see no benefit in limiting ourselves in this fashion--I see API as a toolbox, with the following toolsets:
- REST - Considering Roy Fielding's guidance when it comes to leveraging HTTP in my API design.
- Hypermedia - Considering the web, and the role that links can play in your API design.
- Webhooks - Helping keep APIs a two-way street, expanding your API to respond to events not just single requests.
- GraphQL - Helping put powerful query tooling in the hands of API consumers when working with large and complex datasets.
- gRPC - Acknowledging that the HTTP protocol is evolving, and our API contract can evolve along with the specification.
This represents the major toolsets that are currently in my API toolbox. Sure there are a number of other individual tools and specifications in there as well, but this represents the top tier toolsets I am looking at when it comes to getting things done. In my opinion, there is a time and place for each of these toolsets to be put to use. I can easily geek out on any of these areas, but because I'm not selling you a product or service, I'm more interested in finding the right tool for the situation.
I do not expect that all providers have this robust of API toolbox from day one--it is something that will come with time. My recommendation is everyone begins with REST, developing proficiency in this domain, while you are also learning about hypermedia webhooks. As your understanding evolves, then you should start looking at GraphQL, gRPC, and other more advanced approaches to round off your skills. Do not let your religious technological beliefs or any single vendor motivations limit the scope of your API toolbox, you should be working to make it as robust and diverse as you possibly can.
I have been processing Google's API design guide, and an unexpected part of the work has been learning more about gRPC, which Google is "converging designs of socket-based RPC APIs with HTTP-based REST APIs." -- something I have not seen in an API design guide until now. "gRPC uses protocol buffers as the Interface Definition Language (IDL) for describing both the service interface and the structure of the payload messages", and is something I'm hearing more chatter about from larger providers, which I think represents the evolving world of API design beyond the old REST days.
According to the site, "gRPC is used in last mile of computing in mobile and web client since it can generate libraries for iOS and Android and uses standards based HTTP/2 as transport allowing it to easily traverse proxies and firewalls. There is also work underway to develop a JS library for use in browsers. Beyond that, it is ideal as a microservices interconnect, not just because the core protocol is very efficient but also because the framework has pluggable authentication, load balancing etc. Google itself is also transitioning to use it to connect microservices."
To help me learn anything I need an example to reverse engineer, showing me how things work--so here are the examples they provide from the site:
I'm going to add a research area for gRPC, similar to hypermedia, as well as GraphQL. It helps me better keep track of the news about each approach to crafting APIs, and a single place I can go to reference service providers, and working examples. I've had gRPC / Protocol Buffers on my monitoring list for some time now, but seeing Google invest so heavily in this area gives me a signal that I should be paying attention to gRPC more, and gathering more examples that I can share with my readers.
I'm thinking that I will assemble some sort of toolbox area in my API design research, helping folks understand when a more RESTful approach makes sense, as well as when hypermedia, GraphQL, or maybe a gRPC approach might be more appropriate. Next, I'm going to dive into BigTable, PubSub, and Speech APIs to see gRPC in action--something I'm hoping that will help me better understand Google's approach to API design, which something I'm struggling with grasping completely.
This is a repeat story of one I wrote two years ago, but things haven't changed so I'm going to rant about again, 2017 style. We need someone to develop an open source, visual API design editor. There is the Swagger Editor, but that is more a web IDE, and I'm looking for a well designed, intuitive, visual editor for managing your OpenAPI definitions, that is embeddable and easily integrated into any web or mobile system.
There are examples of nice visual API design editors with some of the leading API service providers, but I'm looking for something any API service provider could put to work. I included these examples in my last story, except I'm including Stoplight this round instead of Gelato, because they gave me an upgrade my account message, and wouldn't let me see the interface. These are currently my favorite three API design editors:
- Restlet - A design, deployment, management service provider.
- APIMATIC - An SDK, API definition transformer, and continous integration provider.
- Stoplight - An API design, definition, documentation, and orchestration provider.
I enjoy using all three of their API design interfaces. They are well thought out and provide what you need to design new APIs, as well manage the design of your existing APIs. While they are doing a good job at this, it is something I'd like to see standardized, and open sourced, so that other service providers do not have to reinvent the wheel here. We shouldn't have to use a different API design interface for each service we depend on, we need a common editor, making things more familiar.
Ok, rant over. This is something I'll keep talking about until there is a solution. Oh, I have to say, I am impressed with what the Materia team has done for API design on the desktop. Between them and Stoplight, there has been positive motion forward. I'm just looking to give things another nudge and help folks understand the opportunity. I'll explore possible business models for such a solution in another post, and keep beating this drum until I get what I want.
Everyone developing APIs struggles with API design. Ok, maybe a few of the gurus out there don't, but the rest of us need education, practice, and ideally someone or something to help guide us through the best practices when it comes to API design. I try to track what different companies are doing when it comes to API design, and maintain a list of any API design guides I come across in my work.
If you are looking to start getting a grasp on API design at your company, organization, institution or agency, I recommend starting on the journey to define your own API design guide--you will be surprised what you can learn along the way. You can kick things off by visiting my API design research, but I strongly recommend you head over to the API Stylebook, and build on this important work.
My friend Arnaud has aggregated 12 of the leading API design guides out there and is currently processing Google's API design guide, adding more knowledge to the repository. He takes each API design guide and breaks them down into common API design topics that you can consider as part of your own process, and resulting guide. His work is all machine readable and forkable so that you can easily assemble your own guide, and get to work learning, defining, and refining--then you can submit your guide to the API Stylebook when you are ready.
API design and API definitions are the top two most trafficked areas of my website in 2017, and most of 2016. API design is top of mind for API providers who are further along in their journey, and the ones who are seriously rocking it usually have some sort of guide, helping keep things consistent. If you need any help, feel free to reach out. I am not an API design expert, but I know my way around the finer details, and concepts at play, and I'm happy to help where I can. I know Arnaud is happy to answer questions as well, and if nothing else, let me help tell the story of your API design journey so that others can learn from it.
<p>I’m working on the next evolution in my API research, and I’m investing more time and energy into the design of the guides I produce as a result of each area of my research. I’ve long produced a 20+ page PDF dumps of the leading areas of my research like API design, definitions, deployment, and management, but with the next waveÂ of industryÂ guides, I want to polish my approach a little more.Â </p>
The biggest critique I get from folks about the API industry guides I produce is that they provide too much information, aren't always polished enough, and sometimes contain some obvious errors. I'm getting better at editing, but this only goes so far, and I'm bringing in a second pair of eyes to review things before they go out. Another thing I'm introducing into the process is the use am of professional design software (Adobe InDesign) rather than just relying on PDF's generated from my internal system with a little HTML spit shine.
While it is taking me longer to dust off my Adobe skills than I anticipated, I am finding the design process to be extremely valuable. I've often dismissed the fact that my API research needed to look good, and that it is more important that it is just available publicly on my websites. This is fineÂ and is something that will continue, but I'm finding a more formal design process is helping me think through all of the material, better understand what is valuable, what is important, and hopefully better tell a story about why it is relevant to the API space. It is helping me take my messy backend data and content, and present it in a more coherent and useful way.
As I'm formalizing my approach using my API definition guide, I'm moving on to my API design guide, and I can't help but see the ironyÂ in learning the value of designÂ while publishing a guide on API design, where I highlight the benefits of API design to make sense of our growing digital assets. Once I've applied this new approach to my definition, design, deployment, DNS, and management guides I am going to go back to my API design for these resources, and take what I've learned and applied to the way I share the raw data and content. The lesson that stands out most at the moment, is that less is more, and simple speaks volumes.
I get the argument from hypermedia and linked data practitioners that we need to model our web API behavior on the web. It makes sense, and I agree that we need to be baking hypermedia into our API design practices. What I have trouble with is the fact that the web is a cornerstone that we should be modeling it after. I do not know what web y'all use every day, but the one I use, and harvest regularly is quite often is a pretty broken thing.
It just feels like we overlooking so much to support this one story. I'm not saying that hypermedia principles don't apply because the web is shit, I'm just saying maybe it isn't as convincing of an anchor to build a story that currently web APIs are shit. I understand that you want to sell your case, and trust me...I want you to sell your case, but using this argument just does not pencil out for me.
There is another aspect of this that I find difficult. That the web was developed and took root in a very different environment than web APIs are. We had more time and space to be more thoughtful about the web, and I do not think we have that luxury with web APIs. The stakes are higher, the competition is greater, and the incentives for doing it thoughtfully really do not exist in the startup environment that has taken hold. We can't be condemning API designers and architects for serving their current master (or can we?).
While I will keep using core web concepts and specs to help guide my views on designing, defining, and deploying my web APIs, I'm going to explore other ways to articulate why we should be putting them to use. I'm going to also be considering the success or failure of these elements based on the shortcomings of the web, and web APIs, while I work to better polish the existing stories we tell, as well as hopefully evolve new ones that help folks understand what the best practices for web APIs are.
I recently carved out some time to read A Web API ecosystem through feature-based reuse by Ruben Verborgh (@RubenVerborgh) and Michel Dumontier. It is a lengthy, very academic proposal on how we can address the fact that "the current Web API landscape does not scale well: every API requires its own hardcoded clients in an unusually short-lived, tightly coupled relationship of highly subjective quality."
I highly recommend reading their proposal, as there are a lot of very useful patterns and suggestions in there that you can put to use in your operations. The paper centers around the notion that the web has succeeded because we were able to better consider interface reuse, and were able to identify the most effective patterns using analytics, and pointing out that there really is no equivalent to web analytics for measuring an APIs effectiveness.
In order to evolve Web API design from an art into a discipline with measurable outcomes, we propose an ecosystem of reusable interaction patterns similar to those on the human Web, and a task-driven method of measuring those.
To help address these challenges in the world of web APIs, Verborgh and Dumontier propose that we work to build web interfaces, similar to what we do with the web, employing a bottom-up to composing reusable features such as full-text search, auto-complete, file uploads, etc.--in order to unlock the benefits of bottom-up interfaces, they propose 5 interface design principles:
- Web APIs consist of features that implement a common interface
- Web APIs partition their interface to maximize feature reuse.
- Web API responses advertise the presence of each relevant feature
- Each feature describes its own invocation and functionality.
- The impact a feature on a Web API should be measured across implementations.
They provide us with a pretty well thought out vision involving implementations and frameworks, and the sharing of documentation, while universally applying metrics for being able to identify the successful patterns. It provides us with a compelling, "feature-based method to construct the interface of Web APIs, favoring reuse overreinvention, analogous to component-driven interaction design on the human Web."
I support everything they propose. I cannot provide any critique on the technical merits of their vision. However, I find it lacks an awareness of the current business and political landscape that I find regularly present in the hypermedia, and linked data material I consume.
Here are a few of the business and political considerations that contribute to the situation we find ourselves in that Verborgh and Dumontier are focused on, which will also work to slow the adoption of their proposed vision:
- Venture Capital - The current venture capital driven climate does not incentivize sharing and reuse, and their startups investing time and energy into web technologies.
- Intellectual Property - Modern views of the intellectual property, partially fueled by VC investment, but further exacerbated by legal cases like Oracle v Google force developers and designers to hold patterns close to their chest, limiting sharing and reuse again.
- Lazy Developers - Not all developers are knowledge seekers like the authors of this paper, and myself, many are just looking to get the job done and get home. There are few rewards for contributing back to the community, and once I have mine, I'm done.
- The Web Is Shit - One area that linked data and hypermedia folks tend to lose me is their focus on modeling things after the web. I agree the web is "working", but I don't know which one you use, but the one I use is shit, and only getting worse--have you scraped web content lately?
- Metrics & Analytics - Google Analytics started out providing us with a set of tools to measure what works and doesn't work when it comes to the parts and pieces of our websites, but now it just does that for advertising. Also we do have analytics in the API space, but due to the other areas cited above, there is no sharing of this wisdom across the space.
These are just a handful of areas I regularly see working against the API design, definition, and hypermedia areas of the space, and will flood in slow the progress of their web API ecosystem vision. It doesn't mean I'm not supportive. I see the essence of a number of positive things present in their proposal, like reuse, sharing, and measurement. I feel the essence of existing currents in the world of APIs like microservices, DevOps, and continuous integration (aka orchestration).
My mission, as it has been since 2010, is make sure really smart folks like Ruben and Michel at institutions, startups, and the enterprise better understand the business and political currents that are flowing around them. It can be very easy to miss significant signals around the currents influencing what is working, or not working with APIs when you are heads down working on a product, or razor focused on getting your degree within an institution. The human aspects of this conversation are always well cited, but I'm thinking we aren't always honest about the human elements present on the API side of the equation. Web != API & API !=Web.
APIs can help make technology better serve us humans when you execute them thoughtfully. This is one of the main reasons I kicked off API Evangelist in 2010. I know that many of my technologist friends like to dismiss me in this area, but this is more about their refusal to give up the power they possess than it is ever about APIs.
I have been working professionally with databases since the 1980s, and have seen the many ways in which data and power go together, and how technology is used as smoke and mirrors as opposed to serving human beings. One of the ways people keep data for themselves is to make it seem big, complicated, and only something a specific group of people (white men with beards (wizards)) can possibly make work.
The dominant industrial metaphors for data do not privilege the position of the individual. Instead, they take power away from the person to which the data refers and give it to those who have the tools to analyze and interpret data. Data then becomes obscured, specialized, and distanced.
We need a new framing of a personal, embodied relationship to data. Embodied metaphors have the potential to bring big data back down to a human scale and ground data in lived experience, which in turn, will help to advance the public’s investment, interpretation, and understanding of our relationship to our data.
DATA IS A MIRROR portrays data as something to reflect on and as a technology for seeing ourselves as others see us. But, like mirrors, data can be distorted, and can drive dysmorphic thought.
This is API for me. The desire to invest, interpret, and understand our relationship to our data is API design. This is why I believe in the potential of APIs, even if the reality of it all often leaves me underwhelmed. There is no reason that the databases have to be obscured, specialized, and distant. If we want to craft meaningful interfaces for our data we can. If we want to craft useful interfaces for our data, that anyone can understand and put to work without specialized skills--we can.
The problem in this process is often complicated through our legacy practices, the quest for profits, or there are vendor-driven objectives in the way of properly defining and opening up frictionless access to our data. Our relationships with our data are out of alignment because it is serving business and technological masters, and do not actually benefit the humans whom it should be serving.
One of the reasons I enjoy profiling APIs is that they give an honest view of what a company does, absent of all the marketing fluff, and the promises that I see from each wave of startups. If designed right, APIs can provide a very functional, distilled down representation of data, content, and algorithmic resources of any company. Some APIs can be very fluffy and verbose, but the good ones are simple, concise, and straight to the point.
As I'm profiling the APIs for the companies included in my API monitoring research, what API Science, Apica, API Metrics, BMC Software, DataDog, New Relic, and Runscope offer quickly become pretty clear. A simple list of valuable resources you can put to use when monitoring your APIs. Crafting an OpenAPI Spec allows me to define each of these companies APIs, and easily articulate what it is that they do--minus all the bullshit that often comes with the businesses side of all of this.
I feel like the detail I include for each company in an APIs.json file provides a nice view of the intent behind an API, while the details I put into the OpenAPI Spec provide insight into whether or not a company actually has any value behind this intent. It can be frustrating to wade through the amount of information some providers feel they need to publish as API documentation, but it all becomes worth it once I have the distilled down OpenAPI Spec, giving an honest view of what each company does.
I was playing with a new API design and deployment solution, from some of my favorite developers out there this weekend called Materia, which bills itself as "a modern development environment to build advanced mobile and web applications"--I would add, "with an API heart".
Materia is slick. it is modern. While very simple, it is also very complete--allowing you to define your underlying data model or entities, design and deploy APIs, and then publish a single page applications (SPA) for use on the web, or mobile devices. Even though I'm one of those back to land, hand-crafted API folks, I could see myself using Materia to quickly design and deploy APIs.
I say this in the most positive light imaginable, but Materia reminds me of the Microsoft Access for APIs. Partly its the diagramming interface for the entities, but it is also the fact that it bridges the backend to the frontend, allowing you to not just design and deploy the database and APIs, but also the resulting user interface that will put them to work.
I know they are just getting going with developing Materia, but I can't help but share a couple things I'd like to see, that would make it continue to be the modern API driven application it is striving to be:
- OpenAPI Specs or Blueprints - Allow users to import, export, and manage my APIs in the popular API definition format my choice.
- Schema.org - Provide users with a wealth of existing entity models to choose from, so they do not reinvent the wheel.
- Github - Allow for the publishing of projects, and importing of them to and from Github, allowing for the sharing of server design and deployment patterns.
There are a number of other things I'd like to see, but I'm sensitive to the fact that they are just getting started. These three areas would significantly widen the initial audience for Materia beyond the developer class, which is who the application solution should be targeting. Like I said, it has the potential to be the Microsoft Access of APIs for small businesses, which isn't quite the Microsoft Excel of APIs, but a close second. ;-)
Nice work guys! It is the another positive advancement in the world of API design alongside Restlet launched their API design studio, and Apiary setting this modern era of API design into motion with Apiary. I'll be tuning into Materia's evolution on Twitter, and play more with the server and designer editions available on Github.
My friend Arnaud Lauret (@arno_di_loreto), the API Handyman, has released a very cool new project called the API Stylebook--a collections of resources for API designers. It is a brilliant aggregation of twelve API design guides from Atlassian, Cisco, Cloud Foundry, Haufe, Heroku, Microsoft, PayPal, Red Hat, The White House, Zalando.
I think the API Stylebook purpose describes it well:
The API Stylebook aims to help API Designers to solve API design matters and build their API design guidelines by providing quick and easy access to selected and categorized resources.
In this first version, the API Stylebook provides direct links to specific topics within publicly available API Design Guidelines. Instead of reinventing the wheel or searching google for hours, API Designers quickly can find solutions and inspiration from these existing guidelines.
The API Stylebook isn't just a list of API design guidelines from these companies, it is a machine readable distillation of the twelve API design guides into a master list of API design topics you can consider when crafting your own API design guide. It is slick. I like Arnaud's approach to analyzing the existing API design patterns across API platforms who have shared their guides. I also really like the approach to breaking things down as YAML, and publish as a very good looking website using Github, and Github Pages.
I'm hoping it is something that will stimulate other API providers to craft and share their own API design guide. If you do, I'm sure Arnaud would add to his work, especially if you use his work as a blueprint for your design guide--keeping things as standardized as possible. Projects like this are critical to the growth of the space, and a lack of resources like API Stylebook, and Webconcepts.info have seriously slowed the consistency and interoperability across the world of APIs.
In the last couple of years, I've seen the concept of API design go from being something the API elite discuss, to something that involves business users, and something that has spawned a whole ecosystem of service and tooling providers.
This API design research covers the concepts people often associate with API design, like REST, and Hypermedia, but also touches on other media types, API definitions, and many of the services, tooling and other solutions that have emerged to serve the space.
I fund my research through consulting, selling my guides, and the generous support of my sponsors. I appreciate your support in helping me continue with my work and hope you find it relevant in your API journey.
I am working on one possible API definition for the Human Services Definition Specification (HSDS), and the next phase of this work involves bringing in a small group of technical, and non-technical folks to have discussions around their API designs, in context of the master specification I am looking to pull together.
To help drive the discussion I am wanted to use the OpenAPI Specification that I created for HSDS, and I also knew I wanted to use Github issue management to help keep track of the synchronous, and asynchronous conversation that would occur around it. However Github tends to have a perceived barrier to entry for many non-developers (which I don't fully get), so I wanted to leverage the platform, but also soften the way everyone discussed the overall specification, as well as each individual endpoint.
The HSDS specification is pretty verbose, and I needed a way to have conversations at the high level, but also down to the endpoint level. To help facilitate this, I got to work on a prototype micro tool which enables a conversation around any API(s) that are indexed within an APIs.json file, producing a human readable list of endpoints, parameters, etc., but then uses Github issue management as the core of the conversation.
Resulting in my APIs.json Annotation tool. It is just a first draft, so there will be a lot of changes that need to occur. I'm going to test it against 20+ APIs.json collections I have defined as part of my API Stack work to try and harden it a little bit. My APIs.json Annotation tool runs in my single repo app style, leveraging Jekyll + Github Pages + Github API to operate--Github is the front and backend.
Anyone can view the conversation, but if you want to participate you have to be added to the Github repository, and pass in a Github personal token. This is something I often automate with a simple Github login, where I use OAuth.io to obtain token, but I kind of see the token as a minimum viable investment to understanding Github for using each tool.
It is really important to me that each app stands on its own feet. Not all of my micro tools that I develop in this way will enjoy zero outside dependencies, but most of them can be easily forked, and ran in any Github user account or org (with little or no setup). Conversations around API is just one area I am looking to simulate with this approach to delivering tooling, and specifically APIs.json tooling, that can be used throughout an API life cycle.
You are welcome to play with the APIs.json Annotation, or any of the other tools I have listed on my home page. I will keep adding them here, so that they can be found, but like all my work they are all a work in progress. Each tool has its own dedicated repo and issue management, where you are welcome to join in the conversation around the road map for each one. I am just looking to develop a robust toolbox of small micro tools that will help be more successful across the life cycle of the APIs I am working on, but maybe you can benefit using them too.
Over the last five years many of us have been pushing forward our API design skills to deliver valuable resources to mobile apps. The multi channel opportunity for delivering data, content, and other resources to not just websites, and web applications, but also to iPhone, Android, Windows, and other mobile platform is clear. Alongside delivering web apps to multiple browsers (IE, Chrome, and Firefox), we had to learn to deliver to an increasingly diverse mobile ecosystem (iOS, Android, Windows).
When you have an API core, crafting new endpoints that speak to new channels is not as daunting of a task. I'm thinking through how to deliver API Evangelist knowledge into a Slack channel, Facebook Messenger discussions, and via Alex Voice for use at our APIStrat Conference in Boston this fall. Whether I am creating bots, or delivering new skills to voice enablement, I'm just adding new design constraints to my already existing stack of things I consider when I'm crafting my API stack(s).
I can't help but be in tune with the frenzy around bots, and the interesting things AWS is doing within the Alexa ecosystem. I do not feel like it is a distraction from my core business, as I'm just applying some new design constraints to what I am already doing. I'm not chasing entirely new things, I'm just responding, and participating in what the latest shifts in what technology are.
I don't feel like any of the 50 areas of the API space that I'm keeping an eye on are distractions, as long as I stay true to my API core, and they help me critically think through, and apply new design constraints to the core business value I am already bringing to the table.
It always makes me smile, when I talk to someone about one or many areas of my API research, sharing how I conduct my work, and they are surprised to find how many areas I track on. My home page has always been a doorway to my research, and I try to keep this front door as open as possible, providing easy access to my more mature areas like API management, all the way to my newer areas like how bots are using APIs.
From time to time, I like to publish my API life cycle research to an individual blog post, which I guess puts my home page, the doorway to my research into my readers Twitter stream, and feed reader. Here is a list of my current research for April 2016, from design to deprecation.
I am constantly working to improve my research, organizing more of the organizations who are doing interesting things, the tooling that is being deployed, and relevant news from across the area. I use all this research, to fuel my analysis, and drive my lists of common building blocks, which I include in my guides, blueprints, and other white papers and tutorials that I produce.
I am currently reworking all the PDF guides for each research area, updating the content, as well as the layout to use my newer minimalist guide format. As each one comes off the assembly line, I will add to its respective research area, and publish an icon + link on the home page of API Evangelist--so check back regularly. If there is any specific area you'd like to see get more attention, always let me know, and I'll see what I can do.
As I made my way through nine of the leading SMS providers, profiling the details of their API plans, trying to bring it all together into a single, machine readable definition, Messente's pricing API stood out for me. Alongside their SMS API, and their number lookup API, they have a pricing API. As I take a moment to look at the simplicity of the endpoint, I am also thinking about the potential, and the details of their API docs really stand out for me.
In addition to a pricing API being a valuable utility for any API consumer, I can't help but think the process of constructing a pricing API would force API providers to think another layer deeper in their API plan(s). Similar to how you have to think through the details of any resource you make available via an API, considering how it will be put to use, crafting a pricing API would force you to do the same for one of the most valuable, and overlooked resources you offer, that is core to all of your API operations.
I've talked about the importance of API providers having management level APIs before, and I think they should also follow the lead of API providers like Messente who exist in a pretty mature sector, and have a handle on their API pricing. I've said it before, and will keeping harping on this issue...that all API providers and their service providers should have pricing APIs, as well as APIs for the other essential building blocks of their API operations.
Another thing to note here, that I will be writing about more in the future, is the presence of two-letter country code as a parameter for an API. While not all APIs will have country specific pricing like SMS providers do, the API providers who are looking to truly operate globally, you will need to continue the automation of your operations, making pricing available on demand, in any currency that consumers will need.
API design and definitions are the number one area when it comes to talks submitted for APIStrat 2015 in Austin, and when it comes to traffic across the API Evangelist network in 2015. After diving into the Amazon API Gateway a little more over the weekend, I was reminded of the opportunity out there when it comes to API design tooling.
Amazon did a good job, providing a GUI interface for crafting the methods, resources, and underlying data models for APIs you are deploying using the gateway. However when you compare to some of the GUI API design editors I mentioned in my last post on this subject, from Restlet, APIMATIC, and Gelato, the Amazon interface clearly needs evolve a little more.
AWS is just getting started with their solution, so I'm not knocking what they have done. I just wanted to keep comparing of all of the solutions as they emerge, and highlight the opportunity for some standardization in this layer of API tooling. I see a pretty big opportunity for some player to step up and provide an open source API design editor that provides a seamless experience across API service providers.
This post is nothing new. I am just trumpeting the call for open API design tooling each time I see another new interface introduced for crafting APIs, their paths, resources, parameters, headers, authentication, and underlying data models. At some point, a new player will emerge with the open source API design editor I am looking for, or one of the existing players open sources their current offering, and evolve it in context of the wider API design space, providing an abstracted layer that supports all API definition formats.
With the growth in the number of service providers I see stepping up to server the API space, the need for common, open tooling when it comes to API design is only going to grow. It took almost four years of waiting for the API management space to figure this out, I'm hoping I don't have to wait as long in the API design side of things.
There was a pretty interesting conversation around API design going on in one of my API slack channels over the last couple days, about what is API design, and what is needed to get developers, and even non-developers more aware of the best practices that exist. It is a private group, so I won't broadcast it as part of this post, but I did want to extract a narrative from it, and help me set a bar for other API design discussions I am having.
The Restafarian, hypermedia, and linked data folk have long been frustrated by the adoption of sensible API design practices across the sector, and the lack of basic HTTP literacy among developers, and non-developers is at dangerously high levels. The good news, is some of this is beginning to change, but we still have so much work to do, something that won't be easy, and unfortunately it won't always have the clean outcomes leaders in the space are looking for.
APIs returning JSON is just the next step in the evolution of the web, and when you consider how much work it took to get everyone on-board with HTML, and building web pages, then web apps, and recently mobile apps, you can begin to understand the work that still lies ahead. We have to help the next generation of developers be more HTTP literate (something the previous generations of developers aren't), and possess a baseline knowledge of common API design best practices. This needs to be done in a world where many of these developers really aren't going to care about the specifics of good API design, like us API architects and early pioneers have.
The average API designer in the future will not always be willing to argue about the nuance of how to craft URLs, whether to use a header or parameter, caching, and how to version. They just want to get the outcome they seek, accomplish their project, and get paid for their work. Consider the solar industry as a quick comparison. The first wave of installers were domain experts, while the following waves of installers who will be focused on scaling and maintaining the industry, will only need to be trained on only what is required to get the job done in a stable, profitable way.
Ok. So how do we do this right? I feel like we are already on the good path. We just need you to publish your own API design guide somewhere that we can all learn from, like other leading API providers already present in my API design research. As we build a critical mass of these, we need to also work to aggregate the best practices across them, so that instructors and textbook publishers can incorporate into their curriculum. If you have an API platform, and have ever wished that there were more highly skilled API designers out there, make sure you have your API design practices documented, and shared with the world.
This will get healthy API design practices out of the trenches of startups, SMBs, enterprise, and government agencies, and get it into the educational institutions around the world. Then we can start equipping the next generation of programmers with the knowledge they will need to be successful in delivering the resources need for the next generation of Internet powered apps, networks, and devices.
I want to add on emore thing. API service companies who are looking to provide tooling that API providers can use to do deploy APIs, will have share in the load here. This is core to my criticism of the AWS API Gateway, in that I applaud their use HAL, but please make sure you also provide a healthy dose of hypermedia literacy along the way, don't just hide it behind the curtain. I really do not want to see the another FrontPage for APIs, so if you are building an API editor, let me know so I can provide you with some ideas. (1) (2).
We all have a lot of work to do, in preparing the next generation of developers, and business users when it comes to a baseline of HTTP literacy, as well as a healthy dose of API awareness. We are going to need an army of API designers to help us deliver on the API economy we are all seeing in our heads--so let's get to work. If you do not have a formal API design strategy get to work on one (let me know if you need help). If you have one, please share it, so I can add it to my API design research, for others reference.
I walked around San Francisco with Jakub Nesetril (@jakubnesetril), the CEO of Apiary, Wednesday evening, talking about the API space. Eventually we sat down in Union Square, continuing our conversation, which is something I wanted to further process, and blend with some existing thoughts I'm working through. Much of our conversation centered around the need for an open abstraction layer for API design, which would reduce the focus on Swagger vs. API Blueprint vs. WADL vs. RAML vs. any other API definition, and make it just about defining and designing our APIs.
Jakub is right, I'm sure he'd love everyone to use API Blueprint (which thousands are), but it is more important to him that people just use API definitions, and commit to a healthy API design strategy. This line of thinking is in alignment with other thoughts I'm having around there being a common open source API design editor, which I'd like to use as vehicle to get us closer to my vision of a perfect API design editor.
I see an abstraction layer consisting of the following elements:
- Import - We need to be able to import ANY API definition format we desire.
- Export - We need to be able to export ANY API definition format we desire.
- Viewer - We a code view for ANY API definition format we want to look at while working.
- Editor - We a visual, GUI editor that is all about visual API design, WYSIWG editor for APIs
API definition format owners should be able to maintain importing, exporting, and viewing layers via some sort of plug-in architecture--meaning the platform is about API definition, the WSDL, WADL, Swagger, API Blueprint, RADL, RAML, Postman Collection, and other formats should be owned by each respective owner. The Open API Abstraction project could provide a single architecture for everyone to plug into, with an emphasis on building the best possible API design editing experience possible.
The Open API Abstraction tooling could be baked into any API service or tooling in the space, allowing the next generation API service providers to focus on what they do best, and not having to build out their own API design editor, while also baking in compatibility for all API definition formats by default. Such a layer would allow API architects and designers to craft APIs in a consistent way, no matter what API definition format designers and developers might be speaking, opening up a wider world for communicating and collaborating through the API design process.
I'm processing all of my thoughts from some travel to the big city of San Francisco. I was providing feedback on Microsoft's API design guide, as part of the OneAPI Technical Advisory Group. As I was thinking about the journey Microsoft is on, the role of the API design guide, and how many other companies like Paypal, and Cisco, are on the same journey.
In parallel to this, I am on my own journey with my own API stack, I've been looking at everything from a slightly different perspective than many other analysts and providers in the space. When I started in 2010, it was all about API management, then after folks kept asking me about options for deploying APIs I expanded my monitoring to API deployment. Then Jakub the CEO of Apiary, moved the dial back ruth on the life-cycle, and got me paying attention to the concept of API design.
Fast-forward to 2015, I am paying attention to almost 20 separate areas, as part of my core API research. I tune into a number of other areas, but these research projects make up the heart of my API storytelling. On the trip back from San Francisco today I had a few thoughts, and needed to organize them in context of my core research.
- A1 - Tech
- Do you use Swagger to define APIs?
- Do you use API Blueprint to define APIs?
- Are there other API definition formats you use?
- Do you employ NO API definitions?
- A2 - Business
- Do you support multiple API definition formats?
- Do you use API definition formats to engage with other services or tools?
- Are your service tiers represented in your aPI definitions?
- A3 - Politics
- Do you share your API definitions publicly?
- How do you license your API definitions? Are they open?
- B1 - Tech
- Do you use Restlet Studio's API design tools?
- Do you use APIMATIC API design tools?
- Do you use Apigee API Studio API design tools?
- Do you use Gelato API design tools?
- B2 - Business
- Do you use a service provider for API design?
- Do you use a consultant(s) for API design?
- Do partners and other stakeholders participate in the API design process?
- B3 - Politics
- Do you JSON:API?
- Do you hypermedia?
- Do you JSON-LD?
- C1 - Tech
- What gateway tech do you use?
- What API frameworks do you use?
- What cloud solutions do you use?
- C2 - Business
- Are you monetizing APIs?
- Are you covering costs of deployment?
- Do you hand API deployment internally?.
- Do you depend on outside resource to deploy APIs?
- C3 - Politics
- Are your API deployment solutions open source?
- Do Your API Deployment resources scale?
- D1 - Tech
- Are you rolling your own API management solution?
- Which API management solution are you using?
- D2 - Business
- Do you require registration to use an API?
- Do you have service tiers?
- Do you have partner levels of access?
- Do all areas of your companies operation apply same approach?
- D3 - Politics
- Is your API management solution open source?
- Do you provide support for your API developers?
- E1 - Tech
- Do you require SSL?
- Do you require API keys?
- Are you employing OAuth?
- E2 - Business
- Are you investing in security?
- What would a security breach cost you?
- Do all areas of your companies operation apply consistent template?
- E3 - Politics
- Are you more secure than your competition?
- Do you share your security practices publicly?
- F1 - Tech
- Which management solution do you use to handle monetization?
- Is there an API for your API monetization layer?
- Is there a developer dashboard for pricing, billing, and revenue sharing?
- F2 - Business
- Do you apply consistent monetization strategy across all APIs?
- Is your billing real-time?
- Is API revenue, your only revenue stream?
- Do you have a credit based system, beyond just API call based?
- F3 - Politics
- Do you share your pricing publicly?
- Is your partner program transparent?
- Do developers have dashboard for managing billing?
- G1 - Tech
- What services to you use for monitoring APIs?
- What open source tooling do you use for your monitoring?
- Does your monitoring include testing, performance, and security?
- G2 - Business
- What do you spend on API monitoring each month?
- Is there a dedicated person(s) to monitoring APIs?
- What has outages cost you in the past?
- G3 - Politics
- Do you publish your monitoring reports publicly?
- Do you keep your ecosystem in tune with monitoring via messaging system(s)?
- H1 - Tech
- What services to you use for testing your APIs?
- What open source tooling do you use for your testing?
- What are your benchmarks?
- H2 - Business
- Do you provide testing tools to your developers?
- Can developers request specific types of testing for APIs?
- H3 - Politics
- Do you publish your monitoring reports publicly?
- I1 - Tech
- What services to you use for testing your APIs?
- What open source tooling do you use for your testing?
- What are your benchmarks?
- I2 - Business
- Do you have SLAs for any tiers of operation?
- Do you generate any revenue from SLAs in place?
- I3 - Politics
- Do you publish your monitoring reports publicly?
- Do you consistently meet your SLAs?
- J1 - Tech
- What services to you use for virtualizing your APIs?
- What open source tooling do you use for your API virtualization?
- J2 - Business
- Is virtualization part of your QA process?
- Do you virtualized instances of your API as an added service?
- Do you provide a sandbox or simulators for developers by default?
- J3 - Politics
- Do you provide virtualization for developers?
- Are your virtualization images openly sourced?
- K1 - Tech
- What services to you use for virtualizing your APIs?
- What open source tooling do you use for your API virtualization?
- K2 - Business
- Do you have dedicated people to managing your API architecture?
- Do you have dedicated services for managing your API architecture?
- K3 - Politics
- Can you migrate between infrastructure providers? (ie. AWS to Google)
- Is your server side API code open source?
- Are you virtualization images openly licensed and available? (ie. Docker Images)
- L1 - Tech
- Do you use oEmbed?
- Do you have bookmarklets?
- L2 - Business
- Do you have a standardize strategy for allowing users to embed API driven resources?
- Is your embeddable strategy integrated with your overall marketing and branding efforts?
- Do you offer embeddable tool builder?
- L3 - Politics
- Are your embeddable tools non-invasive? Protect privacy?
- M1 - Tech
- Do you have robot evangelists? Just sounded cool, and couldn't think of anything to put here.
- M2 - Business
- Do you have dedicated evangelist resources?
- Do you contract with 3rd parties for any evangelist resources?
- Is your evangelism coupled with your marketing?
- M3 - Politics
- Are there opportunities for developers to get involved with evangelism?
- Do you have healthy feedback loop present with your API operations?
- N1 - Tech
- Do you have an APIs.json for your API operations?
- Do you employ API definitions?
- Are you using hypermedia?
- N2 - Business
- Are your APIs plug and play with other platforms?
- Are your APIs public or private?
- N3 - Politics
- Does discovery feed every other layer of API life-cycle?
- Does discover play into your security strategy?
Sorry, I don't mean to be a downer. But...we are just getting started with stabilizing how we do APIs. I feel like we are beginning to formalize how we manage or APIs (thanks Mashery, 3Scale, and APIgee), and we are getting a handle on API deploy (thanks Restlet, Amazon), and we are deep into understanding how we define (thanks Swagger and API Blueprint), and ultimately design (thanks Apiary, Restlet, and Apigee).
We are moving fast into testing, monitoring, performance, and virtualization (thanks Runscope, APITools, API Science, and SmartBear), but our security sucks, embeddability has stagnated, and evangelism and discovery really isn't improving. I'm working on APIs.json, and bringing together the Swagger and API Blueprint communities, but there is so much work left when it comes to discovery, and automating each of the areas listed above.
There really is no point to this post. It is my mental vomit, after a trip to San Francisco, and working on Microsoft's API design guide. Stay tuned for how any of this applies to anything.
If you think there is a link I should have listed here feel free to tweet it at me, or submit as a Github issue. Even though I do this full time, I'm still a one person show, and I miss quite a bit, and depend on my network to help me know what is going on.