Graphql коды ошибок

The GraphQL spec defines a set of validations for the GraphQL server to perform when resolving the query. If some validation fails, the corresponding error message must be returned under the errors entry of the GraphQL response. The actual message to return is not defined in the spec, hence it is customized by each server. More importantly, the spec does not define a code to uniquely identify each of the errors.

In other words, there is no concept of an “error code” in the GraphQL spec — only validations to perform. Indeed, there’s no code entry for the error in the response, only message (and also locations, to indicate where the error is produced in the query). In order to return an error code, the GraphQL server must add it under the extensions sub-entry of the error, which GraphQL servers can use for whatever custom feature they support, and use an error code of its own conception.

For the actual error code to return, servers may currently use one of these alternatives:

1. The section of the GraphQL spec under which the corresponding validation is defined
2. Use a custom error code — i.e., one not defined at the spec level

For instance, whenever a mandatory argument is missing in a given field in the GraphQL query, the GraphQL server may return error code 5.4.2.1 in the first case, or "gql0001" (or something else) in the second case.

Some other languages/frameworks do make use of error codes, such as Rust, React, and TypeScript. Inspired by these other language and framework experiences, a couple of issues requesting to add generic error codes to the GraphQL spec were proposed two years ago:

1. #698 – Discussion: Generic error codes in the spec
2. #708 – [RFC] Standard Error Codes

Both issues had received immediate endorsement from the community, but no PR has come out out of them, and neither of them has seen any activity in the last year.

As the maintainer of my own GraphQL server, I often browse the list of proposals for the GraphQL spec, attempting to identify my next task to implement. I explored the issues concerning generic error codes and found them compelling, so I decided to support them ahead of a PR and not wait for them to become merged to the spec (which also may never happen).

In this article, I’ll share a few insights I uncovered from implementing this feature on my GraphQL server, including:

• The potential benefits of generic error codes in GraphQL
• Designing and implementing the solution
• Insights from the implementation

In the conclusion, I’ll give my opinion if supporting generic error codes in the GraphQL spec may be worth pursuing or not.

The potential benefits of generic error codes in GraphQL

Currently, all GraphQL servers must implement their own test suites to make sure that the validations are supported and working as expected. As every server will return a custom error message for the failing validation — for instance, some server may return Mandatory argument '{0}' in field '{1}' is missing, while another one may return You must provide a value for argument '{0}' — their tests cannot be reused across servers:

{
  "errors": [
    {
      "message": "Mandatory argument 'id' in field 'user' is missing",
      "locations": [ { "line": 1, "column": 12 } ]
    }
  ]
}

But returning a generic error code alongside the error message could solve this. The test suite could be executed against the error code, which would be known to all servers because it would be defined at the spec level.

In this scenario, whenever there is an error, the server will return a code entry, in addition to the message and locations entries:

{
  "errors": [
    {
      "message": "Mandatory argument 'id' in field 'user' is missing",
      "locations": [ { "line": 1, "column": 12 } ],
      "code": "1"
    }
  ]
}

This is indeed the goal stated by the first of the issues, which advances the idea of supporting a generic test suite that could work for all GraphQL servers:

I am researching and planning to build a generic test suite to check the compliance of the various GraphQL libraries […]. There are various implementations of GraphQL and all of them are throwing errors in various ways.

As all errors are thrown in plain text(string). There is no way to test these implementations for such errors.

Say we have given an incorrect query (with circular fragment) to a GraphQL server and we are expecting an error. If the error comes with a specific error code then it would be great to test the libraries and provides and assuring the actual cause of the error, therefore, improving the DX as well.

Using generic error codes could also simplify the logic in our applications whenever it needs to make sense of the error, and make it easier to swap the GraphQL server provider. For instance, if the application needs to show different layouts depending on the severity of the error, this logic could react based on the error code, and if the code is generic, the same logic will work after migrating to a different GraphQL server.

By supporting generic error codes, we can possibly centralize the definitions for all error messages and codes in the application, even defining them all in a single file, which can help developers understand all possible contingencies that must be taken care of. This is the case with React, which defines all its error codes in a single JSON file.

Finally, the same solution also provides the opportunity to return even more information. In particular, a specifiedBy entry could point to the online documentation that explains why the error happens and how to solve it:

{
  "errors": [
    {
      "message": "Mandatory argument 'id' in field 'user' is missing",
      "locations": [ { "line": 1, "column": 12 } ],
      "code": "1",
      "specifiedBy": "https://spec.graphql.org/October2021/#sec-Required-Arguments"
    }
  ]
}

This is the case with Rust, whose online documentation provides a detailed guide for developers to troubleshoot and understand the language.

Designing and implementing the solution

I have recently implemented the above feature in my GraphQL server so that it also returns code and specifiedBy entries for each error.

As I mentioned before, these are still custom features and are not supported by the GraphQL spec and they must appear under the extensions sub-entry. Only once the proposal is approved and merged into the spec — if it ever is — can those new entries appear at the root of the error entry.

Currently, the error codes are not generic, since that depends on the GraphQL spec defining them. Instead, I have decided to use the validation section number for my specific error code.

Now, when executing the following faulty query (because variable $limit is not defined in the operation):

{
  posts(pagination: { limit: $limit }) {
    id
    title
  }
}

…the server attaches the code and specifiedBy elements to the error, in addition to the expected message and locations entries:

{
  "errors": [
    {
      "message": "...",
      "locations": [...],
      "extensions": {
        "code": "gql-5.8.3",
        "specifiedBy": "https://spec.graphal.org/draft/#sec-All-Variable-Uses-Defined"
      }
    }
  ]
}

Below, I detail some design and implementation decisions I made and why I made them, as well as how they affect the usefulness of generic error codes in GraphQL.

Dealing with different types of errors

When we are creating a GraphQL API, there are different kinds of errors that the API may return:

1. GraphQL spec errors: those relevant to parsing, validating and executing the GraphQL document, covered by the spec under the Language, Validation and Execution sections, respectively
2. Client errors: those under the domain of the application, such as validating that inputs have a certain length or format
3. Server errors: those produced when a runtime operation fails, such as resolving a field that fetches data from an external data source and the connection doesn’t go through

Client and server errors are not simply “all the other” errors: they are quite different in nature. Client errors are known in advance, based on the rules defined for the application, and they are public, so the corresponding error message must be added to the response to inform the user of the problem.

Server errors, by contrast, mostly arise from unexpected events, such as a server or database going down. They are private because they could expose security-sensitive data (such as an API key or database connection credentials), so the user must receive a vague There was an unexpected error message, while the detailed error message must only be provided to the admin of the application.

When we are discussing generic error codes for GraphQL, we are originally dealing with the first type of errors only, GraphQL spec errors. But this doesn’t need to be the case — the GraphQL server has the opportunity to generalize the same concept, and its associated logic, for all three types of errors.

This generalization can produce several additional benefits:

• The developer will only have to learn a single way to define errors for the API
• The user will always receive the error in the same format
• Error codes can be managed in a modular way, as I’ll explain below

The modular approach to managing error codes

Defining all error codes for the complete API in a single file (as I mentioned earlier) would no longer be possible in a modular approach because the different kinds of errors will necessarily be implemented at different layers:

• The GraphQL spec and server errors will be implemented at the GraphQL spec layer
• Server and client errors will be implemented at the application layer

At most, only the GraphQL spec errors can be defined in a single file (as done here), and server and client errors could be defined all together for the API, or possibly in different files using a modular approach (as I’ve demonstrated here and here), and in every module installed for the API, which can return errors.

If we needed to have an overview of all errors in the API, the GraphQL server could support retrieving them via introspection, instead of finding and storing them all in a single place.

Selecting the error codes and appropriate online docs

Because of the different types of errors, the GraphQL server could use different strategies to select what error code to return, and what online documentation to point to:

1. For GraphQL spec errors:
   1. Use generic error codes defined in the spec (once provided)
   2. specifiedBy could point to the corresponding section in spec.graphql.org/draft
2. For server errors:
   1. Use custom error codes defined by the GraphQL server
   2. specifiedBy could point to documentation in the GraphQL server vendor’s website
3. For client errors:
   1. Use custom error codes defined by the API developer
   2. specifiedBy could point to the public API’s technical docs (or, possibly, a section alongside the public schema docs) if it is needed, such as if the error message is not descriptive enough, as an input validation should already be

Format of the error code

As is perhaps obvious at this point, different types of errors will return different error codes, and these can use a different format.

For GraphQL spec errors, the format could directly use the spec section code, such as "gql-{section-code}" (producing code "gql-5.4.2.1"), which allows us to quickly understand which error in the GraphQL spec is being referenced. The spec section code is the natural choice in order to establish the same error code across different GraphQL servers, until the corresponding generic error codes are defined in the spec.

However, this solution is not ideal because one section could require more than one validation to perform; hence, it will not map one-to-one with an error code/message. For instance, section 6.1 Executing Requests may return one of four different error messages, for which an additional suffix a-d was also added to their codes on my server:

• gql-6.1.a: Operation with name '%s' does not exist
• gql-6.1.b: When the document contains more than one operation, the operation name to execute must be provided
• gql-6.1.c: The query has not been provided
• gql-6.1.d: No operations defined in the document

For client and server errors, which are custom errors), we could use the format "{namespace}\{feedbackType}{counter}", where:

• The {namespace} ("PoP\ComponentModel") is unique per module installed on the API (more on this later on)
• {feedbackType} ("e" for “error”) represents the type of “feedback”, where an “error” is just one possible option among several (more on this later on)
• {counter} ("24") is just an increasing number to create a unique code per message within each module

Following this format would produce code "PoP\ComponentModel\e24" in my GraphQL server.

Namespacing error codes

An error code is an arbitrary string that has a single requirement: it must be unique, identifying a single error message. Otherwise, if the same error code were assigned to two different error messages, then it wouldn’t be very useful.

As stated earlier on, different modules in the application could return their own errors. Since different errors happen at different layers of the server and application, and different modules may be provided by different providers (such as using some module offered by one third party, another one created by the API development team, and so on), creating a system that assigns a unique code per error quickly becomes difficult.

For this reason, it makes sense simply to “namespace” error codes when printing them in the GraphQL response. This way, two different modules can both internally use error code "1", and these will be treated as "module1\1" and "module2\1" by the application. As long as each namespace is unique per module, then all produced error codes will also be unique.

Providing other types of feedback messages

If we pay attention to the error messages in TypeScript, we will notice that, in addition to code, they also have a category. Errors will have a category with value "Error", but there are also entries with other values, namely "Message" and "Suggestion":

{
  "'use strict' directive used here.": {
    "category": "Error",
    "code": 1349
  },
  "Print the final configuration instead of building.": {
    "category": "Message",
    "code": 1350
  },
  "File is a CommonJS module; it may be converted to an ES module.": {
    "category": "Suggestion",
    "code": 80001
  }
}

In TypeScript, “errors” are simply one type of feedback that the application can provide to the users. There is no reason why GraphQL could not implement the same idea.

Implementing error codes provides an opportunity for GraphQL servers to also support this capability. Since this behavior is not documented in the GraphQL spec, these additional feedback entries will need to be returned under the root extensions entry in the response.

It is to distinguish among these different feedback types that earlier on I suggested we add {feedbackType} to the entry code, with value "e" for “error”, "w" for “warning”, and "s" for “suggestion”, to start.

The GraphQL spec currently covers two types of feedback: “errors” and “deprecations”. These two can internally be treated as “feedback” by the GraphQL server, so that the same logic can handle both of them (or any other type of feedback message). Then, the specifiedBy entry could conveniently also be assigned to deprecations — to further explain why some field in the schema was deprecated, describe related deprecations expected for the API in the future, and others.

For my GraphQL server, I have decided to support these categories, in addition to errors and deprecations:

• Warning
• Notice
• Suggestion
• Log

Messages from these categories provide extra nuance, since not everything is an error. For instance, a query using @export to assign two different values to the same variable does not necessarily need to be a halting issue (returning null in the corresponding field), and I’d rather return a warning instead:

Supporting different types of feedback messages does not complicate the task for the API developers, since defining errors, warnings, or suggestions is based on the same underlying code.

Insights from the implementation

What benefits and disadvantages would be produced by imposing GraphQL servers to return error codes in addition to messages, and making those codes generic by defining them in the GraphQL spec?

The following are my impressions, based on my experience adding support for this feature on my server.

Still no generic error codes, but that’s easy to correct

As there are no generic error codes yet, I’ve used the corresponding section on the spec as the error code. But this is easy to correct: should the issue be approved and merged into the spec, I must only update the codes from PHP constants defined in a single file, and in all references to them spread across the server code, which I can easily find using an editor (in my case, VSCode).

Then, GraphQL servers can decide to implement this feature, and upgrade it in the future at a negligible cost.

No breaking changes concerning the GraphQL spec

Returning error codes is not a breaking change, because the new feature adds a new entry code in the response, without modifying anything else.

For the time being, this entry must be placed under extensions. If it ever becomes merged into the spec, then code can be moved one level up, which is a minimal change.

Breaking changes concerning the server code

Currently, as the GraphQL response must only include the error message, GraphQL servers need only pass a single string to denote the error, as in this code by graphql-js:

throw new GraphQLError(
  'Schema does not define the required query root type.',
  operation,
);

If error codes were to be returned, the GraphQL spec errors should be mandatory (or they’d provide no value), while the server/client errors could be made optional, allowing developers to decide if to support error codes for their own APIs or not (since these error codes are custom to the application, they don’t serve the purpose of standardizing a response across different servers, making them less useful).

As such, all instances where an error is produced in the GraphQL server’s codebase must be adapted to also provide the error code. The APIs based on them could be adapted too, but they wouldn’t necessarily have to.

In addition, it would make sense to transfer all error messages within each module to a single file, as to ensure that a unique error code is assigned to each error message. However, this task could require a considerable refactoring of the GraphQL server code, and possibly of the implementing APIs too (i.e., if the underlying architecture changes, APIs may be forced to adapt their code too).

In summary, it could make sense for GraphQL servers to support error codes, but possibly only if they don’t need to change their current architecture drastically, as to not require implementing APIs to be adapted too (or risk losing users).

On a different note, all tests already implemented by the GraphQL server could be adapted to be based on the error codes, but that’s most likely an optional task: if the error message-based tests are working nowadays, testing against them should still work after introducing the additional error code on the response.

Unit tests can have a better foundation

When testing the GraphQL server, we may want to validate against the error code, which succinctly conveys what the problem is instead of the error message, which also gives unneeded information, and which may occasionally be updated (whereas the code will remain the same).

In addition, if placing all errors in a single place per module, we can also obtain the error message just by invoking some function (which must receive the error code, and possibly some params to customize the message). This way, if the error message gets updated, the unit tests will still work without the need to update the error message also in them.

For instance, validating 5.8.3 All Variable Uses Defined for my server is done like this:

public function testVariableMissing(): void
{
  $this->expectException(InvalidRequestException::class);
  $this->expectExceptionMessage(
    (
      new FeedbackItemResolution(
        GraphQLSpecErrorFeedbackItemProvider::class, GraphQLSpecErrorFeedbackItemProvider::E_5_8_3,
        ['missingVar']
      )
    )->getMessage()
  );
  $this->getParser()->parse('
    query SomeOperation {
      foo(bar: $missingVar) {
        id
      }
    }
  ')->validate();
}

Track unimplemented validations

Using a central place to manage all errors makes it easier to track those validations which have not been implemented yet.

For instance, my server has all GraphQL spec validations defined in a single file, but those which are not satisfied yet have been disabled and given the description "TODO: satisfy", making it easy for me to identify them.

Better overall picture of all errors, but more difficult to visualize the actual validation code

Moving all error codes to a central location makes it easier to have a complete understanding of the application, which is a good thing, but at the same time it makes it more difficult to understand the actual pieces of code executing the validation, at least if the error code is not self-explanatory.

For instance, for the client and server errors, my server uses a simple counter for error codes, i.e. "1", "2", and so on, instead of more meaningful codes, such as "field_arg_missing", "field_arg_cant_be_negative" and so on. That’s because I was lazy, and because naming error codes is hard.

To some extent, the same situation happens with GraphQL spec errors too. Error codes 5.2.1.1, 5.8.3, and so on, are not as descriptive as error codes "recursive_fragments", "missing_variable", and so on.

As a consequence, when I visualize the code that executes the validation and throws the error if the validation fails, it is not so easy to understand what the error is. The code looks like this:

public function getValue(): mixed
{
  if ($this->variable === null) {
    throw new InvalidRequestException(
      new FeedbackItemResolution(
        GraphQLSpecErrorFeedbackItemProvider::class,
        GraphQLSpecErrorFeedbackItemProvider::E_5_8_3,
        [
          $this->name,
        ]
      )
    );
  }

  return $this->variable->getValue();
}

Before migrating the above piece of code to use error codes, the logic was more understandable:

public function getValue(): mixed
{
  if ($this->variable === null) {
    throw new InvalidRequestException(
      \sprintf(
        'Variable \'%s\' has not been defined in the operation',
        $this->name
      )
    );
  }

  return $this->variable->getValue();
}

Generic test suites? Not so sure about them

The original issue proposed generic error codes as a means to support generic test suites, which would work across different GraphQL servers.

I’m not so sure this would really work out. At least, it would not for my server.

The proposed test suite would need to be platform/technology/language agnostic, as to work for all different implementations of GraphQL servers. As such, it would be implemented as an acceptance test, executing requests against the single endpoint of the GraphQL server’s running instance.

My GraphQL server is an implementation for WordPress. If I were to execute an acceptance test suite, I’d need to fire up a complete WordPress environment, including an instance of a MySQL DB. This, in turn, would add a lot of complexity to my GitHub Actions-based CI.

Instead, I rely on unit tests which completely mock the WordPress functionality, so not only I don’t need an actual MySQL instance, but also WordPress is not needed in order to run the tests, making them much simpler, faster and cheaper.

If the acceptance tests were available, I’d actually struggle to take advantage of them. Since the same validations are already being tested via the unit tests I’ve already created, then I wouldn’t really bother in using the acceptance tests.

A previous attempt to bring generic test suites to GraphQL, called Cats, similarly hit several issues that made the proposal impractical.

While the idea of a generic test suite is compelling, the effort required to pull it off does not appear worth it to me.

Conclusion

Would adding generic error codes to the GraphQL spec be worth it? My impression is yes, but not because of the availability of generic test suites (which is the reason stated in the original issue).

The benefits I’ve gained from supporting error codes are:

• It allows the application to work with codes (which do not change), a unit that is more reliable than messages, which may be updated
• It encourages the use of central locations to manage all error codes, which can give developers a better understanding of all validations to perform
• It allows me to track which validations have not been implemented yet
• It allows providing the extra specifiedBy information, pointing to online documentation for the error, at minimal cost
• It allows providing other types of messages to the user, such as “warnings”, “suggestions” and “notices”, using the same logic

One drawback I’ve experienced is that looking at the logic performing the validations got more difficult to understand than before, because a meaningless error code does not explain what the error is about.

The other drawback I’ve suffered is the time and effort that was required for the transformation of my server’s codebase. The overall refactoring took around one month of work, while I felt like I was running in circles, spending energy just to arrive where I already was, constantly feeling I should instead invest my time in adding new features to the server.

But as a result of the migration, I now have the satisfaction of dealing with a sturdier codebase: unit tests have improved, new types of feedback messages were introduced (and other ones can be added at any moment with minimal effort), and documentation is returned to the user accessing the API. Overall, I feel that the quality of the API has gone up.

As the benefits outweigh the drawbacks, I’m convinced that adding generic error codes to the GraphQL spec is worth pursuing.