Moving to Micro-services — Service Discovery

For the past few weeks, part of the development team at Codacy has been working on breaking our application into small microservices, following the ideas learned from this book.

We seem to have a perfect fit; a big monolithic application, with several sub-components that have a clear bounded context; having been built on top of Akka, we can clearly identify several parts of the system that are natural candidates for this.

Historically, on Codacy, we deploy several analysis servers that are composed of several subsystems, containing all the required functionality to properly analyse a project. When we need more computing power, we just deploy more instances of the application.

Although this enables us to quickly grow and scale our application, it is not very performant; sometimes, we just need to scale one individual component in our application. For example, when analysing big PHP projects, we only require more computing power on the PHP engine, and not on any other system.

Using microservices, we aim to be able to quickly grow the individual components that are under heavy load, not only scaling the application by launching new servers but, on each server, only launching the components required to perform quicker analysis.

Service Discovery

Strongly influenced by Jeff Lindsay posts on this issue (one and two), we decided to tackle the service discovery problem by using consul.

To simplify this, Johann (johann at codacy dot com), built a Consul Scala API. This, and with the help of some scala magic, allowed us to make some very simple abstractions.

When we launch a new service (contained in a docker), it automatically registers itself as a service of type X on our consul server.

When we require a type of service in our code, we simply call:

ServiceFinder.withService[IRepositoryService] {
  service =>
    service.someMethod()
}

The service finder encapsulates the discovery of the service and provides with a strongly typed local class that will take care of calling the remote service, parsing all the objects to and from JSON and not allowing the developer to keep the reference to the service; this is especially important as we have no guarantee that a service that is working right now will be healthy in a couple of minutes, so we should always go through the process of discovering a healthy instance when we require any information from it. This is a sample complete implementation of such a class:

abstract class IRepositoryService(address: InetSocketAddress) extends IBaseService(address) {
  val key = DiscoverableServices.repositoryService

def getBranches(branchesRequest: BranchesRequest): Response[ProjectBranches]

def getCommits(commitsRequest: CommitsRequest): Response[List[ExternalCommit]]

def getRepositoryData(request: RepositoryDataRequest): Response[RepositoryData]

def getBlame(request: BlameRequest): Response[Seq[Blame]]

def getDiff(request: DiffRequest): Response[CommitDiff]
}

private[discovery] class RepositoryService(address: InetSocketAddress) extends IRepositoryService(address) {

def getBranches(branchesRequest: BranchesRequest): Response[ProjectBranches] = 
    postData[ProjectBranches]("branches", JsonResponse.toJson(branchesRequest))

def getCommits(commitsRequest: CommitsRequest): Response[List[ExternalCommit]] =
    postData[List[ExternalCommit]]("commits", JsonResponse.toJson(commitsRequest))

def getRepositoryData(request: RepositoryDataRequest): Response[RepositoryData] =
    postData[RepositoryData]("projectData", JsonResponse.toJson(request))

def getBlame(request: BlameRequest): Response[Seq[Blame]] =
    postData[Seq[Blame]]("blame", JsonResponse.toJson(request))

def getDiff(request: DiffRequest): Response[CommitDiff] =
    postData[CommitDiff]("diff", JsonResponse.toJson(request))
}

While we are migrating the system to this new infrastructure, we also built in a fail safe system for the service discovery. If, when trying to find a service, we find that there is no server running the required micro-service, the service finder automatically creates a local JVM class instance of the same code that would be performing the work on the remote server. The choice is done while finding the service:

def withService[A <: IBaseService, T](block: A => Response[T])(implicit factory: ServiceFactory[A], manifest: Manifest[A]): Response[T] = {
  findService[A](factory, manifest).map {
    service =>
      block(service)
  }.getOrElse(Response.error(ResponseErrorCode.MicroServiceError, s"Couldn't find a service for ${factory.key}"))
}

private def findService[A <: IBaseService](implicit factory: ServiceFactory[A], manifest: Manifest[A]): Option[A] = {
  if (SystemConfiguration.current.serviceDiscoveryEnabled) {
    discovery.findService(factory, manifest).collect {
      case service if service.connected => service
    }.orElse {
      logger.warn(s"Could not find service for ${factory.key}, creating local fallback")
      Some(ComponentFactory.instance.getComponent[A])
    }
  } else {
    Some(ComponentFactory.instance.getComponent[A])
  }
}

We expect to mature and evolve our service monitorization to a point where we are confident we won’t need to create this but, while growing the system, we will keep it as safe as possible.

Next time, I’ll write about how we have worked the service monitorization, health reporting, and service auto-scaling, in order to make it transparent for the micro-service developer. In the end, we expect to open-source this to help others grow their own applications.

Until next time.

Edit: We just published an ebook: “The Ultimate Guide to Code Review” based on a survey of 680+ developers. Enjoy!

About Codacy

Codacy is used by thousands of developers to analyze billions of lines of code every day!

Getting started is easy – and free! Just use your  GitHub, Bitbucket or Google account to sign up.

GET STARTED