← Data Services

Upgrading Java-Virtual-Machine versions of Scala services

We want to keep our services working on the newer Java Virtual Machines.

The process of upgrading our services has these steps:

  1. Upgrade the build code in the service repository, to the newer JVM.
  2. Merge and deploy to production.
  3. Upgrade the configuration of the service (in environments repository) to new memory options.
  4. Merge, watch the operation, and remember to deploy to production…

Upgrade the JVM version in the services.

1. Upgrade the version of SBT (file project/build.properties) to latest one. At time of writing this is

sbt.version=1.3.13

2. In file project/plugins.sbt, add the Artifactory resolvers and upgrade the Banno SBT plugin to the latest:

resolvers := Seq(
  "Banno Artifactory Lib Releases" at "https://artifactory.banno-tools.com/artifactory/libs-release"
)

addSbtPlugin("com.banno"                 % "banno-sbt-plugin" % "11.1.0")

The banno-sbt-plugin provides the contains the docker plugin.

Configure Base Docker Image for deploying your service

Banno-SBT-Plugin provides the BannoDockerPlugin, for building and bundling the service into a docker image, which is then run in a container in each environment. By default, an image for Java 8 Virtual Machine is used.

In the SBT file for your project, you should add the following setting:

bannoDockerBaseImage := "docker.artifactory.banno-tools.com/java:16"

This is an SBT setting, that should be under a .settings( call, or part of a local variable of settings.

DETAIL You may have already a setting for the bannoDockerBaseImage, but configured to registry.banno-internal.com. That repository is deprecated and you should use the artifactory one instead.

STUMBLE If your project has many SBT modules, this setting is only available for those that use the BannoDockerPlugin, which will have the line .enablePlugins(BannoDockerPlugin) If you have a models subproject, or a module that is a client to another service, but which is not an executable on its own, then you should not apply to it the p

Configure the JVM version for your build

So, the docker image setting controls on which JVM, and on which container, the service will run. You should also compile it and build it running SBT with the same JVM. To do this, you have to add a file named .java-version, which has a single line with the version like this: 17.

Fix Compatibility Problems

Depending on the service, your code may depend and make use of some Java libraries, bundled with the JVM, that may have been retired from newer versions. This is a potentially list of examples.

  • Example: javax.databind.xml was used in some services was used simply for its functionality to decode base64. For this, we recommend using scodec-bits instead.

Deploy your application.

Once those upgrades are done, and your team accepts them, merge them and get them to Production. If you use Marathon, make sure to look into the sandboxes in UAT, to look for signs of trouble. For instance, you may have some JVM options and flags that were removed in newer versions of the JVM.

Configure New memory options in Environments

WARNING Note that this step must come after you have upgraded the JVM, since some of these options may not be recognised…

For many services we have found that new memory options improve the performance and are easy to configure, in the environments repo.

Each service configuration is composed from:

  • Baseline configuration, located in shared/jsonnet/marathon/apps/SERVICE.jsonnet, and
  • for each environment, a set of environment-specific overrides, located in ENV/marathon/SERVICE/marathon-app-overrides.libsonnet.

The way environments works is that Jsonnet, a json-editing program, applies the commands in the baseline file to assemble the Json. It is this Json, automatically generated that really carries the configuration sent to the environment.

For our JVM upgrades, we need to do the following:

Step 1: In the baseline file, add in env the following:

  mem: 500
  env+: {
    JAVA_TOOL_OPTIONS: '-Xlog:os+container=info -XX:MaxRAMPercentage=80 -XX:MaxHeapFreeRatio=20 -XX:MinHeapFreeRatio=5',
  },

Also, make sure that at the end of the .jsonnet file there is only this:

std.mergePatch(basic_marathon, overrides)

Step 2 In each environment-specific overrides, tweak the mem setting for the total memory of the container. Note that your JVM will adjust to occupy as much as it needs from the container.

In particular, when you do the first PR, try to ensure that this setting is not being modified.

Step 3 Remove any configurations that may be related to previous settings. When we were using the old JVM memory settings, we would have lot of variables, in Jsonnet, such as :

  heapMB:: 768,
  maxMetaspaceSizeMB:: 100,
  reservedCodeCacheSizeMB:: 150,
  maxDirectMemorySizeMB:: 10,
  typicalThreadCount:: 40,
  stackSizeMB:: 1,
  extraSpaceMB:: 70,
  youngGenSizeMB:: $.heapMB / 2,

With the new options you should not need these variables. Remove them, as well as any appearance of them in the overrides, and you should get a notably cleaner configuration :).