Continuous Integration (CI) allows developers to run different type of validation processes automatically at commit / merge time, in order to identify problems earlier. As a result, configuring CI is inherently language and eco-system specific (see the
Release and deployment section below).
The Open Developer Platform facilitates adoption CI tools and services that have been either requested by projects or provided by vendors, like WhiteSource, GitHub and RedHat, who offered their services to FINOS project teams and improve their build pipelines.
Below are listed the CI engines that we have been adopted by FINOS projects; although in the past the majority of project used Travis CI, we've seen our community shifting towards GitHub Actions, following their new release in 2019.
GitHub gives the ability to create, reuse and share actions, which can be easily described as workloads that can be triggered on a wide range of events.
Actions can be only defined at repository level, creating a
YAML file in the
.github/workflows folder. The
Actions tab available in
github.com web UI will list the actions defined and show the execution logs.
The Open Developer Platform uses GitHub Actions for the following use cases:
To learn how to use GitHub Actions, start reading https://help.github.com/en/actions.
When it's time to start writing your first
.yml file, we found extremely helpful the following resources:
Example of Configuring Integration Testing
In order to run this sample (some Symphony sample bots), the Travis build needs to be configured to add the following items:
# Download certificate before the build script runs
before_script: "curl -s https://raw.githubusercontent.com/symphonyoss/contrib-toolbox/master/scripts/download-files.sh | bash"
# Define environment variables with Symphony Pod endpoints
At this point, the certificates are in place and downloaded into the build box; integration tests can resolve User Identity certificates by accessing environment variables.
The symphony-java-sample-bots was amongst the first Foundation projects that was enabled to run integration tests from Travis CI; the Travis build logs for this project show the info below.
- The (hidden) environment variables that identify User Identity certificates (in this case 2, one for the message sender, one for the message receiver)
# Setting environment variables from repository settings
$ export DOWNLOAD_PATH=[secure]
$ export DOWNLOAD_ITEMS=[secure]
$ export TRUSTSTORE_FILE=[secure]
$ export TRUSTSTORE_PASSWORD=[secure]
$ export BOT_USER_EMAIL=[secure]
$ export BOT_USER_CERT_FILE=[secure]
$ export SENDER_USER_EMAIL=[secure]
$ export SENDER_USER_CERT_FILE=[secure]
$ export DOWNLOAD_HOST=[secure]
$ export SENDER_USER_CERT_PASSWORD=[secure]
$ export BOT_USER_CERT_PASSWORD=[secure]
- The integration test logs of a successful run
T E S T S
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.132 sec - in org.symphonyoss.simplebot.EchoBotIT
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
Known Certificate Issues
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded at
This basically means that the certificate password is wrong; to validate the certificate using the following command:
openssl pkcs12 -info -nodes \
-in <certificate_path> \
Reminder for Foundation staff: every password symbol must be escaped by prefixing it with character ‘\’, before setting it as Travis environment variable.
Branch specific build commands
Travis CI does not provide a syntax to specify branch-specific build commands; however, it is possible to use the following bash syntax (applicable to most CI environments) to workaround it; in this example, the Maven build will use the
sonar profile only if the current branch is master.
- "[[ $TRAVIS_BRANCH =~ master ]] && mvn clean deploy -Pintegration-testing,sonar --settings settings.xml"
- "[[ $TRAVIS_BRANCH =~ dev ]] && mvn clean deploy -Pintegration-testing --settings settings.xml"
FINOS can provide projects with Azure Pipelines integration on https://dev.azure.com/finosfoundation.
You can follow docs on the Microsoft docs page
Myget is a friction-free continuous integration & delivery for your nuget, npm, bower and vsix packages; the Foundation provides package repositories, which allow:
- Pre-release build and publishing, using build environments that are equipped with .NET Framework and Visual Studio (headless); it also features source code tagging and version updates on source code based on incremental build number, to fully automate the package publishing pipeline
- Sync with NuGet, that can be either manual (promoting pre-releases to releases using MyGet web interface) or automatic (a committer must create a personal account on MyGet and request access from the Foundation in order to be able to push packages)
In order to request a project to be integrated with MyGet, a Project lead can sign up to MyGet and email email@example.com the following info:
- Project coordinates - GitHub project url and other useful info
- MyGet username
- CSProj and CS files - Where project descriptors are located
- Packages - A list of the packages that need to be published; by default all packages are taken into account
- Publishing strategy to NuGet - Whether to enable automatic publishing based on source code branching or rely on manual package pushing using the MyGet interface; if the latter, a MyGet personal account username must be provided
An example of project building with MyGet is the Symphony RestApiClient project, which is also published on the FINOS MyGet repository and NuGet.
Badges can be added at the top of the project's root-level
README.md file, using the following Markdown syntax:
- Build status:
[![MyGet Build Status](https://www.myget.org/BuildSource/Badge/<feed name>?identifier=<guid>)](https://www.myget.org/feed/<feed name>/package/nuget/<package name>)
- Latest Pre-Release package published:
[![MyGet Pre Release](https://img.shields.io/myget/<feed name>/v/<package name>.svg)](https://www.myget.org/feed/<feed name>/package/nuget/<package name>)
Release and deployment
Continuous (artifact) deployment in Java
When using Java, Travis CI can be easily configured to publish new snapshot artifacts on commit; to enable this feature, a project committer can follow these simple steps:
- Follow the Java Snapshot deployment configuration ; as a result, you should have username/password credentials (mentioned below as
CI_DEPLOY_PASSWORD) to access issues.sonatype.org
- Email firstname.lastname@example.org to request permission to deploy artifacts on Maven remote repositories; this action is not required if the developer has already been granted access previously
- Commit a
settings.xmlfile in the project root folder
CI_DEPLOY_PASSWORDvariables with Travis CI; make sure that they're encrypted and hidden during the build process; the credentials to use are the ones defined on step 1; if you don't want to share your username/password credentials, you can request and use Nexus tokens
- Change the
mvnbuild command to a. Invoke the
deploygoal b. Append
--settings settings.xmlat the end of the build command
This tutorial walks through the mentioned steps, with more details and configuration options.
<settings xmlns="https://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd">
Continuous release in Python
Travis CI can be configured to use python-semantic-release, a package that simplifies and automates versioning for python projects; a project lead can email email@example.com to request Foundation staff to apply the proper Travis CI project settings; packages will be published on behalf of the FINOS pypi user.
Continuous release in NodeJS
Travis CI can easily publish packages to npm using semantic-release.org, which delegates release operations to your CI environment and allows you to control versioning using commits (commitizen); as a result, npm packages will be continuously released on each code repository change.
Follow our instructions on how to register a package and user in the Foundation npm organisation.