Troubleshooting CI Test Failures in Ember JS with Jenkins: A Journey of Discoveries

Tests hold a prominent status in the Ember JS framework, with Qunit serving as the internal testing framework. Writing test cases is made effortless due to the framework's preconfigured support and settings.

However, challenges may arise, particularly when certain errors surface exclusively in the continuous integration (CI) environment and not during local development. These errors can stem from configuration issues or discrepancies in the written test cases.

In this article, I'll share a specific incident where an error manifested solely in the CI environment and how it was identified in the Jenkins logs after some adjustments.

My CI setup utilizes Jenkins, with pipelines designed to execute test jobs. Unexpectedly, the tests were failing and leading to timeouts.

Before delving into the timeout error, it's essential to shed light on Testem, a JavaScript test runner responsible for executing tests in web browsers.

While Testem successfully launched the browsers in headless mode, it encountered issues loading the test cases. Some of these errors were captured in the Jenkins default console log, but they didn't provide the necessary insights. To find a solution, I scoured the Testem Github repository for similar issues. Although I gained a better understanding of the problem, a definitive solution remained elusive.

Subsequently, I discovered that it was possible to log Testem-related information in a dedicated logger file, as detailed in the Ember CLI documentation. This discovery opened up a new avenue for resolving the issue.

To log Testem-related logs, we need to use

ember test --testem-debug <FILE-NAME>

I could easily access those logs during local testing and was thrilled to have them available in the CI environment. However, I encountered an issue: these files were absent from the Jenkins workspace.

It appears that to retain files for later analysis, we must make use of Jenkins Artifacts. To archive a file in a Jenkins pipeline using the Stages strategy, you can simply employ the following syntax at the conclusion of the stages:

post {
      always {
         archiveArtifacts artifacts: 'testem.log'
      }
    }

Or if you use it directly inside a node, something like below (which was in my case):

node('node-name'){
 try{
   .....
 }catch {
   ....
 }finally {
   archiveArtifacts artifacts: 'testem.log'
 }
}

Once the artifacts were configured, I successfully accessed those logs in Jenkins CI.

Thanks to the log information, I managed to identify the root cause: the host application couldn't establish a connection because I was utilizing a custom hostname. To address this issue, I needed to include these hosts in the pipeline script's list of allowed connections.

sudo -- sh -c "echo 127.0.0.1 localhost.demo.com >> /etc/hosts"

With this modification, the errors disappeared, and I regained the ability to run tests smoothly in the Jenkins CI environment.