Investigating build failures with CloudBees Pipeline Explorer

Any of these sound familiar?

• Friday was day five of the sprint. Before lunch, I finished testing a cool new feature and incorporated all review comments. I then pushed changes to the PR branch and planned to let the CI build run checks during lunch. After lunch, the build failed. 

• Similar scenario - in GitHub, you're looking at all these PRs.  I want to make a change to the code. Merge the source code - typical stuff. Next I see these red X’s and green checks. Okay, why does this have a red X????

With any of these - it's up to you to figure out this error - find out what happened or what went wrong in the execution of a Pipeline quickly.  And if you need help - you need to construct the story filled with delicious breadcrumbs for someone else to figure out what the heck is happening. 

This usually entails checking failed stages in a pipeline log view.  Download and open the file in Visual Studio Code. Many intertwined log lines are searched for "failed" and "error." Lots of scrolling, wading, and searching the downloaded file.

That is a lot of steps to walk through when you have a full plate of features to deliver!  Pipeline debugging shouldn't be this complex, especially just FINDING the starting point of the error. 

Let's see how to troubleshoot these with the new CloudBees pipeline explorer.

NOTE: Jenkins will use what is in this Default notification URL setting to figure out what screen to take you to for pipeline troubleshooting.  If CloudBees Pipeline Explorer Plugin is installed and enabled, the default URL will point to the Explorer.  So if you see a different screen - check your Default notification URL in your Manage Jenkins->System view:

Let's assume you see your build failed in GitHub - you click on details to figure out what went wrong and are magically whisked away to the CloudBees Pipeline Explorer page.  The Pipeline Explorer can navigate the stages of your pipelines in a tree view that lets you filter the logs by stages in nested groups.   From this view - you can quickly see where the failures are.  

Note:  Icon for TreeView, clicking on this icon will expand or collapse this view.

• For this example - My pipeline has some nested parallelism and the failures are in my Branch B, C and D stages. You can drill down from my Parallel Stage and see how these errors trickled up.  You can expand each Branch and see exactly where the failures are and select each failure to see this exact point in the log (failure in Branch B for example)

• Or, as an alternative, you can click on the Unsuccessful Steps icon and only see the errors in the panel:

Note: Icon for Unsuccessful Steps, the orange circle indicates how many errors were found. Clicking this icon will expand or collapse this view. 

As you can see - the errors in red are easy to pinpoint - but what if the error doesn’t make sense or if it's in a pipeline owned by someone else? In this example - there are errors in the parallel branches B, C, and D, and you do not own the pipeline in Branch D,  or perhaps - the errors you see do not make sense and you need some advice on how to fix them. There are several ways to share the data with a coworker quickly: 

From the Unsuccessful Steps panel

• Select the “scroll to the line” icon and be brought directly to this error. 

• There, you can select the line by hovering next to the line and clicking on the stacked dots to copy the URL to the exact line of error: 

• You can also select a range of lines to share as well. Select the first line 299 through last line 304 and share this as a range of lines:

• The same can be done from the tree view by clicking on the error and selecting the same stacked dots to capture the lines to share. 

• Send this URL via slack, email - whatever works to get them to the right place quickly.

What if the person you need help from doesn’t have access to Jenkins? Perhaps there is a Java expert in-house who could help with this specific Java error, or maybe you are looking for advice from the wizards on StackOverflow.  How will you share this error without downloading the entire log and sharing this giant txt file for THEM to wade through? 

• From either panel - TreeView or Unsuccessful Stage - you can filter out everything except the desired stage by hovering over it and selecting the filter icon that appears next to it. 

• Your view will be updated to display only this filtered stage, and you can now download the smaller snippet to share/post.  If your log is small to start with - maybe this is not an issue as you and screenshot or attach a small file- but if you have huge MB-sized log files, this gives you a focused log to share with those outside of the Jenkins environment.

The new pipeline explorer can easily get to the error quickly. Additionally, the log viewport can be customized with your preferences for line numbers, displaying ANSI colors from the log text, word wrapping, and custom time formats for timestamps, among other preferences. Let's run through a few of these customizations to get this log viewer to your visualization standards.

• Near the bottom of the left-hand panel - you see this familiar tool/settings icon

• This is the User Preferences panel. Most as self-explanatory, but let me highlight a few:

• Show stage duration: Display the duration of each stage in the tree view

• Show node steps in tree: This option displays agents or nodes in the tree view. Some stages run on specific nodes/agents. So, if a specific stage is failing but on a specific node, it gives you a good indicator of where to check next. 

• Show Timestamps: ​​Choose between Absolute time (matching your system time) and Elapsed time(capturing your build duration time); set your timestamp pattern and timezone.

• Scroll further down in the User Preferences, and you will see the Customized behavior panel:

• Favorite View - Select your view here, or use the icons in the left panel to set your default view to tree view, Unsuccessful Steps view, etc.  

• Theme: Light/Dark mode. I prefer dark mode because bugs like the light (ha ha ehem…just testing to see if you are still reading).

• Scroll to last line on load - no forever scrolling - get right to the end.

• Scroll to failure: This gets you to the root cause of the issue. If it is not turned on….lots of stuff to wade through.  After it is turned on, BAM - right to the error. Regardless if you have parallel or sequential pipelines, the pipeline engine will know the root causes and get you right there.

You may have noticed some other icons in my screen capture examples. I walked you through the TreeView and the Unsuccessful Steps.  What about the others? 

The next two icons in this list are dependent on additional integrations. I will cover these integrations in my next blog in this series:

• Test Insights requires the plugin J-Unit to be installed and enabled.

• Build Failure Analyzer requires the plugin BFA to be installed and enabled.

Note: These icons appear if you use additional J-Unit or BFA plugins. If you do not have these installed/enabled, you will not see these icons.

This brings us to the remaining icons of Build Artifacts and Related builds:

• Build artifacts: Directly access the build artifacts with a link to the build artifacts report. Less clicking around trying to find the artifact. 

• Related Builds: Display the builds related to your current build. The tree view also displays the related builds in the stage in which the build was triggered. The related builds include the builds triggered in the current build and the origin (parent) build that triggered the current build.

Finally - let's cover the Search. That's pretty clear what it does, so why mention it if I am brought directly to the error(s) I must address? Well, you and I both know - there’s a lot of info within these logs themselves. Yes, I can be brought to the errors, but what if I want a summary of the errors with a specific fail code?  I can search the whole log or a filtered set of the log based on my tree view selection - just to see my specific “*Fail” code:

In this example, my log file is small but if you have a large file - say - 700MB - you can use the filter to narrow down just the loglines you need - to share, send, or work with yourself.

Note:  Search is disabled by default and must be enabled by giving users the Run/SearchLog RBAC permission.

The CloudBees Pipeline Explorer was created for Jenkins users by Jenkins fans - who were looking to improve current logviewer tools. From the feedback received, it was clear that those tools were not designed for the job-to-be-done, which is to find out what happened or what went wrong in the execution of a pipeline quickly.  That’s why we designed CloudBees Pipeline Explorer to allow you to troubleshoot pipelines faster, easier, and above all, all in one place. 

Hope this gave you a better look at how to use the CloudBees Pipeline Explorer. To learn more - be sure to visit our technical documentation or visit us on CloudBeesTV to see the “Troubleshooting pipelines with CloudBees Pipeline Explorer” getting started video.

Read the next two in the series of Investigating Build Failures with CloudBees Pipeline Explorer series:

• Blog 2: Investigating build failures: JUnit Integration with CloudBees Pipeline Explorer

• Blog 3: Investigating build failures: Build Failure Analyzer Integration with CloudBees Pipeline Explorer