Skip to content

Full-Cycle Observability with Lightrun and Instana - Technical Docs

Introduction

Permissions

Only users with ROLE_MANAGER permissions can perform these actions.

Lightrun allows Developers and DevOps Engineers to add real-time logs, metrics and traces to live applications on demand. This means, in practice, that information that was once only accessible by pushing new code with more instrumentation or by profiling your application, can now be added on demand - right from the IDE.

In this post we’ll share how to integrate Instana and Lightrun in order to collect, emit and visualize real-time, code-level Lightrun Merics from a running application in an Instana Custom Dashboard.

Set up Instana

To get started with Instana, sign up for an Instana SaaS trial here.

We’ll need to have an Instana Host Agent installed on our host to get started.

Install Instana’s Agent

Lightrun Metrics - which are added during runtime - need to be pushed into Instana for us to visualize them in a dashboard.

When these metrics are piped from the IDE Plugin to the Lightrun Management Server, they can be exported out into a remote StatsD daemon. Since Instana supports StatsD, we could pipe the Metrics from StatsD into Instana - and to do so, the StatsD Sensor must be enabled in Instana’s agent configuration settings.

Before we dive into the configuration, let’s take a look at the eventual topology of the Lightrun Metrics ingestion:

Architecture Diagram

Architecture Diagram

Note that all Lightrun Metrics created in your application will be first piped to the Lightrun Management Server (you can either install it on-prem or use the hosted, Lightrun Cloud service) , then they will be piped into the StatsD Sensor on the Host Agent (usually running inside a container or as a system service), and from there they will be transmitted into the Instana SaaS instance itself for visualization.

Now that we’ve got the topology down, let’s start the configuration process:

  1. Install Instana’s Host Agent on any machine that has at least one incoming port open (we’ll use that port for communication with the Lightrun Management Server).
  2. Enable the StatsD Sensor in Instana by editing the configuration file <agent_install_dir>/etc/instana/configuration.yaml - specifically, edit the com.instana.plugin.statsd block so that appears like the snippet below:
com.instana.plugin.statsd:
  enabled: true
  ports:
    udp: 8125
    mgmt: 8126
  bind-ip: "0.0.0.0" # all IPs by default
  flush-interval: 10 # in seconds

StatsD Ports

While 8125 and 8126 are commonly used in this scenario, you can choose any port you’d like.

Set Up Lightrun

Introduction to Lightrun

Lightrun is an IDE-based, real-time troubleshooting tool for live applications. That means, in practice, that you can ask your applications code-level questions and get on-demand answers, right in your IDE. In addition, Lightrun Metrics can be exported into Instana for easy consumption and visualization.

Lightrun works by adding an agent to your live applications. That agent (which we will install below) allows for real-time instrumentation - and exporting - of Lightrun Actions.

Lightrun’s agent can coexist with Instana’s agents without any issues, as both types of agents rely on different capabilities of the underlying system: Instana’s agent works on the operating system level - it is a separate binary or container[f] - while Lightrun’s agent works on the runtime level (the JVM, V8 etc…) and usually delivered as a runtime agent or language-level package.

In order to add Lightrun Actions to a Live Application, you need to install one of our IDE plugins (which we will also install below) and spring up a Lightrun Agent with your application.

Let’s install Lightrun!

Install Lightrun’s IDE Plugin

This example details how to install Lightrun with the Java agent and the IntelliJ plugin, but similar instructions apply to our Node.js and Python agents as well.

  1. Sign up for a (free!) Lightrun account here.
  2. Once you’ve signed up, you’ll be directed to the IDE Plugin installation page - please choose the relevant IDE from the screen:

    Choose IDE
    Choose IDE

  3. Once the plugin is installed, the Lightrun sidebar will pop up, asking you to log in to the plugin. Click the “Login” button:

    Log into the Plugin
    Log into the Plugin

  4. Once you’ve installed the IDE plugin, a green “Connected” button should show up on top of the plugin:

    Connected Plugin
    Connected Plugin

Install Lightrun’s Agent

Now that you’ve got the IDE plugin installed and configured, we can spin up our application with the Lightrun agent.

Before that, though, make sure you have an application ready to work with - the one we’ll add the Lightrun Logs & Metrics to. For the purposes of demonstration in this tutorial, we will use this sample app.

Now you can install the agent:

  1. Clone the repository locally and open the example folder:

    git clone git@github.com:lightrun-platform/lightrun.git
    cd lightrun/examples/java-with-logging
    
  2. Go back to the Lightrun Cloud website and click “Next”. In the next page download the Lightrun agent into the sample application folder:

    Install the Lightrun Agent
    Install the Lightrun Agent

  3. Compile/pacakage the project in your IDE, or in the command line:

    ./mvnw package
    
  4. Run the application with the Lightrun agent attached (instructions change based on your specific runtime, of course):

    java -agentpath:agent/lightrun_agent.so PrimeMainLogToFile
    

You should see the Lightrun agent attached to the application in the IDE’s sidebar:

Agent Showing in Sidebar

Agent Showing in Sidebar

Now that your application is up with the agent, we’d need to do one more step on the Lightrun side - configuring Lightrun’s StatsD integration.

Java Debug Symbols

The -g flag, that is used for generating debugging information during compilation, is required when using Lightrun. Please note that this is done automatically by most IDEs nowadays.

  1. In order for Instana to correctly consume Lightrun Logs and Metrics, we’d need to modify the agent.config file located in the agent’s folder. Find the following line in the file:
    com.lightrun.DynamicLog.FileHandler.formatter=none
    

    And change it to:

    com.lightrun.DynamicLog.FileHandler.formatter=json
    

This will cause all Lightrun Logs and Metrics to be emitted in JSON format, allowing Instana to consume and parse them in an easier way.

Enabling Lightrun’s StatsD Integration

In order to pipe Lightrun metrics to Instana, we would need to use a tool called StatsD. We’ve enabled the StatsD Sensor in Instana’s Host Agent in a previous part of the tutorial, and now need to configure Lightrun to work with the StatsD daemon.

To do so, we’ll enable the StatsD integration from the Lightrun Management Console:

  1. Go to the management console.
  2. Click on the “Manager” menu and then on “System Integrations”:

    Lightrun Management Console
    Lightrun Management Console

  3. Click the “Settings” button on the StatsD integration:

    Integrations Screen
    Integrations Screen

  4. Fill in the URL of the application host, i.e. where the Instana Host Agent is running, and port 8125 (or whichever port you’ve selected in the previous step):

    StatsD Settings
    StatsD Settings

Lightrun’s StatsD integration is now active, and will pipe all Lightrun Metrics your users create into the Instana.

Connect Lightrun and Instana

Our Application

This demonstration will use a simple Java application, but it's important to note that Instana supports a wide variety of technologies - you can browse the list of all supported technologies here.

The demo application that you've downloaded in the previous step includes a prime number counter that runs through all the numbers in a given range, and outputs - every 1000 numbers counterd - the amount of primes it found from start of execution.

We’ve compiled and run the application in previous stages, so now all that’s left is to add Lightrun Actions and see them in Instana!

Enrich existing Instana Metrics with Lightrun Metrics

Instana auto-instruments a large amount of information for you once you set up your application with it. Once you’re all set up with the Instana agent click the “Infrastructure” tab on the left side of the screen - you should see a view similar to this:

Instana Infrastructure View

Instana Infrastructure View

This map allows you to visually grasp the entirety of your system, and provides a great entrypoint into understanding what sits where in the topology. In order to observe the metrics of the specific host, click the instana/agent host and then “Open Dashboard”:

Host Details

Host Details

In the view that opens you can see CPU and RAM usage metrics for our application:

Usage Metrics

Usage Metrics

In addition, by clicking on the “Applications” tab and then “All Services”, like in the image below:

Instana Services

Instana Services

You will be able to see information about application-level information and not just information related to infrastructure:

Service Metrics

Service Metrics

Note, however, that Instana does not collect any type of code-level information by default, since we did not instrument any such metrics when we wrote the application. Lightrun offers a wide array of metrics you can choose to instrument in real-time:

  1. Counter - Checks how many times a specific line of code was reached
  2. Method Duration - Checks how much time a specific method took to execute
  3. TicToc / Measure Duration - Checks how much time any arbitrarily-selected block of code took to execute
  4. Custom Metric - Checks what is the value of any code-level variable over time

Using Lightrun, you can both instrument these metrics in real-time and visualize them inside Instana! Let’s learn how to do that - we'll start by opening IntelliJ again and adding a Method Duration metric into the application:

Adding a Method Duration

Adding a Method Duration

Let’s give it an indicative name, like InstanaLightrunMethodDuration:

Naming Our Method Duration

Naming Our Method Duration

And click OK. In order for metrics to be forwarded to StatsD we’ll also need to let the Lightrun agent know we’d like the metrics to be piped to the “Plugin”, which also automatically pipes the metrics to StatsD:

Pipe to Plugin

Pipe to Plugin

Once that’s said and done, let’s configure the visualization in Instana. To do so, go to your Instana home page and click the Instana logo, and then the “Create a dashboard” link:

Create a New Instana Dashboard

Create a New Instana Dashboard

Give it an indicative name like “Instana Lightrun Metrics”:

Naming Our Instana Dashboard

Naming Our Instana Dashboard

Once the dashboard is created, make sure the timing of the dashboard is set to “Live” and add a widget:

Add a Widget

Add a Widget

We want to add a time series graph:

Adding a Time Series Graph

Adding a Time Series Graph

In the data source, choose “Infrastructure and Platforms”:

Choosing a Data Source

Choosing a Data Source

In the metrics field, select “StatsD” as your aggregation:

StatsD Aggregation

StatsD Aggregation

In the search bar that opens, enter the metric name we chose before - “InstanaLightrunMethodDuration” to search for the metric - make sure you choose “Mean” (Lightrun exposes various aggregated metrics about the method invocations, and we’d like to see the mean this time around):

Choose Exported Method Duration

Choose Exported Method Duration

Call the widget “Time to IsPrime” and click create:

Naming the Widget

Naming the Widget

And now you have an indication of the mean time it took to check for primality in our application:

Pretty Graph!

Pretty Graph!

Conclusion

When preparing for difficult production bugs it’s always important to remember that building great infrastructure is not enough; it’s also important to be able to understand - in real time - what is happening with it. Using Instana’s and Lightrun’s observability capabilities, you will be able to observe the full stack of your production service - for infrastructure, applications and code-level information - right from inside Instana.

Troubleshooting

For general Lightrun troubleshooting information, please visit our troubleshooting docs.

Enable piping in the Lightrun Plugin

If you’re not seeing Logs or Metrics in Instana or in the Lightrun Console, make sure to pipe the Lightrun Actions - click on the the “pipe” symbol next to the running application, then click on "Plugin" or “Both”:

Toggle Piping

Toggle Piping

Select the correct time frame in Instana

Instana automatically filters displayed information according to a specified time range. If you’re not seeing the relevant Logs or Metrics it might be due to a wrong filter. Click the time range button at the top right of the screen and ensure it is configured correctly:

Choose Timeframe

Choose Timeframe

Last update: September 20, 2022