Full-Cycle Observability with Lightrun and Instana - Technical Docs¶

Introduction¶

Lightrun allows Developers and DevOps Engineers to add real-time logs, metrics and traces to live applications on demand. This means, in practice, that information that was once only accessible by pushing new code with more instrumentation or by profiling your application, can now be added on demand - right from the IDE.

In this post we’ll share how to integrate Instana and Lightrun in order to collect, emit and visualize real-time, code-level Lightrun Merics from a running application in an Instana Custom Dashboard.

Set up Instana¶

To get started with Instana, sign up for an Instana SaaS trial here.

We’ll need to have an Instana Host Agent installed on our host to get started.

Install Instana’s Agent¶

Lightrun Metrics - which are added during runtime - need to be pushed into Instana for us to visualize them in a dashboard.

When these metrics are piped from the IDE Plugin to the Lightrun Management Server, they can be exported out into a remote StatsD daemon. Since Instana supports StatsD, we could pipe the Metrics from StatsD into Instana - and to do so, the StatsD Sensor must be enabled in Instana’s agent configuration settings.

Before we dive into the configuration, let’s take a look at the eventual topology of the Lightrun Metrics ingestion:

Note that all Lightrun Metrics created in your application will be first piped to the Lightrun Management Server (you can either install it on-prem or use the hosted, Lightrun Cloud service) , then they will be piped into the StatsD Sensor on the Host Agent (usually running inside a container or as a system service), and from there they will be transmitted into the Instana SaaS instance itself for visualization.

Now that we’ve got the topology down, let’s start the configuration process:

1. Install Instana’s Host Agent on any machine that has at least one incoming port open (we’ll use that port for communication with the Lightrun Management Server).
2. Enable the StatsD Sensor in Instana by editing the configuration file <agent_install_dir>/etc/instana/configuration.yaml - specifically, edit the com.instana.plugin.statsd block so that appears like the snippet below:

com.instana.plugin.statsd:
  enabled: true
  ports:
    udp: 8125
    mgmt: 8126
  bind-ip: "0.0.0.0" # all IPs by default
  flush-interval: 10 # in seconds

Set Up Lightrun¶

Introduction to Lightrun¶

Lightrun is an IDE-based, real-time troubleshooting tool for live applications. That means, in practice, that you can ask your applications code-level questions and get on-demand answers, right in your IDE. In addition, Lightrun Metrics can be exported into Instana for easy consumption and visualization.

Lightrun works by adding an agent to your live applications. That agent (which we will install below) allows for real-time instrumentation - and exporting - of Lightrun Actions.

Lightrun’s agent can coexist with Instana’s agents without any issues, as both types of agents rely on different capabilities of the underlying system: Instana’s agent works on the operating system level - it is a separate binary or container[f] - while Lightrun’s agent works on the runtime level (the JVM, V8 etc…) and usually delivered as a runtime agent or language-level package.

In order to add Lightrun Actions to a Live Application, you need to install one of our IDE plugins (which we will also install below) and spring up a Lightrun Agent with your application.

Let’s install Lightrun!

Install Lightrun’s IDE Plugin¶

This example details how to install Lightrun with the Java agent and the IntelliJ plugin, but similar instructions apply to our Node.js and Python agents as well.

1. Sign up for a (free!) Lightrun account here.

2. Once you’ve signed up, you’ll be directed to the IDE Plugin installation page - please choose the relevant IDE from the screen:

   

3. Once the plugin is installed, the Lightrun sidebar will pop up, asking you to log in to the plugin. Click the “Login” button:

   

4. Once you’ve installed the IDE plugin, a green “Connected” button should show up on top of the plugin:

   

Install Lightrun’s Agent¶

Now that you’ve got the IDE plugin installed and configured, we can spin up our application with the Lightrun agent.

Before that, though, make sure you have an application ready to work with - the one we’ll add the Lightrun Logs & Metrics to. For the purposes of demonstration in this tutorial, we will use this sample app.

Now you can install the agent:

1. Clone the repository locally and open the example folder:

   git clone git@github.com:lightrun-platform/lightrun.git
   cd lightrun/examples/java-with-logging
   

2. Go back to the Lightrun Cloud website and click “Next”. In the next page download the Lightrun agent into the sample application folder:

   

3. Compile/pacakage the project in your IDE, or in the command line:

   ./mvnw package
   

4. Run the application with the Lightrun agent attached (instructions change based on your specific runtime, of course):

   java -agentpath:agent/lightrun_agent.so PrimeMainLogToFile
   

You should see the Lightrun agent attached to the application in the IDE’s sidebar:

Now that your application is up with the agent, we’d need to do one more step on the Lightrun side - configuring Lightrun’s StatsD integration.

1. In order for Instana to correctly consume Lightrun Logs and Metrics, we’d need to modify the agent.config file located in the agent’s folder. Find the following line in the file:

   com.lightrun.DynamicLog.FileHandler.formatter=none
   

   And change it to:

   com.lightrun.DynamicLog.FileHandler.formatter=json
   

This will cause all Lightrun Logs and Metrics to be emitted in JSON format, allowing Instana to consume and parse them in an easier way.

Enabling Lightrun’s StatsD Integration¶

In order to pipe Lightrun metrics to Instana, we would need to use a tool called StatsD. We’ve enabled the StatsD Sensor in Instana’s Host Agent in a previous part of the tutorial, and now need to configure Lightrun to work with the StatsD daemon.

To do so, we’ll enable the StatsD integration from the Lightrun Management Console:

1. Go to the management console.

2. Click on the “Manager” menu and then on “System Integrations”:

   

3. Click the “Settings” button on the StatsD integration:

   

4. Fill in the URL of the application host, i.e. where the Instana Host Agent is running, and port 8125 (or whichever port you’ve selected in the previous step):

   

Lightrun’s StatsD integration is now active, and will pipe all Lightrun Metrics your users create into the Instana.

Connect Lightrun and Instana¶

Our Application¶

This demonstration will use a simple Java application, but it's important to note that Instana supports a wide variety of technologies - you can browse the list of all supported technologies here.

The demo application that you've downloaded in the previous step includes a prime number counter that runs through all the numbers in a given range, and outputs - every 1000 numbers counterd - the amount of primes it found from start of execution.

We’ve compiled and run the application in previous stages, so now all that’s left is to add Lightrun Actions and see them in Instana!

Enrich existing Instana Metrics with Lightrun Metrics¶

Instana auto-instruments a large amount of information for you once you set up your application with it. Once you’re all set up with the Instana agent click the “Infrastructure” tab on the left side of the screen - you should see a view similar to this:

This map allows you to visually grasp the entirety of your system, and provides a great entrypoint into understanding what sits where in the topology. In order to observe the metrics of the specific host, click the instana/agent host and then “Open Dashboard”:

In the view that opens you can see CPU and RAM usage metrics for our application:

In addition, by clicking on the “Applications” tab and then “All Services”, like in the image below:

You will be able to see information about application-level information and not just information related to infrastructure:

Note, however, that Instana does not collect any type of code-level information by default, since we did not instrument any such metrics when we wrote the application. Lightrun offers a wide array of metrics you can choose to instrument in real-time:

1. Counter - Checks how many times a specific line of code was reached
2. Method Duration - Checks how much time a specific method took to execute
3. TicToc / Measure Duration - Checks how much time any arbitrarily-selected block of code took to execute
4. Custom Metric - Checks what is the value of any code-level variable over time

Using Lightrun, you can both instrument these metrics in real-time and visualize them inside Instana! Let’s learn how to do that - we'll start by opening IntelliJ again and adding a Method Duration metric into the application:

Let’s give it an indicative name, like InstanaLightrunMethodDuration:

And click OK. In order for metrics to be forwarded to StatsD we’ll also need to let the Lightrun agent know we’d like the metrics to be piped to the “Plugin”, which also automatically pipes the metrics to StatsD:

Once that’s said and done, let’s configure the visualization in Instana. To do so, go to your Instana home page and click the Instana logo, and then the “Create a dashboard” link:

Give it an indicative name like “Instana Lightrun Metrics”:

Once the dashboard is created, make sure the timing of the dashboard is set to “Live” and add a widget:

We want to add a time series graph:

In the data source, choose “Infrastructure and Platforms”:

In the metrics field, select “StatsD” as your aggregation:

In the search bar that opens, enter the metric name we chose before - “InstanaLightrunMethodDuration” to search for the metric - make sure you choose “Mean” (Lightrun exposes various aggregated metrics about the method invocations, and we’d like to see the mean this time around):

Call the widget “Time to IsPrime” and click create:

And now you have an indication of the mean time it took to check for primality in our application:

Conclusion¶

When preparing for difficult production bugs it’s always important to remember that building great infrastructure is not enough; it’s also important to be able to understand - in real time - what is happening with it. Using Instana’s and Lightrun’s observability capabilities, you will be able to observe the full stack of your production service - for infrastructure, applications and code-level information - right from inside Instana.

Troubleshooting¶

For general Lightrun troubleshooting information, please visit our troubleshooting docs.

Enable piping in the Lightrun Plugin¶

If you’re not seeing Logs or Metrics in Instana or in the Lightrun Console, make sure to pipe the Lightrun Actions - click on the the “pipe” symbol next to the running application, then click on "Plugin" or “Both”:

Select the correct time frame in Instana¶

Instana automatically filters displayed information according to a specified time range. If you’re not seeing the relevant Logs or Metrics it might be due to a wrong filter. Click the time range button at the top right of the screen and ensure it is configured correctly: