Make Server Logs Fun Again

Everyone hates server logs. Well, maybe not everyone. But you know what I mean. We often go to them when there is an error. And the process of finding an error code among the web of HTTP requests can be a nightmare. Is there a better way?

Image for post
Image for post

(I blurred out visitor’s IP addresses to protect their privacy!)

I wanted to write this article as postmortem to server logs. I don’t want to have to deal with them anymore beyond the current setup explained here.

I recently got into PM2 which is a Node instance manager. As you may know running node index.js works okay for small projects and local environments. But it is never recommended to run your production server in this way.

(You will eventually get the Broken Pipe error if you leave it running for too long with default settings…or your ISP disconnects, another SSH client logs in to the server or your system shuts down due to a lightning strike! Can happen.)

If the error is not generated by something completely outside of your control you can fix the Broken Pike message by tweaking default timeouts. I won’t go into detail here but you can run following commands in your cli/bash:

echo 'ServerAliveInterval 30' | tee -a ~/.ssh/config
echo 'ServerAliveCountMax 1200' | tee -a ~/.ssh/config

This will basically increase “keep alive” intervals in your Linux system. But this isn’t enough. What if your server goes down due to coding error?

PM2

PM2 is great because it will relaunch your server if there is a minor error or even a memory leak. It’s like having someone watch out for your server.

Most people discover PM2 by realizing node index.js fails too often and if you get disconnected from your remote server by terminal or in bash.exe the node index.js command breaks.

PM2 is not the only tool for keeping your server running no matter what. And I’m not saying it’s the best one either. It’s only one of many that helps you keep your server alive even if an API request, for example, generates an error.

The most frequently PM2 commands I use are:

1.) pm2 start index.js --no-daemon (run and output log on the screen)

(Note, without --no-daemon directive, you won’t get on-screen logs)…and…

2.) pm2 kill (ends PM2 session and shuts down the index.js app)

There are other PM2 commands (like pm2 monitor) but for some reason I haven’t really needed to use any of them.

3.) pm2 dashboard opens this view with server memory analytics:

Image for post
Image for post
An example of pm2 dashboard command

I subscribed to my own newsletter, and my server outputs email in purple! So we’re putting boring logs to a better use.

You have to use pm2 kill to stop the server started with pm start if you want it to stop. If you simply exit PM2 with Ctrl-C (pressed two times in a row,) you will only exit from the on screen log monitor but the PM2 will still keep your server running. Which is exactly what you want to use PM2 for.

No more Broken Pipe messages! Log in or out of your SSH, have your PC’s power supply get short circuited by lightning…and it won’t affect your remote server that was launched with PM2.

So how does that make servers fun? It doesn’t. What I wanted to talk about in this article has more to do with the traffic we will track via our PM2 logs.

Can Server Logs Ever Be Fun?

Watching server logs is tedious. So you received a successful 200 request. But does it really need to be shown on the console? Your analytics software is probably already tracking that. So what kind of things can we track?

It’s About The Intent of Your App

Traditionally we only go to server logs when there is a serious error that happened on the back end.

But if you are running an application based on concrete events (user registration, newsletter sign ups, sales, important API calls, etc.) sifting them out from general traffic will not only save space on your server (since you’re no longer logging commonplace events,) but can make watching the server fun!

Now your server is telling you what went right.

Let’s say there is a newsletter sign up. You can log into your Mailchimp or other account to check it. Or you can be notified of it directly via your server log which you would always have open on your screen.

Reducing Junk Requests

No longer track .html or other content files. Why? Your analytics will take care of that. A useful website usually has far more interesting events happening.

Focusing Only On Positive Events

Server logs are known for watching errors. But why can’t we use them to output positive events like sign ups and sales?

Reducing Attacks

Server attacks (not always) but often come from a single IP address. Implement an IP tracker, and if your site is accessed 100 times per minute or you spot an unusual pattern you can disqualify that IP address.

Optimized For Usefulness

As someone who owns and runs a server, you have access to IP addresses of each visitor. They can be converted to geo-location in real time. Based on this information you can feed a different version of your content.

For example, when I was selling my books, many sent me a DM on Twitter saying $34.99 is a fortune (to pay for a book) in their country. So what I did was I reduced the price to $9.99 for countries where $34.99 is a lot of money.

This resulted in a few side-sales from those regions from visitors that otherwise wouldn’t even purchase the book due to price differences.

Now you have a happy customer who can afford to buy your book and you got an extra sale. It was at reduced price but it’s better than missing that opportunity. Everyone wins. All it took was a small tweak to the server.

(Which was basically an if-statement that triggered change to payment link.)

(Sure, people can get a deal by connecting to the sale page by proxy server.)

(But I am not responsible for what people choose to do, only for trying to serve the customers in the best way possible.)

Color Coding Events

Finally, this is actually what makes server logs fun for me. If you color code your events, you can get a good picture of what just happened by simply glancing at the screen.

console.log method can produce color text output. On ode server you can also use a module called colors (install npm install colors) which has a limited set of colors but it’s not a huge issue:

For example: console.log(“ABC”.lightGreen); produces light green text.

You can color code server events using this module. Just branch out with an if statement based on a type of event received. The .lightCyan is another favorite. (cyan and other colors have “light” counterpart in colors module.)

magenta or lightMagenta can indicate newsletter sign ups, green can indicate sales, yellow can be used on API calls coming from particular geo-location and so on and so forth…

You just have to get creative and work out your own color patterns based on what’s important to achieve for your application. It could be image uploads, rankings or comments.

Visualize Data Color Coded Charts

Now that you got your server events color coded, it’s time to visualize them. I quickly coded this analytics output using CSS flex with flex-bottom align for each item inside each bar.

Finally, you can visualize that data in a simple color-coded chart. Here is a test from one of my own experiments for one of my websites.

Image for post
Image for post
Simple color-coded analytics made up based on server-side actions.

I superimposed colorful server-side statistics over my Twitter follower stats. I also re-scaled the data based on averages. I wanted to keep important events visible but I didn’t want them to create ugly high peaks.

So here about 1K “hits” is nicely scaled to something that can be aesthetic.

Purpose: This will help me analyze in the future how creating & distributing new content impacts my Twitter followers and a few other things.

Now each event can be visually deciphered without having to go through lists of HTTP requests. We are not tracking errors, only positive events. In a way we are merging traffic and important calls with self-made analytics software.

(Let’s face it, it’s difficult to set up custom events in some of the well-known analytics software…even if it produces great charts, it’s not tailored to specific purpose of your app. And there are no APIs to log into, just build it all yourself.)

I won’t tell you exactly what each color means in my configuration. But the way I structure them is that the brightest colors at the bottom (yellow in this case) relates to the most important event on the entire server like a newsletter sign up. The orange bars indicate how many times the newsletter sign up form has been viewed. So this way you can see conversion rate visually!

Now you see how we converted boring server logs to something you can actually get excited about. Your work can be reduced to making more content and sharing with the community…and watch the results outlined in this custom color coded analytics chart!

No lists or error logs…just with a simple glance you get to see all important events and focus just on the work that will produce more of them.

Written by

Issues. Every webdev has them. Published author of CSS Visual Dictionary https://amzn.to/2JMWQP3 few others…

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store