Tag Archives: OpenWRT

Computers

September 2023 ROOter updates

Leave a comment

I’ve had a few chances over the last few weeks to make minor updates to the autobuild systems. Nothing major. This month’s news:

  • The autobuild status system has been reconfigured to update every minute, with an approximately one minute lag.
  • The status has also been added to a new Autobuilds headline page, easily accessible from the top menu. This is probably the quickest way to see the current system status. It doesn’t auto update, though, you’ll have to refresh occasionally.
  • A while back we were able to finish uploading all of the old autobuild images to carp4’s Autobuild archive space. Everything should be there, from just after the first runs, sorted by router name. You may find a few routers that are split into several folders due to name changes over time, but everything seems to be where it belongs. This archive also updates with new images within a day or so of them being built.
  • The old 18.06.7 system was retired about a week ago. For at least a year or more it had only been building the WR703N, which barely fit an image with the older kernel. New images no longer fit, so this one is finally wound down.

I have a few more tweaks on my wish list, but everything gets fit between the two real jobs, both of which have been hopping this year. Maybe more time will appear as things go on.

Computers

ROOter Autobuilds Archive

Leave a comment

This has already been linked from the forum and the autobuilds page, but still great news. We only have space on the server to host the most recent two images for each router. However, user carp4 has donated a significant amount of space on a Google Drive to host all of the older images. Without further ado:

ROOter Autobuild Archive

It will take two to three months to upload all of the older images – there are almost half of a terabyte worth already. They’re all on a local drive now, so will trickle online during off-peak hours overnight. I rough estimate would be end of June, 2023 to have them all online, if all goes well.

Computers

Filtering logd in OpenWRT

Leave a comment

So I’ve been running into an odd problem with log spam in my ring buffer log, and just kludged together a solution when I couldn’t find one elsewhere. I use nft-qos on my main OpenWRT router, and some days it spams my logs like crazy with lines that look like this:

Aug 22 04:15:11 hexenbiest nft-qos-monitor: ACTION=update, MACADDR=a8:86:dd:ac:73:ad, IPADDR=192.168.1.191, HOSTNAME=
Aug 22 04:15:12 hexenbiest nft-qos-dynamic: ACTION=update, MACADDR=a8:86:dd:ac:73:ad, IPADDR=192.168.1.191, HOSTNAME=
Aug 22 04:26:46 hexenbiest nft-qos-monitor: ACTION=update, MACADDR=a4:08:01:3c:9e:bb, IPADDR=192.168.1.207, HOSTNAME=FireStick
Aug 22 04:26:47 hexenbiest nft-qos-dynamic: ACTION=update, MACADDR=a4:08:01:3c:9e:bb, IPADDR=192.168.1.207, HOSTNAME=FireStick
Aug 22 04:26:50 hexenbiest nft-qos-monitor: ACTION=update, MACADDR=a4:08:01:3c:9e:bb, IPADDR=192.168.1.207, HOSTNAME=FireStick
Aug 22 04:26:50 hexenbiest nft-qos-dynamic: ACTION=update, MACADDR=a4:08:01:3c:9e:bb, IPADDR=192.168.1.207, HOSTNAME=FireStick
Aug 22 04:26:59 hexenbiest nft-qos-monitor: ACTION=update, MACADDR=a4:08:01:3c:9e:bb, IPADDR=192.168.1.207, HOSTNAME=FireStick
Aug 22 04:26:59 hexenbiest nft-qos-dynamic: ACTION=update, MACADDR=a4:08:01:3c:9e:bb, IPADDR=192.168.1.207, HOSTNAME=FireStick
Aug 22 04:27:15 hexenbiest nft-qos-monitor: ACTION=update, MACADDR=a4:08:01:3c:9e:bb, IPADDR=192.168.1.207, HOSTNAME=FireStick
Aug 22 04:27:15 hexenbiest nft-qos-dynamic: ACTION=update, MACADDR=a4:08:01:3c:9e:bb, IPADDR=192.168.1.207, HOSTNAME=FireStick

That’s just a sampling – it’s not just the one device, more often it will spit out a list of every device on the network. Sometimes it does that multiple times per minute. Other times, it’ll go days and never do it once. I don’t have time to track down why nft-qos is logging this, or figure out how to turn it off, but I did have good luck with this kludge to filter it out.

Most people won’t need to filter this unless you’re using your logs for something else. I send my logs to an rsyslog server in my LAN and filter it there, so this isn’t an issue in my long term logs – only in the ring buffer copy that stays on the router itself. So why does it matter? I have a script that runs once a minute to count missed pings from mwan3 and update a conky dashboard on one of my monitoring servers, and it’s much more efficient if I have it parse the ring buffer, instead of asking the rsyslog server across the network once a minute for the whole day’s log!

In hunting around, I’ve seen a dozen or so people asking how to filter logs on OpenWRT, and the answer is universally to either implement rsyslog on the router, or send it to an external syslog server. My case is obviously a corner case, but it’s one that’s obviously come up enough that other people have asked, and I’ve never seen a solution that actually works with logd and logger on the box itself without an external server.

So I made one. And it turned out to be ridiculously simple. It’s also very easy for you to modify to filter out anything else you need to as well.

The Fix, without further ado

What you need to know: logger on OpenWRT isn’t a real program. Entering “type logger” in your shell will point to /usr/bin/logger, but that’s not a real file, just another link to /bin/busybox. When you type

logger -t testlog "Some test message"

into your terminal on OpenWRT, it just gets redirected into

/bin/busybox logger -t testlog "Some test message"

and busybox does the work. Filtering the log input is as simple as moving the link at /usr/bin/logger out of the way, and creating the following new ash script at /usr/bin/logger:

#!/bin/ash
# We are going to filter input to logger by intercepting it.

# Parse options if they're there, else use defaults.
STDERR=1
TAG=$USER
PRIO=0

while getopts st:p: flag ; do
        case "${flag}" in
                s) STDERR=0 ;;
                t) TAG=${OPTARG} ;;
                p) PRIO=${OPTARG} ;;
                *) exit 1 ;;
        esac
done

shift $((OPTIND -1))

# now filter the input string and exit early if we pick up a filtered string.

# Message filters (chain more if needed):
FILTERED=`echo "$*" | grep -v 'ACTION=update,'`

# Exit trigger lines (add more if needed):
[ -z "$FILTERED" ] && exit 0

# We have a good filtered string. Send it to the real logger.

if [ $STDERR -eq 1 ] ; then
        echo "$FILTERED" | /bin/busybox logger -t $TAG -p $PRIO
else
        echo "$FILTERED" | /bin/busybox logger -s -t $TAG -p $PRIO
fi

Now, here’s what that script does:

  • Parse any flags on the input using getopts. Busybox logger only accepts the three flags shown, so this is fairly simple. We’re setting defaults for all three, and then if a flag is actually given, replacing the defaults with what was given.
  • Shift the input over by the number of flags and flag parameters, leaving only the log message.
  • Using grep -v (think “grep inverse”) to filter according to the message content. This FILTERED line is either going to return exactly what you give it unchanged, or nothing at all if it matches your filter. It’s easy to add more filters here if needed, more info below.
  • Checking to see if the message has been filtered out (the code “[ -z “$FILTERED” ]” just returns success if the string you feed it is zero length) and exiting the script if so, basically discarding this log entry. This line is important to remember later if you want to filter on things like tags or priority, so remember it as an “exit trigger line.”
  • Finally, passing the message to the REAL logger facility, using either the default flag values or the values given in the input. The syslog flag is a true or false, and in doing this quickly I found it easiest to just split that using an if statement.

Modifying the script for your own purposes

Want to know how to modify this to make it your own, but not a bash scripting guru? It’s pretty simple, so I’ll give you the framework for the most common changes.

Changing it to filter based on different text is the easiest – just change what’s in the single quotes on the filter line. If you wanted to filter out messages where the content is “foobar” for example:

FILTERED=`echo "$*" | grep -v 'foobar'`

Changing it to filter multiple different things is also fairly easy. You just need to duplicate the filter line, and change any lines after the first to work with the output of the line before it instead of $*. You can chain together however many you want, at the end FILTERED is either going to contain your original input or just be a zero length “” string if a filter matched. If we wanted my original filter plus foo and bar:

FILTERED=`echo "$*" | grep -v 'ACTION=update,'`
FILTERED=`echo "$FILTERED" | grep -v 'foo'`
FILTERED=`echo "$FILTERED" | grep -v 'bar'`

You can change it to match anything in the message this way. One note, grep as it’s called there is also case sensitive, and all the other usual caveats around that kind of thing still apply – but overall, it’s easy to tweak these message filters any way you want.

What about filtering based on a whole tag? What if you want to filter out all log entries with the tag “testlog”? Now, instead of modifying the filter string, you need to modify the exit trigger line (if you only want to filter on the tag, not the message) or add additional exit trigger lines if you want to filter on both message and tags.

Example, filtering on one or more messages as well as tag “testlog”:

[ -z "$FILTERED" ] && exit 0
[ "$TAG" = "testlog" ] && exit 0

You can add as many exit triggers as you need – matching any one of them will cause the message to be discarded. What if you also want to filter by message priority, discarding any message with priority lower than (number higher than) 5? This is a functionality that’s been broken in OpenWRT since 2013, but it turns out it’s also trivial to fix by adding an exit trigger line:

[ -z "$FILTERED" ] && exit 0
[ "$TAG" = "testlog" ] && exit 0
[ "$TAG" = "foobar" ] && exit 0
[ "$PRIO" -gt 5 ] && exit 0

That one filters one one or more messages, plus two tags and priority. As you can see, you really can easily tweak this to do whatever you want.

In short, log filtering has been broken ever since the switch to logd, and it turns out it was always trivial to implement and no one has bothered. I’ve kludged it myself, but I don’t have the time to go implement a real replacement in the original source either – I wish I did! If you do, please feel free to take this framework and run with it. It would be almost as easy to add reading of filter settings from uci into this script so that we could actually make settings in /etc/config/system like conloglevel functional again. You could trivially include lots of message filter strings in /etc/config/system, and have the script loop through them instead of hard coding it like my examples above. Really, this is a trivial framework for fixing OpenWRT’s long-broken log filtering, and all it needs is someone with time and willingness to run with it.

But meanwhile, since you’re probably like me and just looking for a quick way to fix this – here you go, tweak as you see fit and enjoy your spam-free ring buffer logs!

Computers

Quick and Dirty Live View of rsyslog Output

Leave a comment

I mentioned in a post yesterday that I was watching the syslog of my router to see when it sent a boot image over tftp. However, OpenWRT does not have a “live updating” syslog view – so how was I doing that, just clicking refresh over and over with my third hand, while holding down a reset button with the other two? No, there’s a really easy way to do this with a stupidly simple bash script.

My routers use remote logging to an internal rsyslog server on my LAN, and you’ll see my script is set up to support that. However, this is very easy to modify to support OpenWRT’s default logging, as well.

Without further ado, here’s the script, which lives in my log folder:

#!/bin/sh

# Live log display

while true; do
        tail -n 52 $1
        sleep 5
done

My various consoles I log into this from have anywhere from a 40 to 50 line display set up, hence the “52” – it’s reading and displaying the last 52 lines of the log every five seconds. If you always use shorter consoles, you can easily trim this down.

By passing the name of the script you want to read, this script has also been made “universal” in that it can be used to monitor any log on your machine. I also use it on a couple of my other servers, with no modifications. If you want to monitor “hexenbiest.log” you simply cd into the appropriate log folder, and run:

./loglive hexenbiest.log

Stock OpenWRT doesn’t write the log to a real file, it uses a ring buffer in memory that may be accessed using the command logread. To modify this script to run on a stock OpenWRT router, place it in the home folder (/root/) instead, and modify it to read accordingly:

#!/bin/sh

# Live log display

while true; do
        logread | tail -n 52
        sleep 5
done

This way, instead of reading the last 52 lines of a file every five seconds, it’s reading the last 52 lines of the logread output instead.

You might think it would make sense to clear the terminal before each output, but I didn’t personally find that helpful. In fact, it resulted in a visible flicker every time the log updated. Helpful if you need to know each time it reads, but I didn’t find that useful myself.

Computers

Using dnsmasq under OpenWRT as a TFTP boot server

Leave a comment

Update (2022-03-13): this past year, Mikrotik did something that broke their ability to netboot over DHCP (affecting any RouterOS version newer than 6.46.6), which makes flashing these routers using most people’s usual methods (often tftp32, which is distributed with ROOter images, for example) much more difficult. The method in this article is unaffected, and still works fine. Using this method, dnsmasq is actually transparently handling the Mikrotik using BOOTP which is not broken, and is Mikrotik’s default netboot mode. I didn’t even realize this when first writing the article, because at the time it didn’t matter. However, I just flashed a new hEX yesterday using these instructions, and it was on RouterOS 6.47.9.

Lots of routers now offer a nice little web interface you can use to upload firmware. However, there are still a lot of routers that are easiest to flash using netboot methods like tftp. There are plenty of tutorials on doing this, but most focus on using a server installed on your computer. If this is a second router and you already have a working OpenWRT main router, it’s often actually much easier to just use your main router to TFTP boot, which is something dnsmasq (the default DHCP and DNS server) can do out of the box.

In my case, I already have a primary router with external USB storage up and running. This brief tutorial gives the bare bones steps on what you need to do to use this to flash a second router that supports netboot. I’ll be flashing a Mikrotik hEX RB750Gr3 in this example, since I had one I needed to do anyway. If you don’t already have some external storage set up on your main router, take care of that first – the existing tutorials for that are pretty good, so I won’t duplicate that here.

First, boot up your new router at least once and get its MAC address. For some reason things will go more smoothly if you assign it a static IP when it first boots up as a DHCP client.

Configure /etc/config/dhcp (which controls dnsmasq) on your main router. First, turn on the tftp server, and point it to your USB storage:

config dnsmasq
     ...
     option enable_tftp '1'
     option tftp_root '/mnt/stor/tftp'

Make sure that second line you added points to the correct folder on your USB storage.

Add a static IP for the box you’ll flash:

config host
      option mac 'B8:27:EB:2B:08:85'
      option name 'somehost'
      option ip '192.168.1.240'

Change that MAC to your new router, and give it whatever name and address on its WAN you can remember. You won’t actually need it once it boots up, and you can delete this section once your new router is flashed.

Now, drop the file in the appropriate folder. For TFTP booted routers, you usually need two firmware images: one it can netboot off of from TFTP (which usually has “factory” in the name), and the real copy that gets flashed to the flash memory (usually has “upgrade”). This is a two step process – the netbooted image will not actually be saved to the router, and this is actually a great way to test an OpenWRT build before you flash. You then use the netbooted “factory” image to flash the router using the permanent “upgrade” image. If you don’t do that second step, when you reboot the router, it’ll go straight back to its original OS and settings from memory.

Now, the critical part – take that netboot image in your folder (mine is “openwrt-RB750gr3-GO2021-02-03-factory.bin” for the OpenWRT ROOter fork), and rename it “vmlinux”.

Some router manufacturers also need to find your TFTP server at a specific address, as well. Mikrotik apparently expects 192.168.1.10. If your LAN is already at 192.168.1.0 and the .10 address is free, it is trivial to add .10 as a second address for your main router (this will not affect anything else of significance). From your main router’s command line, simply run:

ip addr add 192.168.1.10 dev eth0.5

Change the bit after “dev” to match whichever interface or port connects to your LAN. In my case, my LAN is on a VLAN port of my router, hence eth0.5.

Now, it’s time to netboot. Shut down your new router if it isn’t already, and plug is WAN port into your existing network.

For the Mikrotik hEX, to trigger a netboot, you plug in the power jack with the reset button already held down. The button does several things depending on how long you hold it down; it comes on lit steady, then starts flashing, then goes off completely. Once it’s off completely you can release the button, as it will be looking for a netboot. If you’re watching your log on your main DHCP router, it’ll post up a log line when it sends the boot image to a DHCP client.

Give it time to boot up, and then try connecting from a client plugged into the LAN side of the new router. One advantage of doing it this way is that you don’t tie up your main computer as both the boot tftp server and the tool you need to log into the new router with. If your OpenWRT image netbooted successfully, you should find your new router at 192.168.1.1 from your test computer.

Now, for the last important part – flash the permanent image! You need to go to System -> Backup / Flash Firmware on the new router and flash that upgrade image, or what you’ve just done won’t stay permanent.

Administrative, Computers

ROOter GoldenOrb Hosting

Leave a comment

We’re helping provide overflow hosting space for the wonderful team that keeps this OpenWRT fork going! However, during this morning’s transition, I hear a few people are having cache problems that have redirected them here to the blog front page, instead of the upload and build folders.

If that’s you, here are your direct links to the new folder locations:

http://www.aturnofthenut.com/upload/

http://www.aturnofthenut.com/builds/

Hopefully the redirect issues will clear up quickly. However, if you ever land on my front page accidentally, there will also always be a link at the top of the page with direct links.

Thanks for your patience!

Computers

Remote Logging from OpenWRT to Rsyslog

Leave a comment

This one is brief and simple. I have six routers going right now (and a ridiculously long article still in draft explaining why), all running OpenWRT. I had them set to save logs to local thumb drives, which, frankly, was a pain in the butt. I concluded that I wanted them all logging to a single remote system for simplicity – the old EEE PC netbook that I use as a network terminal for basic maintenance. It has a good old fashioned spinning disk hard drive, and won’t suffer from a ton of log writes like the thumb drives (or heavens forbid the internal storage) on the routers would.

After going through several tutorials that were either a bit complicated or a bit incomplete for my specific use, it turned out to be obnoxiously simple to implement. I could’ve gotten it all done in under half an hour if I’d already known exactly what I was doing, and most of that time was repetitively ssh-ing into six different routers.

That said, here it is: quick, dirty, with no missing or extra steps!

Set up your log server first

My logserver is running Debian Buster, which already came with rsyslog configured with very sensible basic settings (logging to file in /var/log/, and rotation already set up). All I had to do was enable listening on TCP or UDP 514 (I’ve opened both but am using TCP), then set up some very basic filtering to sort the remote messages the way I wanted.

All changes can be accomplished quickly in /etc/rsyslog.conf. Starting at the top, we uncomment the lines that start the server listening:

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

# List of sub networks authorized to connect :
$AllowedSender UDP, 127.0.0.1, 192.168.0.0/16
$AllowedSender TCP, 127.0.0.1, 192.168.0.0/16

The last group there was added based on the recommendations of a tutorial, and restricts senders to localhost and my local network (I have hosts on five subnets, most people could be using 192.168.1.0/24 or whichever single subnet they’ve configured).

Next, near the bottom of the file, you need to decide how you want your messages stored. If you don’t change anything, they’ll be mixed into your other logs from your localhost. You can do a lot of more complicated things, but I wanted one subdirectory per remote host, with all messages in a single syslog.log. Here’s how you get that, in the rules section and above your rules for normal localhost messages:

###############
#### RULES ####
###############

#
# ADDED BY CHUCK
# Redirect all messages received from the network to subfolders first
# From example on stackexchange saved in notes.
#

$template uzzellnet,"/var/log/%HOSTNAME%/syslog.log"

if $fromhost-ip startswith "192.168." then -?uzzellnet
& stop

The template can be named anything. This test checks all log messages to see if they are from remote hosts in my local net – if so, it sends them all to a single file based on the remote hostname. The template statement must be before the test, and “& stop” tells it that any logs meeting this test should not be further processed below with localhost messages.

Obviously your log server will need a static IP to do this job. If you haven’t set one already, you can either set it manually from the server, or (my recommendation) just configure your DHCP router to automatically provision that machine with a static IP.

That’s it for configuring the server! It really is that simple. Just restart rsyslog on your server:

chuck@raul:/etc$ sudo systemctl restart rsyslog

Now, set up each remote OpenWRT host

All settings for logging are stored in /etc/config/system. By default, everything is logged to a ring buffer in memory, and lost on reboot. Not useful if something happens that causes a lockup, etc., but it is awfully handy to read from the command line when you’re already logged in via ssh, so we want to keep that functionality – messages should both be stored in the ring buffer and sent to the remote server.

In /etc/config/system, add or change the following three lines (using the static IP address you’ve provisioned for your log server):

        option log_ip '192.168.1.209'
        option log_port '514'
        option log_proto 'tcp'

You can leave it the default UDP if you prefer – there’s less network overhead, but most of us aren’t really hurting for network capacity. TCP is generally worth it for logging unless you really don’t care if you miss the occasional message.

Now, just restart your logs so the new settings are picked up:

/etc/init.d/log restart
/etc/init.d/system restart

Next, log a test message. It can say anything. This was the one from the last of my six routers to configure, a test machine I’m still setting up to replace one of my production routers soon:

root@FASTer2:~# logger "First test message from Faster2!"

That should produce a log line both locally and remotely. Check the ring buffer:

root@FASTer2:~# logread
Thu Dec 17 20:22:07 2020 daemon.info logread[424]: Logread connected to 192.168.1.209:514
Thu Dec 17 20:22:21 2020 user.notice root: First test message from Faster2!

Now, on your log server, you should see a new directory for your host created in your log folder (probably /var/log/ if you’re using Debian defaults). We said in rsyslog.conf earlier that the file should be in that subfolder and named syslog.log, so let’s test receipt:

chuck@raul:~$ sudo cat /var/log/FASTer2/syslog.log
[sudo] password for chuck:
Dec 17 20:22:07 FASTer2 logread[424]: Logread connected to 192.168.1.209:514
Dec 17 20:22:21 FASTer2 root: First test message from Faster2!

That’s it! We’re all set to go. You can obviously get way more elaborate than this, but a simple 1:1 replacement of OpenWRT’s default ring buffer with a neatly sorted single log file will probably cover most users’ needs.

Enjoy!