Updating Arch Linux — filesystem drama

Today I updated Arch Linux on a VM that I haven’t used in a while. When I first did this I received a mes­sage like:

:: Pro­ceed with instal­la­tion? [Y/n] y
(73/73) check­ing keys in keyring                     [######################] 100%
(73/73) check­ing pack­age integrity                   [######################] 100%
(73/73) load­ing pack­age files                        [######################] 100%
(73/73) check­ing for file con­flicts                  [######################] 100%
error: failed to com­mit trans­ac­tion (con­flict­ing files)
filesys­tem: /bin exists in filesys­tem
filesys­tem: /sbin exists in filesys­tem
filesys­tem: /usr/sbin exists in filesys­tem
Errors occurred, no pack­ages were upgraded.

Puz­zling… Every­one that’s writ­ten a script knows to call #!/bin/bash on the first line. How could /bin exist­ing be a problem?

I googled a bit and came across this won­der­ful post to get a list of pack­ages that owns files in /bin, /sbin, and /usr/sbin: https://bbs.archlinux.org/viewtopic.php?pid=1280576#p1280576

So I ran the command:

grep ‘^s\?bin/\|usr/sbin’ /var/lib/pacman/local/*/files | cut –d “:” –f 1 | uniq | cut –d “/” –f 6

The result was a list­ing of 20 some pack­ages. I then upgraded each them, one at a time, with pac­man –S , leav­ing filesys­tem to the end, along with bash and glibc.

So far so good. I updated bash. No prob­lems. Then I updated glibc…

/usr/bin/locale-gen: /bin/sh: bad interpreter

That isn’t promis­ing. So I tried to run /bin/sh, and it wasn’t there. Not only was /bin/sh gone, but the entire /bin direc­tory was gone. I poked around and found bash, sh, and the other usual sus­pects in /usr/bin. I cre­ated a sys­link for /bin -> /usr/bin with: ln –s /usr/bin /bin

Then I ran pac­man –S glibc and it worked. Next up filesys­tem. Same error as before, except this time it was just on /bin. I removed the sym­link, updated filesys­tem, and every­thing was in har­mony once again. The filesys­tem pack­age will actu­ally cre­ate /bin -> /usr/bin and /sbin -> /usr/bin so that all of your scripts will con­tinue to func­tion when you call #!/bin/bash at the top.

True to form, I didn’t come across this post by Arch Linux until after I’d writ­ten this blog entry. Take a look for another method of solv­ing the problem.

Binding Media Keys in i3 on Arch Linux to Control MPD

My Log­itech key­board has sev­eral but­tons at the top that I wanted to use to con­trol mpd (man|arch). They were Prev, Play/Pause, Next, and Mute/Unmute.

I knew that my win­dow man­ager i3 (man|arch) was able to bind key com­bi­na­tions to per­form func­tions such as exe­cut­ing a pro­gram, switch­ing the vis­i­ble work­space, or mov­ing a win­dow. The for­mat of the .i3/config file to launch a pro­gram on a cer­tain key com­bi­na­tion is:

bindsym <key com­bi­na­tion> exec <com­mand to execute>

With that known, we need to know how the sys­tem refers to those media keys. This is where xev (man|arch) is use­ful. Xev is a pro­gram that will print out data about mouse move­ments, key press events, and key release events. When the key is unbound you’ll get a Key­Press event like the below:

Key­Press event, ser­ial 32, syn­thetic NO, win­dow 0x4800001,
    root 0x28e, subw 0x0, time 654012911, (-169,-224), root:(1755,327),
    state 0x10, key­code 20 (keysym 0x2d, minus), same_screen YES,
    XLookup­String gives 1 bytes: (2d) “-“
    XmbLookup­String gives 1 bytes: (2d) “-“
    XFil­terEvent returns: False

Notice the sec­tion on the 2nd line, “(keysym 0x2d, minus)”, in this exam­ple, I had pressed the “-” key and found that the sys­tem refers to it as “minus”. I did this for each of the media keys and found their names to be:
Prev: XF86AudioPrev
Next: XF86AudioNext
Play/Pause: XF86AudioPlay
Mute/Unmute: XF86AudioMute

We can now fill in the key com­bi­na­tion sec­tion of the .i3/config:

bindsym XF86AudioPrev exec <com­mand to exe­cute>
bindsym XF86AudioNext exec <com­mand to exe­cute>
bindsym XF86AudioPlay exec <com­mand to exe­cute>
bindsym XF86AudioMute exec <com­mand to execute>

By read­ing the man page of mpc(man|arch), we know that it can per­form the fol­low­ing func­tions given the cor­re­spond­ing com­mand:
Prev: mpc prev
Next: mpc next
Play: mpc play
Pause: mpc pause

This makes two of the key bind­ings straight for­ward, but leaves us need­ing a tog­gle for Play/Pause, as we have one but­ton that must send one of two com­mands depend­ing on the cur­rent state of mpd. Here’s a bash script that will per­form just such a tog­gle: https://github.com/rarmknecht/utilities/blob/master/mpc_toggle

With the mpc_toggle script copied to /usr/local/bin our .i3/config now looks like:

bindsym XF86AudioPrev exec mpc prev
bindsym XF86AudioNext exec mpc next
bindsym XF86AudioPlay exec mpc_toggle
bindsym XF86AudioMute exec <com­mand to execute>

Next up, Mute/Unmute. Because I’m using the ALSA sound dri­vers I can use the tool amixer (man|arch) to con­trol the Mas­ter out­put. Again, here’s a bash script that allows for a tog­gle by read­ing the state of Mas­ter before set­ting it: https://github.com/rarmknecht/utilities/blob/master/my_mute

And that’s it once we drop the script in /usr/bin/local and update the i3 con­fig­u­ra­tion file. We can now play/pause, mute/unmute, go back a song, or skip to the next from the keyboard.

Final .i3/config snippet:

bindsym XF86AudioPrev exec mpc prev
bindsym XF86AudioNext exec mpc next
bindsym XF86AudioPlay exec mpc_toggle
bindsym XF86AudioMute exec my_mute

Rebuilt the Workstation

Today I decided to rebuild my per­sonal work­sta­tion again. I’d had Win­dows 7 installed since Novem­ber 2011 as I had started play­ing games like Skyrim, Dia­blo 3, and Guild Wars 2. I’ve real­ized that I haven’t really played a game in many months and have no spe­cific need for the Win­dows OS.

I’ve went with my old friend Arch and tried out a new win­dow man­ager (i3).

randr is now fully sup­ported within the nvidia binary dri­vers, so i3 had no trou­ble rec­og­niz­ing the mul­ti­ple mon­i­tors. Both .i3status.conf and .i3/config were straight-forward to edit to my liking.

So far so good!

Bluetooth Keyboard on Andoird

I kind soul at my client brought in a Log­itech Tablet Key­board for Android today for me try out.  The touch of the keys is as respon­sive and com­fort­able as a stan­dard key­board at my desk­top and works quite well for long form tasks.  Set­ting it up is pretty sim­ple and the included case dou­bles as an effec­tive stand for the tablet.  I’m writ­ing this on the key­board through the Word­Press appli­ca­tion on the house­hold Galaxy Nexus 10.

I can imag­ine that if I were to work mainly on emails and other doc­u­ments, that the key­board and tablet make a rather portable and func­tional lap­top replace­ment. In fact, if I were still in a role that required long hours at the CLI in linux for devel­op­ing Python code or exe­cut­ing a vari­ety of secu­rity tools, the com­bi­na­tion of JuiceSSH and the tablet+keyboard make for a rather nice setup. I’m sit­ting with the tablet on the din­ning room table, and the key­board is com­fort­able to type at. This would also make trav­el­ing light quite a bit eas­ier were it paired up with a blue­tooth mouse and Cit­rix access to crit­i­cal applications.

You can read a full review of this device over at Android Guys.

5 Days of SSHD Stats on a Public IP

Over the last 5 days my pub­licly acces­si­ble sys­tem at has had 5,092 attempted SSH logins from 8 IP Addresses.

Unsur­pris­ingly, the most com­monly attempted ssh user­name is “root”, the default admin­is­tra­tive account on Linux sys­tems.

What I did find sur­pris­ing, was that most of the IPs gave up after a rel­a­tively low num­ber of attempts.

Even more so was that for each user­name tried which wasn’t root, the num­ber of pass­words attempted was rarely more than a dozen.

Here are the IPs observed attempt­ing the unau­tho­rized logins:


All data was col­lected via syslog-ng from an Arch Linux server hosted by Rack­space sent to Splunk Storm

Tracking SSHD Login Activity in SplunkStorm

After a night out with my wife I decided to cre­ate a search for SSHD logins in Splunk Storm. See­ing that port 22 is open to the world on, I won­der how long before ran­dom bots start attempt­ing to log into it. As a secu­rity prac­ti­tioner its always inter­est­ing to see the effects of just being on the inter­net; and SplunkStorm is a great way to mon­i­tor those effects.

SSHD — Logins (Accepts and Failures)

sshd | rex field=_raw ”]: (?<sshd_action>.*) pass­word for (?<sshd_username>.*) from (?<sshd_ip>.*) port” | search sshd_action=”*” | table _time host sshd_action sshd_username sshd_ip

Here’s a look at the result:

If you’re run­ning syslog-ng, it’s super sim­ple to send your logs over to a Splunk instance. Below is what I added to my /etc/syslog-ng/syslog-ng.conf file where the X’s are val­ues pro­vided by your SplunkStorm admin console:

des­ti­na­tion d_splunk { tcp(logsX.splunkstorm.com” port(XXXXX)); };
log { source(src); destination(d_splunk); };

Splunk Query: Processes Created & Terminated

This morn­ing I installed and con­fig­ured Snare on my AWS EC2 Win­dows 2012 Server to ship logs to my SplunkStorm project.

Below is a basic query for chart­ing the top processes that have started on the sys­tem:
Even­tID 4688 — A new process has been cre­ated. (Details)

4688  | rex field=_raw “Process Name: (?<Process_Name>.*) Token Ele­va­tion Type:”  | sort Process_Name | top limit=10000 Process_Name

Here is a mod­i­fied ver­sion of the query for processes that have ter­mi­nated:
Even­tID 4689 — A process has exited. (Details)

4689  | rex field=_raw “Process Name: (?<Process_Name>.*) Exit Sta­tus:”  | sort Process_Name | top limit=10000 Process_Name

More to come.…