Type your search keyword, and press enter

Maplins weather station fun

Your ads will be inserted here by

Easy Plugin for AdSense.

Please go to the plugin admin page to
Paste your ad code OR
Suppress this ad slot.

weather station control unit weather station sensor pole
A while ago I bought one of the Maplins weather stations, just for fun really no plans with it. The device came with a pole to which several sensors could be attached: temperature, humitidy, wind speed, wind direction and rain level. The other important bit in the box was the control station, which is a large LCD screen based output for the information which is gathered in the sensors, it also gives an attempt at forecasting. Areas of the screen are pressure sensitive which allows for a touchscreen style interface for changing views etc.

All pretty cool, and looks great on the worktop displaying weather info. But the geek inside of me wants more!!! Along with the aforementioned components comes a USB cable and a cdrom, the brains also does data collection which can be extracted using the software on the CD.
weather station in situe
Thats also cool, but unfortunately no software for Linux included on the disk, not to worry pywws  from Jim Easterbrook steps in here. I won’t go into detail on how I set this up as there is a good tutorial here: http://www.weather.dragontail.co.uk/index.php?page=station_setup but what it gives you is a way to extract the weather data from the brains of the weather station store it on a pc then display it through a pretty web interface. Gives allsorts of graphs and fancy data tables.

This was quite cool, but got me thinking about what to do with this data, can it be uploaded to some weather data aggregation service for the greater good? Thats where I found weather underground, or wunderground, and their API. Since finding it I’ve been using it to both upload my data and also retrieve data for various projects (nagios monitoring, temperature monitoring). Pywws contains the ability to post the data to wunderground built in. Great, I have a nice solution the head unit connected via usb to a server in my garage. I can’t see the unit, but I can see the data via Pywws.

It was all going well, that is until the batteries in the head unit run out and the project headed back onto the todo pile. That was until I was working on a seperate project where I was trying to retrieve the RF signal on 433mHz home plugs and saw some random traffic. After thinking it must be a neighbour or a cars remote locking, i realised the data was being sent on a regular basis. It eventually dawned on me after a bit of googling that the data may be from the sensors on the weather station as the batteries were still active on that side. A small amount of googling later and I found that I was able to not only retrieve that data but also decode it into readable data.

To explain, I am using an RTL DVB dongle which is popular in Software Defined radio circles as its frequency can be set to pretty much anything. I haven’t delved into the SDR side of things, but I’ve been using it to scan 433 and 868 mHz frequencies to try retrieve data on RF control devices within my home. Using the dongle and the RTL-SDR software http://sdr.osmocom.org/trac/wiki/rtl-sdr I can achieve this.

My aforementioned googling around the weather station data led me back to a project I was already using, OOK-Decoder, OOK being the form of modulation used in many RF devices and seemingly the weather station. The OOK-Decoder project also comes with another executable wh1080, which it turns out is my weather station.

The project was written by a chap who has a remote weather station, so he was sending the data over a multicast network, recieving it locally and then decoding it. This means its a bit overkill for what i’m achieving but I’m achieving it. Essentially, there is a machine which sits within the vicinity of the weather station sensors with the DVB dongle attached and running OOKD which takes tunes to the 433mHz frequency by default, and streams the raw data over the network. On the receiving (network) end, a PC runs OOKDUMP to output the stream of OOK data raw or runs the wh1080 binary which takes the stream of OOK data and output a json formatted text file.

{
“temperature”:6.9,
“humidity”:85.0,
“avgWindSpeed”:0.0,
“gustSpeed”:0.0,
“rain”:0.0,
“batteryLow”:0,
“windDirection”:450
}

Why it’s overkill for me is that I run both ookd and wh1080 on the same box, I’ll probably figure out combining the two at some point to reduce the load but for now it works.

The next step is to do something useful with that data, so I wrote a fairly rough python script to upload the data to Wunderground. I’ve also made the wh1080 output the file to a directory which is accessible via a webserver to make it useful to any scripts I write for other projects. Having the data stream directly to the server now also means I can relocate the previously USB tied control station back into the house, double win!

Your ads will be inserted here by

Easy Plugin for AdSense.

Please go to the plugin admin page to
Paste your ad code OR
Suppress this ad slot.

So in a nutshell the process to get weather data directly from a Maplin weather station:

  • Install rtl-sdr as per: http://sdr.osmocom.org/trac/wiki/rtl-sdr (dont forget to blacklist the dvb driver)
  • install ook-decode as per: https://github.com/jimstudt/ook-decoder
  • Run ookd and wh1080 (don’t forget to run with & at the end to allow access back to the console), this outputs to /tmp/current-weather.json by default
  • Create a weatherunderground account http://www.wunderground.com/ and create a personal weather station ID
  • Use my python script with the ID generated above and password to upload the data

Links:

Maplin weather station

http://www.maplin.co.uk/p/black-usb-wireless-touchscreen-weather-centre-n96gy

Jim Easterbrooks PyWws

https://github.com/jim-easterbrook/pywws

Jim Studt ook-decoder (and wh1080)

http://www.wunderground.com/

My script to publish json data to wunderground

https://github.com/jfarcher/wunderpost

Ensure systemd services restart on failure

I wrote a post a while ago covering the use of Monit to monitor services running and the use case I covered was to ensure these services restarted on failure. While a useful feature of Monit, it seems to be now a little redundant with SystemD having a built in restart feature.

Same use case where MySQL (or MariaDB in this case) is being killed by Apache’s oom killer.

I first copied the original systemd file associated with Mariadb from /usr/lib/systemd/system/mariadb.service to /etc/systemd/system/mariadb.service

Then under the [Service] section in the file i added the following 2 lines:

Restart=always
RestartSec=3

After saving the file we need to reload the daemon configurations to ensure systemd is aware of the new file

systemctl daemon-reload

Then restart the service to enable the changes.

systemctl restart mariadb

You can test this configuration by firing a kill to the process, e.g.:

 ps -ef|grep maria
mysql    22701 22542  0 06:52 ?        00:00:01 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
kill 22701
watch "ps -ef|grep maria"

You should see the process restart.

Hardening SSH with OTP for 2 factor authentication

Something I’ve been meaning to do for a while is look into the possibility of using 2 factor authentication, or 2FA, with SSH connections. This would add a much needed level of security to servers I host out in the wild.

Here’s how I did it:

The Google Authenticator mobile app used to be an open source project, it isn’t any more but the project has been kindly forked and looked after by Red Hat under the guise of the FreeOTP project. The first step is to download the app, which is available for Android and iOS there is even a Pebble project in the works. https://fedorahosted.org/freeotp/

Google Play: https://play.google.com/store/apps/details?id=org.fedorahosted.freeotp

iTunes: https://itunes.apple.com/us/app/freeotp/id872559395

Next we need to configure PAM, this is the component in Linux which ties authentication together. It allows us to add modules for various authentication sources into various applications which require authentication, in this case we need a module compatible with FreeOTP to provide authentication to SSH.

We’ll be using the pam_oath for this, the OATH toolkit is designed for building one-time password based systems. http://www.nongnu.org/oath-toolkit/

yum install pam_oath oathtool gen-oath-safe

This gives us the tools needed to link in to pam, and also generate the initial keys to share between the devices.

Next we need to edit /etc/pam.d/ssh to recognise this module by adding the following line to the top:

auth sufficient pam_oath.so usersfile=/etc/liboath/users.oath window=10 digits=6

Notice we specify a users file, this is where the users who harness OTP will have their details stored. Once this is saved we need to restart sshd

service sshd restart

or

systemctl restart sshd

So thats ssh configured, from now on when a user logs into the system via ssh, they will be prompted for a One-time password.

Next up we need to generate the keys which are to be shared between the target host (SSH) and the client generating the OTP (Android or iOS app)

gen-oath-safe jon hotp

Replacing jon with your username. hotp denotes the type of key to be generated, hotp being a counter based key and totp being a time based – choice is yours here.

This command will generate a number of codes, HEX, b32, QR and yubikey. The keys we are interested in are the HEX and the QR:

gen-oath-safe

From the app select the QR scanning option on the top tool bar:

FreeOTP

Scan the generated QR code which will then store the key in the application.

The final step is to add the HEX code to the file we referenced earlier in the sshd pam config file. Drop the following line /etc/liboath/users.oath (making sure you use your generated key and username):

HOTP jon -  da50cc2e1ee6726c847c5b960a62751e9bbea3a9

Once that file is saved we can go ahead and login via ssh with the specified user. You will now be prompted for a One-time password which can be generated by pressing on the entry within FreeOTP.

Note: if ssh keys are setup then these will be preferred over OTP, i’m sure a modification of the pam config would allow for both but haven’t spent any more time on this yet.

 

 

 

Etckeeper – config version control

A valuable tool I have been using for many years is etckeeper, it works by essentially turning your /etc directory into a git repository.

This is a fantasticly useful set of tools as any configuration changes can be logged and also reverted quite easily. Install and setup is exeptionally easy too!

Packages are available for most distributions, but my scenario (Fedora,CentOS,RHEL) was:

yum install etckeeper

Once the package was installed an initialisation must be performed:

etckeeper init

This essentially runs a “git init” in the /etc directory setting up the directory ready.

That’s all there is to the installation.

Using it is a matter of committing changes when they are made, my workflow generally consists of running a check to see if all previous changes were committed, make the change, commit the change.

etckeeper unclean

Will check the /etc directory for uncommitted changes, if they exist they can be committed in the same way as any new changes:

etckeeper commit

Running this command will present the familiar commit log screen in your favourite editor as it is essentially running a git commit from within the etc directory. Once the commit log is saved any changes are then stored within the version control system. A cron job is also in place to ensure a daily commit takes place, incase commits have been missed.

Now this is cool and extremely useful, but extending the git elements to push to a remote repository gives your etc that extra bit of resilience. Hook scripts are already present within /etc/etckeeper/commit.d/99push to recognise if a remote repository is configured and push to it on commit. Adding a remote repository is fairly simple, in my case I push to a gitlab (think self hosted github) server which I run.

First up a repository needs to be created in which to push to, I won’t go into detail here as there are hundreds if not thousands of Git tutorials out there. Gitlab has a repository created for each server and the ssh public key of each server stored to enable access.

cd /etc
git remote add origin git@gitlab01:etckeeper/server01.git
git push -u origin master

Will set the remote repository and populate it.

The last element to configure is the etckeeper config file, changing

PUSH_REMOTE=""

to

PUSH_REMOTE="origin"

(or whatever branch you choose to use)
And thats it! You an amazingly simple piece of software which could potentially save your Apache server, your Dovecot server or maybe even your job!

 

SSH known hosts verification failure one liner

WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!

Those who regularly build and rebuild machines or virtual machines on a dhcp network will probably be faced with this quite often, this is due to the known fingerprint for the previous host being different to a new one which has aquired the same IP address.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
c5:ab:00:3c:88:7e:18:8f:46:49:1d:af:f1:8b:4e:98.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /root/.ssh/known_hosts:66
ECDSA host key for 192.168.1.165 has changed and you have requested strict checking.
Host key verification failed.

There is an option to have SSH ignore these when connecting, however i find that cleaning out the old line before connecting far quicker and i do this with a Sed one liner.

The line in the known_hosts file we are interested in can be found at the end of the line:

Offending ECDSA key in /root/.ssh/known_hosts:66

66 in this case, so we can get sed to simply delete that line using:

sed -i '66d' ~/.ssh/known_hosts

An SSH session can now be opened without Host key verification failure.

Hope this helps someone.

Getting Fedora 21 on the Raspberry Pi 2

The recent release of the Raspberry Pi 2 uses a newer version of the ARM architecture spec, the ARM Cortex-A7 uses ARMv7 whereas the previous model ARM11 uses ARMv6. The great thing about this is the majority of Linux distros already provide an Image for this architecture. More importantly, Fedora already have images.

There is a slight caveat to the above statement however, that being they won’t just work with the Pi 2. The process isn’t that difficult either just a few steps:

  1. Download the image you require, for this we’ll go with the Fedora 21 minimal – http://dl.fedoraproject.org/pub/fedora/linux/releases/21/Images/armhfp/Fedora-Minimal-armhfp-21-5-sda.raw.xz
  2. Flash the image to an SD card xzcat Fedora-Minimal-armhfp-21-5-sda.raw.xz |dd of=/dev/mmcblk0 bs=1M
  3. Make sure the card is unmounted
  4. fdisk the card:
    1. remove partition 1
    2. add a new partition where the old partition 1 was, with type B (FAT32)
    3. write and exit
  5. mkfs.vfat /dev/mmcblk0p1
  6. Clone the Pi firmware repository – git clone https://github.com/raspberrypi/firmware.git
  7. Mount the card again
    1. mkdir /mnt/sdcard
    2. mount /dev/mmcblk0p3 /mnt/sdcard
    3. mount /dev/mmcblk0p1 /mnt/sdcard/boot
  8. Copy the contents of the boot directory from the repository you just cloned to the new boot directory and the kernel modules to the lib/modules directory on the main root partition
    1. cp -r firmware/boot/* /mnt/sdcard/boot/
    2. cp -r firmware/modules/3.18.7-v7+/* /mnt/sdcard/lib/modules/
  9. Edit the fstab file to reflect the new UUID of the partition and change from being an ext to a vfat type
    1. blkid /dev/mmcblk0p1 – this will give the UUID of the partition
    2. vi /mnt/sdcard/etc/fstab and edit the line which contains /boot to contain the above info
  10. Create a /mnt/sdcard/boot/cmdline.txt file containing the following:

    dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p3 rootfstype=ext4 elevator=deadline rootwait

  11. Create a /mnt/sdcard/boot/config.txt file containing the following:
    #uncomment to overclock the arm. 700 MHz is the default.
    arm_freq=700# NOOBS Auto-generated Settings:
    hdmi_force_hotplug=1
    config_hdmi_boost=4
    overscan_left=24
    overscan_right=24
    overscan_top=16
    overscan_bottom=16
    disable_overscan=0
    core_freq=250
    sdram_freq=400
    over_voltage=0
  12. save and close any open files on the sd card then unmount and ensure all writes are complete
    1. umount /mnt/sdcard/boot
    2. umount /mnt/sdcard
    3. sync
  13. You should now be able to remove the SD card from your PC and boot it in your new shiny Raspberry Pi 2

I’m sure it won’t be long before dedicated images are available, but for now this seems to work for me. I haven’t tried any more than the minimal install, with these your mileage may vary.

Note: Please remember this will only work on the newer Raspberry Pi 2.

Update:

Extra steps suggested by Tim Bosse

14. Install rpi-update.
Install binutils and tar.

Download and setup rpi-update.

# curl -L --output /usr/bin/rpi-update https://raw.githubusercontent.com/Hexxeh/rpi-update/master/rpi-update && sudo chmod +x /usr/bin/rpi-update

15. Run rpi-update as root.

I find this is important to run any time you get kernel updates from Fedora repos.

I have a wireless USB dongle that I use.

16. Install NetworkManager-wifi and NetworkManager-tui (because I find nmcli not so much fun).

I’ve created an image based on steps 1-13 it’s fairly rough and ready so YMMV

https://jfarcher.fedorapeople.org/Fedora-Minimal-armhfp-rpi-21-5-sda.raw.xz

Monit – monitor your processes and services simply

Monit is an application I’ve been meaning to setup for a while, I was first made aware of it from a chap I had the pleasure of talking to at OggCamp this year, he seemed to use it to the n’th degree to monitor files and services within docker containers to ensure a development environment was as it should be. This was far more than I really needed, but the monitoring of services definitely caught my attention so I set about installing and configuring. I was pleasantly surprised with the result, and how simple the whole process was.

Scenario: small hosting server with low spec, occasionally gets hit with a large amount of traffic resulting in either apache or mysql dying.

Configuration: In this instance a CentOS 6 server with standard LAMP stack, but i’m sure this will work with other distributions such as Fedora or CentOS 7 just replacing the relevant commands for systemd based commands.

How?

First off lets install monit, this comes from the rpmforge (http://repoforge.org/) repositories so if you haven’t already got them installed do so

yum localinstall http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm

It would be worth checking the website to ensure that rpm version is correct (http://repoforge.org/use/)

Once thats installed we can install the monit software

yum install monit

lets enable the service to start on boot, and also start it to ensure it works OK before configuring:

chkconfig monit on

service monit start

Note: if using a systemd based distro such as Fedora or CentOS 7 then systemctl commands will need to be used instead of the above (systemctl enable monit and systemctl start monit)

If all is good then we can now tailor the configuration to our needs, monit uses the common approach for config files by having a master config at /etc/monit.conf which also reads in files from /etc/monit.d/. The only directive I changed in the master config file was to uncomment the following line:

set logfile syslog facility log_daemon

Which turns on logging, whether this is needed further down the line is to be decided but for now its great to have during configuration.

Next we can create some config files in /etc/monit.d/ for our services (apache httpd and mysql in this case)

vi /etc/monit.d/mysqld.conf

check process mysqld with pidfile /var/run/mysqld/mysqld.pid
start program = "/sbin/service mysqld start"
stop program = "/sbin/service mysqld stop"
if failed host 127.0.0.1 port 3306 then restart
if 5 restarts within 5 cycles then timeout

vi /etc/monit.d/httpd.conf


check process httpd with pidfile /var/run/httpd/httpd.pid
start program = "/sbin/service httpd start"
stop program = "/sbin/service httpd stop"
if failed host 127.0.0.1 port 80 then restart
if 5 restarts within 5 cycles then timeout

These two config files will check the pid files for activity outside monit, namely if the process stops without monit stopping it, and take action based on the status. The also monitor the respective tcp ports for the particular applications, 3306 for mysqld and 80 for apache.

Note: these configurations should also work with Debian based distributions but check the location of the pid files, also the service names are slightly different (mysql and apache2 if memory serves correctly).

Lets restart Monit and run some tests, for this I will run a tail on the log file while stopping services and killing processes:

tailf /var/log/messages

service monit restart

[root@web1 monit.d]# service monit restart
Stopping monit: Dec 31 12:20:56 web1 monit[5338]: Shutting down monit HTTP server
Dec 31 12:20:56 web1 monit[5338]: monit HTTP server stopped
Dec 31 12:20:56 web1 monit[5338]: monit daemon with pid [5338] killed
Dec 31 12:20:56 web1 monit[5338]: 'web1' Monit stopped
[ OK ]
Starting monit: Starting monit daemon with http interface at [localhost:2812]
[ OK ]
Dec 31 12:20:57 web1 monit[6232]: Starting monit daemon with http interface at [localhost:2812]
[root@web1 monit.d]# Dec 31 12:20:57 web1 monit[6236]: Starting monit HTTP server at [localhost:2812]
Dec 31 12:20:57 web1 monit[6236]: monit HTTP server started
Dec 31 12:20:57 web1 monit[6236]: 'web1' Monit started

Lets stop mysqld

service mysqld stop

[root@web1 monit.d]# service mysqld stop
Stopping mysqld: [ OK ]

[root@web1 monit.d]# service mysqld stop
Stopping mysqld: [ OK ]

Within seconds an entry in the log file is presented:

Dec 31 12:22:57 web1 monit[6236]: 'mysqld' process is not running
Dec 31 12:22:57 web1 monit[6236]: 'mysqld' trying to restart
Dec 31 12:22:57 web1 monit[6236]: 'mysqld' start: /sbin/service

[root@web1 monit.d]# service mysqld status
mysqld (pid 6526) is running...

OK so that worked nicely, lets try something a little less clean

[root@web1 monit.d]# ps -ef|grep mysqld
root 6679 1 0 12:23 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql 6867 6679 1 12:23 ? 00:00:00 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock

[root@web1 monit.d]# kill 6867
[root@web1 monit.d]# service mysqld status
mysqld dead but subsys locked

And as if by magic:

Dec 31 12:25:59 web1 monit[6236]: 'mysqld' process is not running
Dec 31 12:25:59 web1 monit[6236]: 'mysqld' trying to restart
Dec 31 12:25:59 web1 monit[6236]: 'mysqld' start: /sbin/service

Brilliant, it seemed to perform exactly as expected. I wont bore you with the detail, but Apache restarted just the same.
And that is it, a really easy to configure monitoring solution. Here, however, I was just scratching the surface of the monitoring capabilities. Take a look at the Monit website and wiki for more details on the vast array of configurables. http://mmonit.com/monit/documentation/ http://mmonit.com/monit/

Barcamp Manchester

IMG_20141018_092931

I’ve been meaning to write this post for some time, but things have been a little hectic recently. That said I really wanted to write something, even if it is a little short, about Barcamp Manchester. The event took place over the weekend of 18th & 19th October and was just a fantastic weekend.

After a fairly decent break from the Barcamp scene, Manchester really came back and did it justice. Set in the fantastic SpacePort building on Lever street which is a meet and workspace, I arrived earlyish on the Saturday morning with fellow members of RossLUG. Carting in my bundle of swag I was shown my table in the main space and setup the Fedora table. As most will know I am a proud ambassador for the Fedora project and more proud of the fact we were able to sponsor the event.

 

The table looked great and we had plenty of swag and disks to give to the myriads of folks visiting the table throughout the day, there was great conversation with many people and the traditional barrage of questions from everyones friend Gino.

IMG_20141018_140632

IMG_20141018_105717

@tommybobbins and myself setup the timelapse cameras and live stream of the weekend, the timelapses can be viewed:

Saturday Main room:

Sunday Main room:

Other Room:

Werewolf:

Finally getting to meet fellow ambassador Dave ‘Kubblai’ McNulty who helped man the table and also give me the opportunity to attend a few talks. Plenty of great content in the talks I attended so if they are anything to go by there quality of the contect over the weekend was exceptionally high quality.

All in all a great weekend, seeing the regular attendees is always a highlight for me but also a high level of interest at the Fedora table in the Fedora project from people who have never heard of the project right up to those who are of an expert level. The organisers of the event need a huge pat on the back for this event and I will certainly look forward to next years!

Gluster, CIFS, ZFS – kind of part 2

A while ago I put together a post detailing the installation and configuration of 2 hosts running glusterfs, which was then presented as CIFS based storage.

http://jonarcher.info/2014/06/windows-cifs-fileshares-using-glusterfs-ctdb-highly-available-data/

This post gained a bit of interest through the comments and social networks, one of the comments I got was from John Mark Walker suggesting I look at the samba-gluster vfs method instead of mounting the filesystem using fuse (directly access the volume from samba, instead of mounting then presenting). On top of this I’ve also been looking quite a bit at ZFS, whereas previously I had a Linux RAID as the base filesystem. So here is a slightly different approach to my previous post.

Getting prepared

As before, we’re looking at 2 hosts, virtual in the case of this build but more than likely physical in a real world scenario, either way it’s irrelevant. Both of these hosts are running CentOS 6 minimal installs (I’ll update to 7 at a later date), static IP addresses assigned and DNS entries created. I’ll also be running everything under a root session, if you don’t do the same just prefix the commands with sudo. For purposes of this I have also disabled SELINUX and removed all firewall rules. I will one day leave SELINUX enabled in this configuration but for now lets leave it out of the equation.

In my case these names and addresses are as follows:

arcstor01 – 192.168.1.210

arcstor02 – 192.168.1.211

First off lets get the relevant repositories installed (EPEL, ZFS and Gluster)

yum localinstall --nogpgcheck http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release.el6.noarch.rpm
curl -o /etc/yum.repos.d/gluster.repo http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/glusterfs-epel.repo
curl -o /etc/yum.repos.d/glusterfs-samba-epel.repo http://download.gluster.org/pub/gluster/glusterfs/samba/EPEL.repo/glusterfs-samba-epel.repo

Local filesystem

As previously mentioned, this configuration will be hosted from 2 virtual machines, each will have 3 disks. 1 for the OS, and the other 2 to be used in a ZFS pool.

First off we need to install ZFS itself, once you have the above zfs-release repo installed this can be done with the following command:

yum install kernel-devel zfs

Perform this on both hosts.

We can now create a zfs pool. In my case the disk device names are vdX but they could be sdX,

fdisk -l

can help you identify the device names, whatever they are just replace them in the following commands.

Create a ZFS pool

zpool create -f  -m /gluster gluster mirror /dev/vdb /dev/vdc

this command will create a zfs pool mounted at /gluster, without -m /gluster it would mount at /{poolname} while in this case it’s the same I just added the option for clarity. The volume name is gluster, the redundancy level is mirrored which is similar to RAID1, there are a number of raid levels available in ZFS all are best explained here: http://www.zfsbuild.com/2010/05/26/zfs-raid-levels/. The final element to the command is where to host the pool, in our case on /dev/vdb and /dev/vdc. The -f option specified is to force creation of the pool, this is required remove the need to create partitions prior to the creation of the pool.

Running the command

zpool status

Will return the status of the created pool, which if successful should look something similar to:

[root@arcstor01 ~]# zpool status
 pool: gluster
 state: ONLINE
 scan: none requested
 config:
NAME STATE READ WRITE CKSUM
 gluster ONLINE 0 0 0
 mirror-0 ONLINE 0 0 0
 vdb1 ONLINE 0 0 0
 vdc1 ONLINE 0 0 0

errors: No known data errors

A quick ls and df will also show us that the /gluster mountpoint is present and the pool is mounted, the df should show the size as being half the sum of both drives in the pool:

[root@arcstor01 ~]# ls /
 bin boot cgroup dev etc gluster home lib lib64 lost+found media mnt opt proc root sbin selinux srv sys tmp usr var
 [root@arcstor01 ~]# df -h
 Filesystem Size Used Avail Use% Mounted on
 /dev/vda1 15G 1.2G 13G 9% /
 tmpfs 498M 0 498M 0% /dev/shm
 gluster 20G 0 20G 0% /gluster

If this is the case, rinse and repeat on host 2. If this is also successful then we now have a resilient base filesystem on which to host our gluster volumes. There is a bucket load more to ZFS and it’s capabilities but it’s way outside the confines of this configuration, well worth looking into though.

Glusterising our pool

So now we have a filesystem, lets make it better. Next up, installing glusterfs, enabling it then preparing the directories, for this part it is pretty much identical to the previous post:

yum install glusterfs-server -y

chkconfig glusterd on

service glusterd start

mkdir  -p /gluster/bricks/share/brick1

This needs to be done on both hosts.

Now only on host1 lets make the two nodes friends, create and then start the gluster volume:

# gluster peer probe arcstor02
peer probe: success.

# gluster vol create share replica 2 arcstor01:/gluster/bricks/share/brick1 arcstor02:/gluster/bricks/share/brick1
volume create: share: success: please start the volume to access data

# gluster vol start share
volume start: share: success

[root@arcstor01 ~]# gluster vol info share

Volume Name: data1
Type: Replicate
Volume ID: 73df25d6-1689-430d-9da8-bff8b43d0e8b
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: arcstor01:/gluster/bricks/share1/brick1
Brick2: arcstor02:/gluster/bricks/share1/brick1

If all goes well above we should have a gluster volume ready to go, this volume will be presented via samba directly. For this configuration a locally available shared area is required, for this we will create another gluster volume to mount locally in which to store lockfiles and shared config files.

mkdir  -p /gluster/bricks/config/brick1
gluster vol create config replica 2 arcstor01:/gluster/bricks/config/brick1 arcstor02:/gluster/bricks/config/brick1
gluster vol start config
mkdir  /opt/samba-config
mount -t glusterfs localhost:config /opt/samba-config

The share volume could probably be used by using a different path in the samba config but for simplicity we’ll keep them seperate for now.
The mountpoint for /opt/samba-config will need to be added to fstab to ensure it mounts at boot time.

echo "localhost:config /opt/samba-config glusterfs defaults,_netdev 0 0" >>/etc/fstab

Should take care of that, remember that needs to be on both hosts.

Samba and CTDB

We now have a highly resilient datastore which could withstand both disk and host downtime, but we need to make that datastore available for consumption and also highly available in the process, for this we will use CTDB, as in the previous post. CTDB is a cluster version of the TDB database which sits under Samba. The majority of this section will be the same as the previous post except for the extra packages and a slightly different config for samba. Lets install the required packages:

yum -y install ctdb samba samba-common samba-winbind-clients samba-client samba-vfs-glusterfs

For the majority of config files we will create them in our shared config volume and symlink them to their expected location. First file we need to create is /etc/sysconfig/ctdb but we will do this as /opt/samba-config/ctdb then link it afterwards

Note: The files which are created in the shared area should be done only on one host, but the linking needs to be done on both.

vi /opt/samba-config/ctdb
CTDB_RECOVERY_LOCK=/opt/samba-config/lockfile
 #CIFS only
 CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
 CTDB_MANAGES_SAMBA=yes
 #CIFS only
 CTDB_NODES=/etc/ctdb/nodes

We’ll need to remove the existing file in /etc/sysconfig then we can create the symlink

rm /etc/sysconfig/ctdb
ln -s /opt/samba-config/ctdb /etc/sysconfig/ctdb

Although we are using Samba the service we will be using is CTDB which allows for the extra clustering components, we need to stop and disable the samba services and enable the ctdb ones:

service smb stop
chkconfig smb off
chkconfig ctdb on

With this configuration being a cluster of essentially a single datapoint we should really use a single entry point, for this a 3rd “floating” or virtual IP address is employed, more than one could be used but lets keep this simple – 192.168.1.212. We also need to create a ctdb config file which contains a list of all the nodes in the cluster. Both these files need to be created in the shared location:

vi /opt/samba-config/public_addresses
192.168.1.212/24 eth0
vi /opt/samba-config/nodes
192.168.1.210
192.168.1.211

They both then need to be linked to their expected locations – neither of these exist so don’t need to be removed.

ln -s /opt/samba-config/nodes /etc/ctdb/nodes
ln -s /opt/samba-config/public_addresses /etc/ctdb/public_addresses

The last step is to modify the samba configuration to present the volume via cifs, I seemed to have issues using a linked file for samba so will only use the shared area for storing a copy of the config which can then be copied to both nodes to keep them identical.

cp /etc/samba/smb.conf /opt/samba-config/

Lets edit that file:

vi /opt/samba-config/smb.conf

Near the top add the following options

clustering = yes
idmap backend = tdb2
private dir = /opt/samba-config/

These turn the clustering (CTDB) features on and specify the shared directory where samba will create lockfiles. You can test starting ctdb at this point to ensure all is working, on both hosts:

cp /opt/samba-config/smb.conf /etc/samba/
service ctdb start

It should start OK, then health status of the cluster can be checked with

ctdb status

At this point I was finding that CTDB was not starting correctly, after a little bit of logwatching I found an error in the samba logs suggesting:

Failed to create pipe directory /run/samba/ncalrpc - No such file or directory

Also, to be search engine friendly the CTDB logfile was outputting

50.samba OUTPUT:ERROR: Samba tcp port 445 is not responding

This was a red herring, the port wasn’t responding as the samba part of CTDB wasn’t starting, 50.samba is a script in /etc/ctdb/events/ which actually starts the smb process.

So I created the directory /run/samba and restarted ctdb and the issue seems to have disappeared.

Now we have a started service, we can go ahead and add the configuration for the share. A regular samba share would look something like:

[share]
 comment = just a share
 path = /share
 read only = no
 guest ok = yes
 valid users = jon

In the previous post this would have been ideal if our gluster volume was mounted at share, but for this we are removing a layer and want samba to talk directly to gluster rather than via the fuse layer. This is achieved using a VFS object, we installed the samba-vfs-glusterfs package earlier. The configuration is slightly different within the smb.conf file also. Adding the following to our file should enable access to the share volume we created:

[share]
 comment = gluster vfs share
 path = /
 read only = No
 guest ok = Yes
 kernel share modes = No
 vfs objects = glusterfs
 glusterfs:loglevel = 7
 glusterfs:logfile = /var/log/samba/glusterfs-testvol.log
 glusterfs:volume = share

Notice the glusterfs: options near the bottom, these are specific to the glusterfs vfs object which is called further up (vfs objects = glusterfs). Another point to note is that the path is / this is relative to the volume rather than the filesystem, so a path to /test would be a test directory inside the gluster volume.

We can now reload the samba config, lets restart for completeness (on both nodes)

service ctdb restart

From a cifs client you should now be able to browse to \\192.168.1.212\share (or whatever IP you specified as the floating IP).

ctdb

 

All done!

To conclude, here we have created a highly resilient, highly available, very scalable storage solution using some fantastic technologies. We have created a single access method (Cifs on a floating  IP) to a datastore which is then stored on multiple hosts, which in turn store upon multiple disks. Talk about redundancy!

Useful links:

http://www.centos.org

http://zfsonlinux.org/

http://www.gluster.org/

http://ctdb.samba.org/

 

Upgrade CentOS 6 to 7 with Upgrade Tools

I decided to try the upgrade process from EL 6 to 7 on the servers I used in my previous blog post “Windows (CIFS) fileshares using GlusterFS and CTDB for Highly available data”

Following the instructions here I found the process fairly painless. However there were 1 or two little niggles which caused various issues which I will detail here.

The servers were minimal CentOS 6.5 installs, with Gluster volumes shared via CTDB. The extra packages installed had mostly come from the EPEL or Glusterfs repositories, and I believe this is where the issues arise – third party repositories.

My initial attempt saw me running:

preupg -l

which gave me the output: CentOS6_7

This meant that I had CentOS 6 to 7 upgrade content available to me, this could now be utilised by running:

preupg -s CentOS6_7

Which then ran through the preupgrade checks and produced the report of whether my system could, or should, be upgraded.

The results came back with several informational items, but more importantly 4 “needs_action” items.

These included “Packages not signed by CentOS”, “Removed RPMs”, “General” and “Content for enabling and disabling services based on CnentOS 6 system”

Firing up links and pointing it at the output preupgrade/result.html file I took a deeper look into the above details.

“Packages not signed by CentOS” as expected covered the third party installed applications, in my case the glusterfs rpms and the epel-release, which were to be expected. The other sections didn’t present any great worries so I pressed on with the upgrade:

centos-upgrade-tool-cli --network 7 --instrepo=http://mirror.centos.org/centos/7/os/x86_64/

running this takes the data from the previous report and runs an upgrade process based on it. Interestingly the first part of the process (redhat_upgrade_tool.yum) checks out the yum repos that are configured and EPEL “seems OK” whereas the glusterfs-epel ones don’t. This called for a little more investigation, as on my first upgrade trial run these packages failed to upgrade, luckily I took a snapshot of the machine before upgrading so could try again.

Strangely, even though the $basearch and $releasever variables were used in the config file, manually changing the $releasever to 7 (as $releasever translates to 7.0) seemed to do the trick. I manually edited the EPEL file too as this contained epel-6 in the url. After this I also noticed that the gluster services were no longer listed in the INPLACERISK: HIGH categories but had been moved to the MEDIUM.

Continue with upgrade [Y/N]?.

yes please!

The upgrade tool then goes through the process of downloading the boot images and packages ready for the upgrade, for some reason I got a message about the CentOS 7 GPG key being listed but not installed, so while I hunted out the key to import I re-ran the upgrade tool with the –nogpgcheck switch to skip that check. The tool finished successfully then and then prompted me with:

Finished. Reboot to start upgrade.

Ok then, here goes….

Bringing up the console to that machine showed me it booting into the images it downloaded in preparation for the upgrade. Mostly a screen of RPM package updates and reconfiguration. The update completed fairly quickly then automatically rebooted.

As mentioned above this was the second attempt at an upgrade on this machine, the first time it was upgraded I was prompted with the emergengy login screen after reboot. This turned out, strangely, to be that the glusterfs packages hadn’t been upgraded so I logged onto the console brought up eth0 and ran yum update. After a reboot I was faced with a working system.

The second attempt I managed to ensure the gluster packages were included in the upgrade so after crossing fingers the reboot ended with a login prompt. Great News!

The only issues I faced were Gluster volumes not mounting at boot time, but I’m sure this is a systemd configuration which can be easily rectified and really don’t change the success of the upgrade process.

All in all, good work from the Red Hat and CentOS teams, happy with the upgrade process. It’s not too far removed from Fedup in Fedora of which I’m sure it’s based.

Update: The issues I faced with my gluster volumes not mounting locally were resolved by adding the _netdev directive after defaults in fstab e.g.:

localhost:data1 /data/data1 glusterfs defaults,_netdev 0 0

All that was occurring was systemd was trying to mount the device as a local filesystem, which would try to run before the glusterd service had started. Adding this option delayed the mounting until all network-target was complete essentially.

The other issue that became apparent after I resolved the gluster mounting issue was the CTDB service not running once boot had completed, this was due to the CTDB service trying to start before filesystems were active, I modified the ctdb.service file to ensure that it only started after gluster had started which seemed to be enough. I guess that getting it to start after the filesystems had mounted would be better but for now it works. To do this I modified the  /usr/lib/systemd/system/ctdb.service file and changed the line:

After=network.target

in the [Unit] section to

After=network.target glusterd.service