February 10th, 2012

Tunneling ILO over ssh

Do you want to access HP ILO via a SSH tunnel?

HP’s iLO (Integrated Lights Out) is great, especially if you wish to administer the server remotely for the installation of the OS, check the video output for a kernel panic when the OS is unresponsive, configure the BIOS, or just so that you can avoid standing in a cold server room or because you’re just not at the premises.

The servers that I administer come with iLO1, iLO2 or iLO3 and it’s great to see the interface developing at every release.

If the remote server is located on a remote LAN that does not expose the server to the outside world e.g. a DMZ you can tunnel a connection over ssh to provide access both to the remote iLO web interface over https, and the remote console that allows you to access the display of the remote machine via a browser based Java applet, as if you were using keyboard, mouse and monitor plugged into the machine itself.

So, how do you access iLO remote console over ssh?

To do so, you just need to forward a couple of ports via a handy ssh server somewhere on the remote network.

You may need to refer to some of the port settings within the iLO administration pages, I’ve included a screenshot of where to find them within iLO1 below:

hp ilo1 administration globalsettings remote console port

The below steps assume that you have a proven, working ssh server on the remote network and the necessary ports are forwarded on your remote router to allow you access to the remote network and therefore, the iLO on the remote server.

  1. find the necessary ports for java, Administration -> Access Settings -> Servicenote the port that your iLO is listening on for “remote console” – this is the port that the server will expect the Java remote console to be able to communicate over.
  2. Now you need to create the relevant ssh tunnel, I’ve broken this down to a command with some bash variables to make it easier to understand what’s what.
    1. the IP of iLO on your remote network 
      iloaddress=192.168.1.10
    2. the local https port to forward 
      localhttps=8443
    3. the https port the iLO is listening on 
      remotehttps=443
    4. local port to forward for Remote Console (probably should be the same as the Remote Console Port within: Administration -> Access Settings -> Service) 
      localconsole=17990
    5. the Remote Console Port within: Administration -> Access Settings -> Service 
      remoteconsole=17990
    6. Your username for ssh-ing into a server on the remote network 
      sshusername=<your-user-name>
    7. Your remote IP or domain name for the remote network 
      sshserver=my-secret-router.dyndns.org
    8. the port on the remote router for forwarding to an ssh server on the remote network 
      sshport=2205
  3. Now run the ssh command to bring up the tunnel using the above parameters: 
    ssh -fnNL ${localhttps}:${iloaddress}:${remotehttps} -L ${localconsole}:${iloaddress}:${remoteconsole} ${sshusername}@${sshserver} -p ${sshport}
  4. Finally, if all is well, access the iLO via your browser, I’m launching it via the command line here to provide the correct port based upon the pre-populated variables: 
    firefox https://127.0.0.1:${remoteconsole}
  5. You can check that the tunnel command is still running with the following command: 
    ps faux | grep -v grep | grep "ssh -fnNL ${localhttps}:${iloaddress}:${remotehttps}\|USER"
  6. Handily, this also gives you the PID(Process id) of your tunnel, so you can kill your tunnel like this: 
    kill <process id>

Let’s put the above together as a single block that you can paste into a text editor to make it into a bash script, all in one go along with bash comments to document it (once you’ve edited the parameters of course):

#!/bin/bash
# the IP of iLO on your remote network
iloaddress=192.168.1.10
# the local https port to forward
localhttps=8443
# the https port the iLO is listening on
remotehttps=443
# local port to forward for Remote Console
localconsole=17990
# the Remote Console Port
remoteconsole=17990
# Your username for ssh-ing into a server on the remote network
sshusername=<your-user-name>
# Your remote IP or domain name for the remote network
sshserver=my-secret-router.dyndns.org
# the port on the remote router for forwarding
sshport=2205
# launch tunnel
ssh -fnNL ${localhttps}:${iloaddress}:${remotehttps} -L ${localconsole}:${iloaddress}:${remoteconsole} ${sshusername}@${sshserver} -p ${sshport}
# check tunnel
ps faux | grep -v grep | grep "ssh -fnNL ${localhttps}:${iloaddress}:${remotehttps}\|USER"
# launch browser
firefox https://127.0.0.1:${remoteconsole} &

Remember to make the script executable once you’ve saved it with a chmod +x /path/to/script

Posted in iLO, server administration, ssh | No Comments »

November 5th, 2010

Permissions, permissions, permissions – part 1

Whenever “the guys” have a problem accessing some shared resource, it usually comes down to permissions, either:

  • file/directory permissions on the server
  • samba permissions
  • user permissions
  • permission masking on the users local machine for their mounts

Today I’d like to talk about file/directory permissions.

Your number’s up

The first time that I was introduced to *nix style file permissions, the listing of “permission types”

  • execute
  • write
  • read

were explained to me “numerically”.  The explanation of how to set the permission mode on a file using chmod “CHangeMODe” were demonstrated “Numerically”:

chmod 0654 somefile.txt

and do you know what? In all of these years, I *still* cannot remember what each number means.  I know that each permission has a numeric value, and that they can be combined using binary “and-ing”, but I just can’t recall that 1=execute, 2=write and 4=read.

It’s not big and it’s not clever

I remember programming as a kid, learning ASCII and coding in BASIC on a Tandy TRS 80, on a Toshiba MSX and various friends Sinclair Spectrums.  I learnt that you could check which key had been pressed by comparing the trapped key with it’s ASCII value, so to see if the “A” key had been pressed, you’d check it against it’s ASCII value “65″.

But what I later learnt is that proramming code littered with obscure codes does little to aid readability and maintainability, it does not lend itself to error-free code.

So I switched, now if I were to write some code to compare with say, the letter “A” I would do just that – compare with the letter “A”, not the ASCII value 65.

Keep it simple, stupid

So it is also with permissions.  I don’t “chmod” numerically, I do it the much simpler, easier and less error prone way.

chmod supports textual permissions specification, so that’s what I use, for example, to specify that the “user” of a file may read,write and execute it, that memebers of the files’ group may read and execute it, and that any others may only execute it (ok, a somewhat contrived example I’ll admit)  we can choose to specify the permissions either numerically, or textually:

  • chmod 751 somefile
  • chmod u=rwx,g=rw,o=x somefile

Ok, so which is more readable and explicit in it’s meaning? Yes I’ll admit the “textual” example is a little more verbose and perhaps doesn’t carry the “geek cred” that some might seek to attain, but come on, we’re supposed to be getting on with stuff, not pretending to have some mystical knowledge based upon some obscure numerical code.

Perhaps it’s a sign that I don’t use the numerical codes – but whilst putting the numeric example together above, I had to go back and make a correction to one of the permission codes as I got it wrong…

Suddenly, It all makes sense

The other thing about using the textual method to specify permissions, is that, if you take a look at a directory listing, it all comes together nicely, take a look at some example files:

---x------ 1 root root     0 Oct 31 10:05 0100.perm
--w------- 1 root root     0 Oct 31 10:05 0200.perm
--wx------ 1 root root     0 Oct 31 10:05 0300.perm
-r-------- 1 root root     0 Oct 31 10:05 0400.perm
-r-x------ 1 root root     0 Oct 31 10:06 0500.perm
-rw------- 1 root root     0 Oct 31 10:06 0600.perm
-rwx------ 1 root root     0 Oct 31 10:06 0700.perm

Yup, the permissions are expressed textually.  Because it’s more accessible, because it’s readable and it’s not so obscure.

But as you might have guessed from the filenames, what I’ve actually done here is effectively create a “crib sheet” for myself, – the filename of each file is actually the numerical, octal equivalent of the permissions assigned to that file.

Why would I do that?  Well, that’s simple, because when you’re editing config files for systems such as logrotate you still have to specify the permissions for newly created empty logfiles in octal.

Is there cake? Can I eat it?

As I can never remember them, I made the “crib sheet”.  In my opinion, it’d be “safer” and easier to specify the permissions in either the same style that we see in the directory listing: -rwx—— for example, or preferably, as you would with chmod: u=rwx,go=

But I guess you can’t have everything.

Posted in GNU, Linux | No Comments »

October 27th, 2010

SED – Edit in place

example of "in-place" editing with sed

example of "in-place" editing with sed

I use the GNU sed command (sed = Stream EDitor) to process text files on a regular basis.

Sed is brilliantly fast.  It’s perfect for tricks such as altering the content of a huge database dump, or a spreadsheet or csv file.

One of my colleagues routinely opens csv files in a spreadsheet application to edit some item of other, only to find that upon saving, the dates have get converted from ISO to UK format.  Using sed can avoid this frustration and speed up his workflow.

“S” is for “Substitute”

To my shame, I typically only use the ’s’ option – to substitute one piece of text for another.

But today I finally found another use, which I had been sure must surely exist; that of deleting a specific line from a file based upon it’s position.

The scenario is that I often ssh into boxes, when setting up servers for instance, that have the same IP address as a previous server, (because they initially use DHCP to acquire an address).

This means that when ssh compares the finger print to it’s known_hosts file, it balks, tells me the fingerprint has changed, what line the existing fingerprint is on but wont let me connect.  This usually means that I edit my known hosts file, remove the offending line and then I can get on.

Now that I’ve found out how to do this with sed however, it’s much easier.

“i” is for “in-place”

In the past when I’ve used sed, I’ve either piped the output to another command using the “|” pipe character, or redirected it to another file using “>” .
But this isn’t so convenient when you want to actually edit a file.  The magic switch is “-i” which means “edit in place” the command syntax is:

sed -i ‘<nth line>,<number of lines to apply action to><action>’ <filename>

e.g. sed -i ‘2,1d’

Find the 2nd line, then for 1 line, delete the line.

Testing

First I created a test file:

$ cat > test.txt
one
two
three
four
[CTRL + D]

We can then test using sed to modify the file (as sed will typically output the results to standard out – i.e. to the terminal)

$ sed -e '2,1d' ./test.txt

Let’s take a look at the file now:

$ cat test.txt
one
three
four

Result

Hurrah! the line has gone!

So now, when ssh balks because the host key of the remote system differs to line x of ~/.ssh/known_hosts changed (usually because I’ve re-installed the OS at the other end) instead of firing up an editor, such as nano or gedit to remove the offending line, it’s just a straight forward “sed -i ‘x,1d’ ~/.ssh/known_hosts” and the old key is gone.

Posted in GNU, Linux | 1 Comment »

August 5th, 2010

Guide to CentOS Network installation via HTTP

You may have seen my voicing of my frustrations at the apparent lack of guidance to installation of CentOS 5.4 via Network / HTTP.
This is my step-by-step guide for installing CentOS 5.4 via network over HTTP. Or FTP.

I was installing upon a virtual machine using Virtual Box OSE, but the steps below should pretty much apply to a real/physical installation.
This is a description of what *I* did, your mileage may vary. The usual warnings apply; do this at your own risk – I accept no responsibility for your actions, whether you follow my *guide* or not. It is after all, *my guide*, that is, it is a guide for my own future reference.

  1. Getting the Net installer ISO.
    1. Use your web-browser to navigate to the
      centos.org mirror list
    2. I’m going for the 64bit Install of CentOS 5.x: so I visited the following Urls to find a mirror in the UK:
  2. Mount the ISO onto your virtual machine
  3. Start the virtual machine
  4. Textual set-up (Pre GUI)
    1. When prompted, I pressed the [ENTER] key for a GUI based install
    2. I selected my language, “English”
    3. I selected my keyboard type, “uk”
    4. I selected my Installation Method, “HTTP”
    5. I used the default settings for the “Configure TCP/IP” screen
    6. HTTP Setup
      You need to find an online source for the installation files.
      I decided to use the same mirror for the Net/HTTP installation as I did to download the Net installer, so I did the following:

      1. navigated two levels up from http://anorien.csc.warwick.ac.uk/mirrors/centos/5.4/isos/x86_64/
        to the directory http://anorien.csc.warwick.ac.uk/mirrors/centos/5.4/
      2. navigated down via “os” -> “x86_64″ to http://anorien.csc.warwick.ac.uk/mirrors/centos/5.4/os/x86_64/
      3. I broke the url down into it’s constituent parts; server “Web site name” and the subfolders “CentOS directory” thus:
        Web site name: anorien.csc.warwick.ac.uk
        CentOS directory: mirrors/centos/5.4/os/x86_64/
      4. The Net installer will attempt to retrieve a file, “stage2.img” from the details provided in the previous step.If this fails, then either:
        • The url is wrong
        • the file doesn’t exist
        • there’s a problem with your networking
  5. GUI installer phase
    1. Eventually, after about 10 minutes, I was greeted with an X-session/GUI installer. I clicked [Next].
    2. This is a fresh (virtual) machine, so I selected the defaults for disk partitioning. I clicked [Next].
    3. I’m only using a small amount of ram, about 320MB for this VM. So I allowed the installer to activate a swap partition.
    4. I selected the nearest city representative of my timezone and location via the map;
      London, Englad, United Kingdom. [Next].
    5. This is going to be a cli based server, so I de-selected the “Desktop – Gnome” option and selected the “Server” option.
      I clicked [Next].
    6. I entered a nice, long, obscure password for my “root” user administration account.
  6. The installer was ready to begin installing the system. I clicked [Next]
  7. I watched the install whilst attending to other duties.
    Eventually I was greeted with the “Congratulations, the installation is complete” message. I clicked [Reboot].
  8. After un-mounting/ejecting the Net installer ISO, my machine booted.I was presented with some options via the “Setup Agent” – I went with the defaults, selecting the “Exit” option.
    Eventually I was presented with a nice, textual, login prompt.

Sometimes you just need a shell

One issue that I had whilst using the Graphical installer, occurred during a subsequent CentOS installation on a physical (rack) server.
A post-partitioning screen asks the user to specify upon which device to install the bootloader.
Unsure which device was the correct one, I tried an old fallback plan to get a shell (command line terminal); hitting CTRL + ALT + F1, which actually worked.
This yielded a prompt that allowed me to mount and list the contents of the various disk devices to determine which was the correct one.
Returning to the graphical installer was merely just a case of sequentially working through the F-keys in combination with CTRL + ALT until I found the right one.

Posted in CentOS, Linux | No Comments »

May 13th, 2010

Frustrations with CentOS 5.4 via Net Installation

It’s raining outside, my girlfriend is in bed with a cold and I’ve retreated to the office with the cat for company.
Today I’m setting up a pair of web and database servers, using virtual box for replication and disaster recovery testing.
The OS of choice for todays experiments has naturally been chosen to match that of some of our real-world production web and database servers: CentOS 5.4

Despite being a self confessed Ubuntu/Debian derivative fan, I’ve found that CentOS does provide a comfortable environment within which to work, although there are a few niggles that bother me with the installation:

  1. You can’t install from the LiveCD.  Well, not unless you’re willing to jump through hoops to do so. (Some instructions expect a “roll-your-own ISO” approach)
  2. If you choose to go the “Net Install” route, you are not provided with a list of http or ftp mirrors from which to install – you are provided merely with a couple of entry boxes in which to specify the source ftp/http server and path.

Now come on folks, it isn’t hard to to provide a default list of mirrors within the installer, or at the very least, a facility within the installer to at least download a list of mirrors, or else a straight forward guide on the CentOS.org site with links leading to the appropriate urls.

The CentOS website appears to be based upon the basic assumption that users are interested in installation via the downloading of a CD or DVD .ISO image.  Were installation via the Live CD a straight forward matter, then this wouldn’t be such an issue, but it’s not a straight forward matter.

Sure, I could download the Installation DVD image – but that’s comparitively huge.  I don’t really want *ALL* of those packages for a server install do I?
No, the Net install .ISO is perfect at a mere 9.4MB.  Yup – that’s all it is.  I think that the target audience is that of sys-admins looking to roll out across multiple machines on a LAN. But it also works for single installations, across the internet.

The process is, simple.  You boot from the Net Install .ISO, answer some questions regarding language, keyboard, and installation method.  It’s perfect.

CentOS 5.4 Net Install dialogue - FTP

CentOS 5.4 Net Install dialogue - FTP

Not quite perfect

Perfect save for one problem – trying to get the URL details for those “precious” servers that are able to provide the packages and resources for the Net install to work.
Finding those details via the CentOS website just doesn’t seem that easy.

Now remember, Google is our friend, so we turn to the Big “G” next, surely this’ll find a mirror list and a guide on the CentOS site for us?

Er… No.

No, the links all seem to centre around requests for help from those stranded out at sea, waiting on some assumed bug or other (I think it’s just a problem of the wrong URL being used) to be fixed, or questions about how to set up an ftp server on their network to then proceed with the network installation.

No official concrete guide.

Come on people, we can do better than this.  A How-To and a mirror list that is tailored to Network installation via ftp or http is a must and should be immediately accessible on the CentOS installer and download pages.

Eventually, I found a url;

http://ftp.linux.ncsu.edu/mirror/mirror.centos.org/5.4/os/x86_64/images/boot.iso

Entering it into the dialogue box and hitting [Ok] caused the dialogue to disappear, and then re-appear – no error information, no user feedback at all.

after much url editing, tweaking and testing, I managed to get the details down to:

FTP site name: ftp.linux.ncsu.edu
CentOS directory: /mirror.centos.org/5.4/os/X86_64

After a short delay and some flashing from the virtual NIC, I was presented with the following message:

Unable to retrieve
ftp://ftp/linux.ncsu.edu//mirror.centos.org/5.4/os/X86_64/images/stage2.img.

Aside from everything else, (i.e. the relevant file not being present) it looks like that trailing forward slash isn’t necessary for the ftp server.

More googling yielded the following thread:

http://centos.org/modules/newbb/viewtopic.php?viewmode=thread&topic_id=24961&forum=37&post_id=102175

Armed with the info from the “not quite successful” stage2.img we can start inferring what info we really need, so it’s time to hit the mirror list at http://mirror.centos.org/centos/5/ to see whether we can find a good, local mirror and try a bit more URL tweaking to get us to where we want to be.

http://www.centos.org/ -> Downloads -> Mirrors leads us to:

CentOS Downloads -> Mirrors

http://www.centos.org/modules/tinycontent/index.php?id=15

Again, this page is structured around the assumption that we want to download an ISO. We don’t, but we do want 5.4, so lets follow that “CentOS-5 ISOs” link and see whether we can find something more to our liking in the tree

Interesting, lets climb the directory tree and see whether there are any sibling folders that we may make use of?
Ah! the “OS” directory – let’s explore that and see whether the files that we need can be found in there?

Bingo, we finally navigate to http://mirror.centos.org/centos/5/os/x86_64/images/ and find this, complete with “stage2.img”

Lets use that url, selecting http rather than ftp.

CentOS Net Installation HTTP Setup

CentOS Net Installation HTTP Setup

Bingo!

After about 10 minutes (with my connection, machine etc..) We get an X screen and a sexy GUI based installer.

Sexy CentOS 5.4 GUI Installation screen

See my “Guide to CentOS Network installation via HTTP” for the straight forward installation procedure.

Posted in CentOS, Linux | No Comments »

May 3rd, 2010

Who, what, why. An introduction.

Who am I?

I’m Isaac, I’ve been working as a developer and IT specialist since the mid to late 1990’s.  Over the years I’ve been employed, contracted and self-employed.  I’ve handled hardware, networking, developing and administering.

I’ve always said that the WORST thing about being in I.T. is that you can sometimes spend half your working day on the floor, crawling around under someones desk, but that the *BEST* thing about being in I.T. is that you get to spend half-your working day on the floor, crawling around under somebodies desk.

What do I do?

I’ve worked as an application developer in MS Access, VB and C. As a web developer in ASP 2 and 3 with IIS, MS Access and MS Sql server, Macromedia Director, Java and all of the usual technologies; html, css, javascript.

Sometime during the “noughties” I switched into Apache, Mysql, PHP,  PostgreSQL and embraced the open source route.

These days my time is split between development with PHP/PostgreSQL and being a Linux sys admin, managing web and database server clusters, lan servers and desktops, using distributions such as Ubuntu, CentOS, RHEL, Fedora and occasionally Gentoo.

*NIX

I first discovered that I liked the “Unix way” whilst studying software development in C, using a Xenix based multi-user system.  Something just seemed “right” about it.  I enquired of my instructor how I could find a version of Xenix for home – he mentioned the existence of Minix.  I never got into Minix, but then I stumbled upon GNU/Linux.

Over the following years I tinkered with Caldera, DSL, DSL-N, Eeebuntu, Kanotix,  Knoppix, Linux Mint, Mandrake, Mandriva, MuLinux, the old Red Hat releases, Slackware, SUSE and probably flirted with a few other Linux distributions, before finally falling in love with Ubuntu.

I do have to admit that I favour the Debian derived distributions.  I think that it’s something to do with the well thought out documentation and file system layout that I prefer.

But that’s not to say that I think that the Debian way is necessarily “better” than say the Red Hat Enterprise Linux / CentOS way – I just find the Debian way a little more comfortable, in the same way that I prefer my old diesel saloon to my newer petrol estate car.

Why this blog?

This blog is intended as my repository for the things that I learn and wish to share, to provide a resource for myself for those moments when I catch myself staring at a screen whispering the words – “now how did I do this last time?”

If anybody else benefits from these notes then so much the better!

Posted in Me, Myself and I. | No Comments »