Knowing the Moment a Port Opens

Automated Attempts

Sometimes when a server is rebooted, whether a clean soft reboot or a hard reboot after a crash, I need to perform a task on it as quickly as possible. This can be for many reasons like ensuring all services are started to making a quick change. Sometimes I just need to know the moment a certain service is started to notify everyone of this fact. The point is that every second counts.

When the servers starts up and the network is joined you can start receiving ping responses from the server. At this point all the services haven’t started up yet (on most configurations at least), so I can’t necessarily log into the server or access the specific service, yet. Attempting to do so I would get a connection refused or port closed error.

What I usually do in cases where I urgently need to log back into the server is ping the IP address and wait for the first response packet. When I receive this packet I know the server is almost finished booting up. Now I just need to wait for the remote access service to start up. For Linux boxes this is SSH and for Windows boxes it’s RDP (remote desktop protocol).

I could try to repeatedly connect to it, but this is unnecessarily manual, and when every second counts probably less than optimal. Depending on what I’m trying to do I have different methods of automating this.

If I just needed to know that a certain service is started and available again, I would put a netcat session in a loop, which would repeatedly attempt a connection. As long as the service isn’t ready (the port is closed), the netcat command will fail and exit. The loop will then wait for 1 second and try again. As soon as the port opens the connection will succeed and netcat will print a message stating the connection is established and then wait for input (meaning the loop will stop iterating). At this point I can just cancel the whole command and notify everyone that it’s up and running. The command for doing this is as follows:

while true; do nc -v 10.0.0.221 80; sleep 1; done

If I needed remote access to the server, I would use a similar command as above, but use the remote access command instead, and add a break statement to quit the loop after the command was successful. For example, for an SSH session I would use the ssh command, and for a remote desktop session the rdesktop command. A typical SSH command will look like:

while true; do ssh 10.0.0.221 && break; sleep 1; done

This will simply keep trying the ssh command until a connection has been established. As soon as a connection was successful I will receive a shell, which when exited from will break the loop and return me to my local command prompt.

Automatically Running a Command

If you had to run some command the moment you are able to do so, you could use the above SSH command with some minor modifications.

Lets say you wanted to remove the file /opt/repository.lock as soon as possible. To keep it simple we’re assuming the user you log in as has permission to do so.

The basic idea is that each time you fail to connect, SSH will return a non-zero status. As soon as you connect and run the command you will break out of the loop. In order to do so, we need a zero exit status to distinguish between a failed and successful connect.

The exit status during a successful connect, however, will depend on the command being run on the other end of the connection. If it fails for some reason, you don’t want SSH to repeatedly try and fail, effectively ending up in a loop that won’t exit by itself. So you need to ensure it’s exit status is 0, whether it fails or not. You can handle the failure manually.

This can be achieved by executing the true command after the rm command. All the true command does is to immediately exit with a zero (success) exit status. It’s the same command we use to create an infinite while loop in all these examples.

The resulting command is as follows:

while true; do \
  ssh 10.0.0.221 "rm -f /opt/repository.lock ; true" && break; \
  sleep 1; \
done

This will create an infinite while loop and execute the ssh and sleep commands. As soon as a SSH connection is established, it will remove the /opt/repository.lock file and run the true command, which will return a 0 status. The SSH instance will exit with success status, which will cause a break from the while loop and end the command, returning back to the command prompt. As with all the previous examples, when the connection fails the loop will pause for a second, and then try again.

Conclusion

By using these commands instead of repeatedly trying to connect yourself, there is a max of 1 second from the time the service started till when you’re connected. This can be very useful in emergency situations where every second you have some problem could cost you money or reputation.

The Linux terminal is a powerful place and I sometimes wonder if those who designed the Unix terminal knew what they were creating and how powerful it would become.

So Why Love Linux? Because the Linux terminal allows you to optimize your tasks beyond humanly capability.

 

Terminal Auto Complete

Overview

Command line, or Bash completion is a Bash user’s best friend. If you had to type out everything you do on the command line, you would be doing a lot of typing. Similar to auto completion in most text fields these days, Bash completion allows you to start typing something and then have Bash complete it for you.

Basic Usage

You activate completion with the TAB key.

So lets say you wanted to change into a directory named books, and there were 3 directories available, namely books, movies and music. If you were to type

$ cd b

and then press the TAB key, it would end up as

$ cd books

Though, if there was another directory next to books that also starts with bo, for examples boxes, then Bash would only complete up till what they have in common, which is bo. You can then press the TAB key again to have it show you the available options, as so:

$ cd bo
books/ boxes/

From here you can just type one more character to distinguish them and then press TAB again to have it complete the whole word. So you would type o and press TAB, and your command will be completed.

You get used to this very quickly, and it becomes a very powerful tool.

Smart Completion

The basic completion of files and directories are the default. You get smart completion which is command specific. These are done based on what you have typed so far and computed with procedures. For example, if you type:

$ unzip

and then press the TAB key, it could complete only ZIP files, instead of suggesting all available file.

It doesn’t stop there, though. Non-file completions are also possible. For example the apt-get command has various sub commands like install, remove, source and upgrade. All it’s sub commands are also completed. For example, typing

$ apt

and pressing the TAB key twice, gives you

$ apt-get 
autoclean  autoremove    build-dep        check
clean      dist-upgrade  dselect-upgrade  install
purge      remove        source           update
upgrade

You can now start typing one of them and press TAB to have it complete. For example, typing i and pressing TAB will complete into install.

Customizing

It gets even better. All this is very easily customized. If I wanted to have the perl command be completed only with files ending in the .pl extension, I can achieve this by creating a file at /etc/bash_completion.d/perl and giving it the following contents:

complete -f -X '!*.pl' perl

That’s it. Whenever I complete the arguments of the perl command, it will only do so for files ending in .pl.

Other Shortcuts

There are some other shortcuts also related to command line editing and history.

You can select your previous command by pressing the Up key. Repeatedly pressing this will allow you to browse through your history of commands. Pressing the Down will make you go forward into your history.

Or, if you start typing a command and realize your missed something and want to clear the whole line to start over, you can press Alt+R. This is like pressing and holding backspace until the line is cleared. An alternative is to just press Ctrl+C and return to the command line.

Sometimes you want to perform a series of commands on the same file. For example, to change the ownership and permissions of the same file you could type:

chown root:root myfile
chmod 0644 myfile

This requires typing the filename twice. Bash will do this for you on the second line if you press Alt+. (Alt and the full stop character). For example, typing:

chown root:root myfile
chmod 0644

and then pressing Alt+., will complete your command by taking the last argument of the previous command (ie. myfile) and inserting it at your current position, resulting in:

chmod 0644 myfile

Conclusion

So Why Love Linux? Because the command line, one of Linux’s true powers, offers powerful editing capabilities.

Easily Create In-Memory Directory

In Memory

If you have ever had to run a task which performs a lot of on-disk file operations you know how much of a bottleneck the hard drive could be. It’s just not fast enough to keep up with the demands of the program running in the CPU.

So what I usually do when I need to run some very disk intensive tasks is to move it all to an in-memory filesystem. Linux comes with a module called tmpfs, which allows you to mount such a filesystem.

So, assuming my file operations would be happening at /opt/work. What I would do is

1. Move /opt/work out of the way, to /opt/work.tmp
2. Make a new directory called /opt/work
3. Mount the in-memory filesystem at /opt/work
   sudo mount -t tmpfs none /opt/work
4. Copy the contents of /opt/work.tmp into /opt/work.
5. Start the program.
6. Then when the program completes I would copy everything back, unmount the in-memory filesystem and clean up.

Now, since the directory where all the operations will be happening isn’t actually on disk (but in memory), it can happen without any hard drive bottleneck.

Here are some benchmark results showing how much faster this is. I was writing 1GB of data to each filesystem. Specifically look at the last bit of the last line.

For the memory filesystem:
quintin@quintin-VAIO:/dev/shm$ dd if=/dev/zero of=test bs=$((1024 * 1024)) count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.984447 s, 1.1 GB/s

And for the hard drive filesystem:
quintin@quintin-VAIO:/tmp$ dd if=/dev/zero of=test bs=$((1024 * 1024)) count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 24.2657 s, 44.2 MB/s

As you can see, the memory filesystem was more than 24 times faster than the hard drive filesystem. I was able to write at only 44.2MB/s on the hard drive, but 1.1 GB/s on the memory filesystem. Note that this isn’t it’s maximum, since I had some other bottlenecks here. If you were to optimize this you would be able to write even faster to the memory filesystem. The fact remains that filesystem intensive tasks can run much faster when done this way.

There are some risks involved, in that loosing power to the system will cause everything that was in the memory filesystem to be lost. Keep this in mind when using it. In other words, don’t store any critical/important data only in an in-memory filesystem.

The Core of it All

So in the end it all comes down the the fact that you can easily create a in-memory filesystem. All you need to do to achieve this is decide on a directory you want to be in-memory, and mount it as such.

For example, if we were to choose /home/mem as an in-memory directory, we can mount it as follows:
sudo mount -t tmpfs none /home/mem

If we need it to be persistently mounted as such (across system boots), we can add the following line to /etc/fstab:
none /home/mem tmpfs defaults 0 0

Conclusion

So Why Love Linux? Because with a single command you can get a temporary in-memory filesystem running to help you speed up a filesystem intensive task.