Advanced Topics

Debian GNU/Linux: Guide to Installation and Usage by John Goerzen and Ossama Othman is part of the HackerNoon Books Series. You can jump to any chapter in this book here. Advanced Topics

13. Advanced Topics

By now, you should have a strong base for which to build your GNU/Linux skills on. In this chapter we cover some very useful information regarding some advanced GNU/Linux features.

13.1 Regular Expressions

A regular expression is a description of a set of characters. This description can be used to search through a file by looking for text that matches the regular expression. Regular expressions are analogous to shell wildcards (see section 6.6 on page [*]), but they are both more complicated and more powerful.

A regular expression is made up of text and metacharacters. A metacharacter is just a character with a special meaning. Metacharacters include the following: . * [] - ^ $.

If a regular expression contains only text (no metacharacters), it matches that text. For example, the regular expression “my regular expression” matches the text “my regular expression,” and nothing else. Regular expressions are usually case sensitive.

You can use the egrep command to display all lines in a file that contain a regular expression. Its syntax is as follows:

egrep ’regexp’ filename1 ...

The single quotation marks are not always needed, but they never hurt.

For example, to find all lines in the GPL that contain the word GNU, you type

egrep ’GNU’ /usr/doc/copyright/GPL

egrep will print the lines to standard output. If you want all lines that contain freedom followed by some indeterminate text, followed by GNU, you can do this:

egrep ’freedom.*GNU’ /usr/doc/copyright/GPL

The . means “any character,” and the * means “zero or more of the preceding thing,” in this case “zero or more of any character.” So .* matches pretty much any text at all. egrep only matches on a line-by-line basis, so freedom and GNU have to be on the same line.

Here’s a summary of regular expression metacharacters:

. Matches any single character except newline.

* Matches zero or more occurrences of the preceding thing. So the expression a* matches zero or more lowercase a, and .* matches zero or more characters.

[characters] The brackets must contain one or more characters; the whole bracketed expression matches exactly one character out of the set. So [abc]matches one a, one b, or one c; it does not match zero characters, and it does not match a character other than these three.

^ Anchors your search at the beginning of the line. The expression ^The matches The when it appears at the beginning of a line; there can’t be spaces or other text before The. If you want to allow spaces, you can permit 0 or more space characters like this: ^ *The.

$ Anchors at the end of the line. end$ requires the text end to be at the end of the line, with no intervening spaces or text.

[^characters] This reverses the sense of a bracketed character list. So [^abc] matches any single character, except a, b, or c.

[character-character] You can include ranges in a bracketed character list. To match any lowercase letter, use [a-z]. You can have more than one range; so to match the first three or last three letters of the alphabet, try [a-cx-z]. To get any letter, any case, try [a-zA-Z]. You can mix ranges with single characters and with the ^metacharacter; for example, [^a-zBZ]means “anything except a lowercase letter, capital B, or capital Z.”

() You can use parentheses to group parts of the regular expression, just as you do in a mathematical expression.

|| means “or.” You can use it to provide a series of alternative expressions. Usually you want to put the alternatives in parentheses, like this: c(ad|ab|at)matches cad or cab or cat. Without the parentheses, it would match cad or ab or at instead

\ Escapes any special characters; if you want to find a literal *, you type *. The slash means to ignore *’s usual special meaning.

Here are some more examples to help you get a feel for things:

c.pe matches cope, cape, caper.

c\ .pe matches c.pe, c.per.

sto*p matches stp, stop, stoop.

car.*n matches carton, cartoon, carmen.

xyz.* matches xyz and anything after it; some tools, like egrep, only match until the end of the line.

^The matches The at the beginning of a line.

atime$ matches atime at the end of a line.

^Only$ matches a line that consists solely of the word Only—no spaces, no other characters, nothing. Only Only is allowed.

b[aou]rn matches barn, born, burn.

Ver[D-F] matches VerD, VerE, VerF.

Ver[^0-9] matches Ver followed by any non-digit.

the[ir][re] matches their, therr, there, theie.

[A-Za-z][A-Za-z]* matches any word which consists of only letters, and at least one letter. It will not match numbers or spaces.

13.2 Advanced Files

Now that you have a basic understanding of files, it is time to learn more advanced things about them.

13.2.1 The Real Nature of Files: Hard Links and Inodes

Each file on your system is represented by an inode (for Information Node; pronounced “eye-node”). An inode contains all the information about the file. However, the inode is not directly visible. Instead, each inode is linked into the filesystem by one or more hard links. Hard links contain the name of the file and the inode number. The inode contains the file itself, i.e., the location of the information being stored on disk, its access permissions, the file type, and so on. The system can find any inode if it has the inode number.

A single file can have more than one hard link. What this means is that multiple filenames refer to the same file (that is, they are associated with the same inode number). However, you can’t make hard links across filesystems: All hard references to a particular file (inode) must be on the same filesystem. This is because each filesystem has its own set of inodes, and there can be duplicate inode numbers on different filesystems.

Because all hard links to a given inode refer to the same file, you can make changes to the file, referring to it by one name, and then see those changes when referring to it by a different name. Try this:

cd; echo "hello" > firstlink

cd to your home directory and create a file called firstlink containing the word “hello.” What you’ve actually done is redirect the output of echo (echo just echoes back what you give to it), placing the output in firstlink. See the chapter on shells for a full explanation.

cat firstlink

Confirms the contents of firstlink.

ln firstlink secondlink

Creates a hard link: secondlink now points to the same inode as firstlink.

cat secondlink

Confirms that secondlink is the same as firstlink.

ls -l

Notice that the number of hard links listed for firstlink and secondlinkfiles!inodes is 2.

echo "change" >> secondlink

This is another shell redirection trick (don’t worry about the details). You’ve appended the word “change” to secondlink. Confirm this with cat secondlink.

cat firstlink

firstlink also has the word “change” appended! That’s because firstlink and secondlink refer to the same file. It doesn’t matter what you call it when you change it.

chmod a+rwx firstlink

Changes permissions on firstlink. Enter the command ls -l to confirm that permissions on secondlink were also changed. This means that permissions information is stored in the inode, not in links.

rm firstlink

Deletes this link. This is a subtlety of rm. It really removes links, not files. Now type ls -l and notice that secondlink is still there. Also notice that the number of hard links for secondlink has been reduced to one.

rm secondlink

Deletes the other link. When there are no more links to a file, Linux deletes the file itself, that is, its inode.

All files work like this—even special types of files such as devices (e.g. /dev/hda).

A directory is simply a list of filenames and inode numbers, that is, a list of hard links. When you create a hard link, you’re just adding a name-number pair to a directory. When you delete a file, you’re just removing a hard link from a directory.

13.2.2 Types of Files

One detail we’ve been concealing up to now is that the Linux kernel considers nearly everything to be a file. That includes directories and devices: They’re just special kinds of files.

As you may remember, the first character of an ls -l display represents the type of the file. For an ordinary file, this will be simply -. Other possibilities include the following:

ddirectory lsymbolic link bblock device ccharacter device pnamed pipe ssocket

Symbolic Links

Symbolic links (also called “symlinks” or “soft links”) are the other kind of link besides hard links. A symlink is a special file that “points to” a hard link on any mounted filesystem. When you try to read the contents of a symlink, it gives the contents of the file it’s pointing to rather than the contents of the symlink itself. Because directories, devices, and other symlinks are types of files, you can point a symlink at any of those things.

So a hard link is a filename and an inode number. A file is really an inode: a location on disk, file type, permissions mode, etc. A symlink is an inode that contains the name of a hard link. A symlink pairs one filename with a second filename, whereas a hard link pairs a filename with an inode number.

All hard links to the same file have equal status. That is, one is as good as another; if you perform any operation on one, it’s just the same as performing that operation on any of the others. This is because the hard links all refer to the same inode. Operations on symlinks, on the other hand, sometimes affect the symlink’s own inode (the one containing the name of a hard link) and sometimes affect the hard link being pointed to.

There are a number of important differences between symlinks and hard links.

Symlinks can cross filesystems. This is because they contain complete filenames, starting with the root directory, and all complete filenames are unique. Because hard links point to inode numbers, and inode numbers are unique only within a single filesystem, they would be ambiguous if the filesystem wasn’t known.

You can make symlinks to directories, but you can’t make hard links to them. Each directory has hard links—its listing in its parent directory, its . entry, and the .. entry in each of its subdirectories—but to impose order on the filesystem, no other hard links to directories are allowed. Consequently, the number of files in a directory is equal to the number of hard links to that directory minus two (you subtract the directory’s name and the . link). comparing!hard links and symlinks You can only make a hard link to a file that exists, because there must be an inode number to refer to. However, you can make a symlink to any filename, whether or not there actually is such a filename.

Removing a symlink removes only the link. It has no effect on the linked-to file. Removing the only hard link to a file removes the file.

Try this:

cd; ln -s /tmp/me MyTmp

cd to your home directory. ln with the -s option makes a symbolic link - in this case, one called MyTmp that points to the filename /tmp/me.

ls -l MyTmp

Output should look like this:

lrwxrwxrwx 1 havoc havoc 7 Dec 6 12:50 MyTmp -> /tmp/me

The date and user/group names will be different for you, of course. Notice that the file type is l, indicating that this is a symbolic link. Also notice the permissions: Symbolic links always have these permissions. If you attempt to chmod a symlink, you’ll actually change the permissions on the file being pointed to.

chmod 700 MyTmp

You will get a No such file or directory error, because the file /tmp/me doesn’t exist. Notice that you could create a symlink to it anyway.

mkdir /tmp/me

Creates the directory /tmp/me.

chmod 700 MyTmp

Should work now.

touch MyTmp/myfile

Creates a file in MyTmp.

ls /tmp/me

The file is actually created in /tmp/me.

rm MyTmp

Removes the symbolic link. Notice that this removes the link, not what it points to. Thus you use rm not rmdir.

rm /tmp/me/myfile; rmdir /tmp/me

Lets you clean up after yourself. symlinks!removing

Device Files

Device files refer to physical or virtual devices on your system, such as your hard disk, video card, screen, and keyboard. An example of a virtual device is the console, represented by /dev/console.

There are two kinds of devices:character and block. Character devices can be accessed one character at a time. Remember the smallest unit of data that can be written to or read from the device is a character (byte).

Block devices must be accessed in larger units called blocks, which contain a number of characters. Your hard disk is a block device.

You can read and write device files just as you can from other kinds of files, though the file may well contain some strange incomprehensible-to-humans gibberish. Writing random data to these files is probably a bad idea. Sometimes it’s useful, though. For example, you can dump a postscript file into the printer device /dev/lp0 or send modem commands to the device file for the appropriate serial port.

/dev/null

/dev/null is a special device file that discards anything you write to it. If you don’t want something, throw it in /dev/null. It’s essentially a bottomless pit. If you read /dev/null, you’ll get an end-of-file (EOF) character immediately. /dev/zero is similar, except that you read from it you get the \0 character (not the same as the number zero).

Named Pipes (FIFOs)

A named pipe is a file that acts like a pipe. You put something into the file, and it comes out the other end. Thus it’s called a FIFO, or First-In-First-Out, because the first thing you put in the pipe is the first thing to come out the other end.

If you write to a named pipe, the process that is writing to the pipe doesn’t terminate until the information being written is read from the pipe. If you read from a named pipe, the reading process waits until there’s something to read before terminating. The size of the pipe is always zero: It doesn’t store data, it just links two processes like the shell |. However, because this pipe has a name, the two processes don’t have to be on the same command line or even be run by the same user.

You can try it by doing the following:

cd; mkfifo mypipe

Makes the pipe.

echo "hello" > mypipe &

Puts a process in the background that tries to write “hello” to the pipe. Notice that the process doesn’t return from the background; it is waiting for someone to read from the pipe.

cat mypipe

At this point, the echo process should return, because cat read from the pipe, and the cat process will print hello.

rm mypipe

You can delete pipes just like any other file.

Sockets

Sockets are similar to pipes, only they work over the network. This is how your computer does networking. You may have heard of “WinSock,” which is sockets for Windows.

We won’t go into these further because you probably won’t have occasion to use them unless you’re programming. However, if you see a file marked with type son your computer, you know what it is.

13.2.3 The proc Filesystem

The Linux kernel makes a special filesystem available, which is mounted under /proc on Debian systems. This is a “pseudo-filesystem” because it doesn’t really exist on any of your physical devices.

The proc filesystem contains information about the system and running processes. Some of the “files” in /proc are reasonably understandable to humans (try typing cat /proc/meminfo or cat /proc/cpuinfo); others are arcane collections of numbers. Often, system utilities use these to gather information and present it to you in a more understandable way.

People frequently panic when they notice one file in particular— /proc/kcore —which is generally huge. This is (more or less) a copy of the contents of your computer’s memory. It’s used to debug the kernel. It doesn’t actually exist anywhere, so don’t worry about its size.

If you want to know about all the things in /proc, type man 5 proc.

13.2.4 Large-Scale Copying

Sometimes you may want to copy one directory to another location. Maybe you’re adding a new hard disk and you want to copy /usr/local to it. There are several ways you can do this.

The first is to use cp. The command cp -a will tell cp to do a copy preserving all the information it can. So, you might use

cp -a /usr/local /destination

However, there are some things that cp -a won’t catch[1]. So, the best way to do a large copy job is to chain two tar commands together, like so:

[1] Sparse files and hard links are two examples.

tar -cSpf - /usr/local | tar -xvSpf - -C /destination

The first tar command will archive the existing directory and pipe it to the second. The second command will unpack the archive into the location you specify with -C.

13.3 Security

Back in section 7.1 on page [*], we discussed file permissions in Linux. This is a fundamental way to keep your system secure. If you are running a multi-user system or a server, it is important to make sure that permissions are correct. A good rule of thumb is to set files to have the minimum permissions necessary for use.

If you are running a network server, there are some other things to be aware of as well. First, you ought to uninstall or turn off any network services you’re not using. A good place to start is the file /etc/inetd.conf; you can probably disable some of these. For most network services, it’s also possible to control who has access to them; the /etc/hosts.allow and /etc/hosts.deny files (documented in man 5 hosts_access) can control who has access to which services. You also ought to keep up-to-date with patches or updates to Debian; these can be found on your nearest Debian FTP mirror.

Some other commonsense rules apply:

◼ Never tell anyone your password.

◼ Never send your password in cleartext across the Internet by using something like telnet or FTP. Instead, use encrypted protocols or avoid logging in remotely.

◼ Avoid using root as much as possible.

◼ Don’t install untrusted software, and don’t install it as root.

◼ Avoid making things world-writable whenever possible. /tmp is one exception to this rule.

While this is probably not of as much use to somebody not running a server, it is still pays to know a bit about security. Debian’s security mechanism is what protects your system from many viruses.

13.4 Software Development with Debian

Debian makes a great platform for software development and programming. Among the languages and near-languages it supports are: C, C++, Objective-C, Perl, Python, m4, Ada, Pascal, Java, awk, Tcl/Tk, SQL, assembler, Bourne shell, csh, and more. Writing programs is beyond the scope of this book, but here are some of the more popular development programs in Debian:

gcc The GNU C Compiler, a modern optimizing C compiler.

g++ The C++ compiler from the gcc line.

cpp The C preprocessor from gcc.

perl The Perl interpreter. Perl is a great “glue” language.

gdb GNU Debugger, used to debug programs in many different languages.

gprof Used for profiling, this program helps you to find ways to improve the performance of your programs.

emacs GNU Emacs is a programmers’ editor and IDE.

as The GNU Assembler.

About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books.

This book is part of the public domain. John Goerzen and Ossama Othman (2004). Debian GNU/Linux : Guide to Installation and Usage. Urbana, Illinois: Project Gutenberg. Retrieved https://www.gutenberg.org/cache/epub/6527/pg6527-images.html

This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.org, located at https://www.gutenberg.org/policy/license.html.