System administrators need a solid background in the fundamentals of operating systems. Unfortunately, good defaults and better tools mask the need for that until something breaks. On top of that there is very little recognition of the theory behind how operating systems work and its use in practical, day-to-day system administration.
This morning my team was faced with a puzzling problem. A file system which had 34% free space was no longer allowing new files to be created in it, but pre-existing files were still being appended to? How does that work? The junior admin who was the first responder was puzzled. Could it be a permission problem? Maybe a reboot will fix it…
Inodes are one of those things a system administrator might go his whole life not worrying about. More seasoned UNIX admins will remember playing tricks with the number of inodes to squeeze more space from a small disk, not to mention the whole hard link thing. It is somewhat depressing to have to explain hard links, but in the last year I’ve done it at least twice. The gargantuan, inexpensive disks available now make hard links somewhat irrelevant, and it seems that symbolic links just make more sense to people. I agree, for what it’s worth. The symbolic link’s arrow says precisely what’s going on.
Operating systems also have much better defaults now, so creating inodes just happens magically. Heck, even the default output from ‘df’ does not have inode counts, at least in Linux. There just isn’t anything to force the issue with new system administrators, so they don’t even know they exist until they are forced to. And the forcing is usually under pressure.
File system concepts appear first at the “Junior System Administrator” level of the SAGE “Job Descriptions for System Administrators.” It lists as a required skill:
Fundamental understanding of an operating system; for example, understands job control, soft and hard links or shortcuts, distinctions between the kernel and the user environment.
I was fortunate that my educational experience took me through a computer science program that teaches a solid operating systems course at the undergraduate level. Many of my colleagues have picked up system administration out of other careers, or through degree programs that do not require, or have, an operating systems course. Looking at MIT, for example, their course that teaches operating system fundamentals is 6.828, which is graduate-level.
Looking through the SAGE “Recommended Reading” list I see nothing that strikes me as providing any fundamental knowledge of operating systems. One book is described as “provides a clear, concise, practical guide to the real-world issues that anyone responsible for a Unix system faces daily.” To move from junior system administrator to senior system administrator requires less practical and more theory, though. Especially when it comes to such “practical” exercises like tuning:
A solid understanding of an operating system; understands paging and swapping, inter-process communications, devices and what device drivers do, filesystem concepts (inode, clustering, logical partitions), can use performance analysis to tune systems.
Where does a system administrator get the knowledge to do these things?
I do not have an answer for this, though I’ve been searching all day for a good guide to these things. All I know is that the best system administrators know this stuff, and they know it because they have to. I know that I have a book, Operating System Concepts, on a shelf behind me that I read when I was a novice system administrator. I still use that when I need a good way to explain what is stored in an inode, how process scheduling algorithms work, or a good description of semaphores. If I had to identify one book that has been instrumental to my work as a system administrator it would be that one. I have never seen a book like it in a recommended reading list for system administrators. It needs to be.
This morning, it took a senior system administrator to run “df -i” and fix a problem. Concurrently, one junior admin saw the mystery of inodes revealed to him. I just wish he, and so many others, would have known it to begin with.
[tags]lopsa, sage, sysadmin[/tags]