linux-utils: Linux system administration tools for Python

https://travis-ci.org/xolox/python-linux-utils.svg?branch=master https://coveralls.io/repos/xolox/python-linux-utils/badge.svg?branch=master

The Python package linux-utils provides utility functions that make it easy to script system administration tasks on Linux systems in Python. The following functionality is currently implemented:

The package is currently tested on cPython 2.6, 2.7, 3.4, 3.5, 3.6 and PyPy (2.7) on Ubuntu Linux (using Travis CI).

Installation

The linux-utils package is available on PyPI which means installation should be as simple as:

$ pip install linux-utils

There’s actually a multitude of ways to install Python packages (e.g. the per user site-packages directory, virtual environments or just installing system wide) and I have no intention of getting into that discussion here, so if this intimidates you then read up on your options before returning to these instructions ;-).

Usage

For details about the Python API please refer to the API documentation available on Read the Docs.

The Python implementation of cryptdisks_start and cryptdisks_stop is available on the command line as two programs:

  • cryptdisks-start-fallback
  • cryptdisks-stop-fallback

As the names imply these programs are not functional equivalents of their “official” counterparts, because they only support LUKS encryption and a small subset of the available encryption options.

History

Back in 2015 I wrote some Python code to parse the Linux configuration files /etc/fstab and /etc/crypttab for use in crypto-drive-manager. Fast forward to 2017 and I found myself wanting to use the same functionality in rsync-system-backup. Three options presented themselves to me:

  1. Copy/paste the relevant code. Having to maintain the same code in multiple places causes lower quality code because having to duplicate the effort of writing documentation, developing tests and fixing bugs is a very demotivating endeavor.

    In fact sometime in 2016 I did copy/paste parts of this code into a project at work, because I needed similar functionality there. Of course since then the two implementations have started diverging :-p.

  2. Make crypto-drive-manager a dependency of rsync-system-backup. Although this approach is less ugly than copy/pasting the code, it still isn’t exactly elegant because the two projects have nothing to do with each other apart from working with LUKS encrypted disks on Linux.

  3. Extract the functionality into a new package. In my opinion this is clearly the most elegant approach, unfortunately it also requires the most work from me :-). On the plus side I’m publishing the new package with a test suite which means less untested code remains in crypto-drive-manager (which doesn’t have a test suite at the time of writing).

While extracting the code I shortly considered integrating the functionality into debuntu-tools, however the /etc/fstab and /etc/crypttab parsing isn’t specific to Debian or Ubuntu at all and debuntu-tools has several dependencies that aren’t relevant to Linux configuration file parsing.

Since then it has become clear that this was a good choice (not merging the functionality into debuntu-tools) because the package now provides a Python implementation of cryptdisks_start and cryptdisks_stop, which is mostly useful on Linux systems that aren’t based on Debian :-).

Contact

The latest version of linux-utils is available on PyPI and GitHub. The documentation is hosted on Read the Docs. For bug reports please create an issue on GitHub. If you have questions, suggestions, etc. feel free to send me an e-mail at peter@peterodding.com.

License

This software is licensed under the MIT license.

© 2017 Peter Odding.

API documentation

The following documentation is based on the source code of version 0.5 of the linux-utils package.

Available modules

linux_utils

Linux system administration tools for Python.

linux_utils.coerce_context(value)[source]

Coerce a value to an execution context.

Parameters:value – The value to coerce (an execution context created by executor.contexts or None).
Returns:An execution context created by executor.contexts.
Raises:ValueError when value isn’t None but also isn’t a valid execution context.
linux_utils.coerce_device_file(expression)[source]

Coerce a device identifier to a device file.

Parameters:expression – The device identifier (a string).
Returns:The pathname of the device file (a string).
Raises:ValueError when an unsupported device identifier is encountered.

If you pass in a LABEL="..." or UUID=... expression (as found in e.g. /etc/fstab) you will get back a pathname starting with /dev/disk/by-label or /dev/disk/by-uuid:

>>> from linux_utils import coerce_device_file
>>> print(coerce_device_file('LABEL="Linux Boot"'))
/dev/disk/by-label/Linux\x20Boot
>>> print(coerce_device_file('UUID=7801a1c2-7ad7-4c0b-9fbb-2a47ae802f71'))
/dev/disk/by-uuid/7801a1c2-7ad7-4c0b-9fbb-2a47ae802f71

If expression is already a pathname it will pass through untouched:

>>> coerce_device_file('/dev/mapper/backups')
'/dev/mapper/backups'

Unsupported device identifiers raise an exception:

>>> coerce_device_file('PARTUUID=e6c021cc-d0d8-400c-8f5c-b10adeff65fe')
Traceback (most recent call last):
  File "linux_utils/__init__.py", line 90, in coerce_device_file
    raise ValueError(msg % expression)
ValueError: Unsupported device identifier! ('PARTUUID=e6c021cc-d0d8-400c-8f5c-b10adeff65fe')
linux_utils.coerce_size(value)[source]

Coerce a human readable data size to the number of bytes.

Parameters:value – The value to coerce (a number or string).
Returns:The number of bytes (a number).
Raises:ValueError when value isn’t a number or a string supported by parse_size().

linux_utils.atomic

Atomic filesystem operations for Linux in Python.

The most useful functions in this module are make_dirs(), touch(), write_contents() and write_file().

The copy_stat() and get_temporary_file() functions were originally part of the logic in write_file() but have since been extracted to improve the readability and reusability of the code.

linux_utils.atomic.copy_stat(filename, reference=None, mode=None, uid=None, gid=None)[source]

The Python equivalent of chmod --reference && chown --reference.

Parameters:
  • filename – The pathname of the file whose permissions and ownership should be modified (a string).
  • reference – The pathname of the file to use as reference (a string or None).
  • mode – The permissions to set when reference isn’t given or doesn’t exist (a number or None).
  • uid – The user id to set when reference isn’t given or doesn’t exist (a number or None).
  • gid – The group id to set when reference isn’t given or doesn’t exist (a number or None).
linux_utils.atomic.get_temporary_file(filename)[source]

Generate a non-obtrusive temporary filename.

Parameters:filename – The filename on which the name of the temporary file should be based (a string).
Returns:The filename of a temporary file (a string).

This function tries to generate the most non-obtrusive temporary filenames:

  1. The temporary file will be located in the same directory as the file to replace, because this is the only location somewhat guaranteed to support “rename into place” semantics (see write_file()).
  2. The temporary file will be hidden from directory listings and common filename patterns because it has a leading dot.
  3. The temporary file will have a different extension then the file to replace (in case of filename patterns that do match dotfiles).
  4. The temporary filename has a decent chance of not conflicting with temporary filenames generated by concurrent processes.
linux_utils.atomic.make_dirs(directory, mode=511)[source]

Create a directory if it doesn’t already exist (keeping concurrency in mind).

Parameters:directory – The pathname of a directory (a string).
Returns:True if the directory was created, False if it already existed.
Raises:Any exceptions raised by os.makedirs().

This function is a wrapper for os.makedirs() that swallows OSError in the case of EEXIST.

linux_utils.atomic.touch(filename)[source]

The equivalent of the touch program in Python.

Parameters:filename – The pathname of the file to touch (a string).

This function uses make_dirs() to automatically create missing directory components in filename.

linux_utils.atomic.write_contents(filename, contents, encoding='UTF-8', mode=None)[source]

Atomically create or update a file’s contents.

Parameters:
  • filename – The pathname of the file (a string).
  • contents – The (new) contents of the file (a byte string or a Unicode string).
  • encoding – The text encoding used to encode contents when it is a Unicode string.
  • mode – The permissions to use when the file doesn’t exist yet (a number like accepted by os.chmod() or None).
linux_utils.atomic.write_file(*args, **kwds)[source]

Atomically create or update a file (avoiding partial reads).

Parameters:
  • filename – The pathname of the file (a string).
  • mode – The permissions to use when the file doesn’t exist yet (a number like accepted by os.chmod() or None).
Returns:

A writable file object whose contents will be used to create or atomically replace filename.

linux_utils.cli

Command line interface for cryptdisks_start() and cryptdisks_stop().

linux_utils.cli.cryptdisks_start_cli()[source]

Usage: cryptdisks-start-fallback NAME

Reads /etc/crypttab and unlocks the encrypted filesystem with the given NAME.

This program emulates the functionality of Debian’s cryptdisks_start program, but it only supports LUKS encryption and a small subset of the available encryption options.

linux_utils.cli.cryptdisks_stop_cli()[source]

Usage: cryptdisks-stop-fallback NAME

Reads /etc/crypttab and locks the encrypted filesystem with the given NAME.

This program emulates the functionality of Debian’s cryptdisks_stop program, but it only supports LUKS encryption and a small subset of the available encryption options.

linux_utils.crypttab

Parsing of /etc/crypttab configuration files.

The cryptsetup program is used to manage LUKS based full disk encryption and Debian provides some niceties around cryptsetup to make it easier to use, specifically:

  • The /etc/crypttab configuration file contains static information about encrypted filesystems and enables unlocking of encrypted filesystems when the system is booted. Refer to the crypttab man page for more information.
  • The cryptdisks_start and cryptdisks_stop commands can be used to manually unlock encrypted filesystems that are configured in /etc/crypttab with the noauto option (meaning the device is ignored during boot).
linux_utils.crypttab.parse_crypttab(filename='/etc/crypttab', context=None)[source]

Parse the Debian Linux configuration file /etc/crypttab.

Parameters:
  • filename – The absolute pathname of the file to parse (a string, defaults to /etc/crypttab).
  • context – An execution context created by executor.contexts (coerced using coerce_context()).
Returns:

A generator of EncryptedFileSystemEntry objects.

Here’s an example:

>>> from linux_utils.crypttab import parse_crypttab
>>> print(next(parse_crypttab()))
EncryptedFileSystemEntry(
    configuration_file='/etc/crypttab',
    line_number=3,
    target='ssd',
    source='UUID=31678141-3931-4683-a4d2-09eadec81d01',
    source_device='/dev/disk/by-uuid/31678141-3931-4683-a4d2-09eadec81d01',
    key_file='none',
    options=['luks', 'discard'],
)
class linux_utils.crypttab.EncryptedFileSystemEntry(**kw)[source]

An entry parsed from /etc/crypttab.

Each entry in the crypttab file has four fields, these are mapped to the following properties:

  1. target
  2. source
  3. key_file
  4. options

Refer to the crypttab man page for more information about these fields. The computed properties is_available, is_unlocked and source_device are based on the parsed values of the four fields above.

is_available

True if source_device exists, False otherwise.

is_unlocked

True if target_device exists, False otherwise.

key_file

The file to use as a key for decrypting the data of the source device (a string or None).

When the key file field in /etc/crypttab is set to none the value of this property will be None, this makes checking whether an encrypted filesystem has a key file configured much more Pythonic.

options[source]

The encryption options for the filesystem (a list of strings).

Note

The options property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

source

The block special device or file that contains the encrypted data (a string).

The value of this property may be a UUID=... expression instead of the pathname of a block special device or file.

source_device[source]

The block special device or file that contains the encrypted data (a string).

The value of this property is computed by passing source to coerce_device_file().

Note

The source_device property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

target

The mapped device name (a string).

target_device[source]

The absolute pathname of the device file corresponding to target (a string).

Note

The target_device property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

linux_utils.fstab

Parsing of /etc/fstab configuration files.

linux_utils.fstab.find_mounted_filesystems(filename='/proc/mounts', context=None)[source]

Get information about mounted filesystem from /proc/mounts.

Parameters:
  • filename – The absolute pathname of the file to parse (a string, defaults to /proc/mounts).
  • context – An execution context created by executor.contexts (coerced using coerce_context()).
Returns:

A generator of FileSystemEntry objects.

This function is a trivial wrapper for parse_fstab() that instructs it to parse /proc/mounts instead of /etc/fstab. Here’s an example:

>>> from humanfriendly import format_table
>>> from linux_utils.fstab import find_mounted_filesystems
>>> print(format_table(
...    data=[
...        (entry.mount_point, entry.device_file, entry.vfs_type)
...        for entry in find_mounted_filesystems()
...        if entry.vfs_type not in (
...            # While writing this example I was actually surprised to
...            # see how many `virtual filesystems' a modern Linux system
...            # has mounted by default (based on Ubuntu 16.04).
...            'autofs', 'cgroup', 'debugfs', 'devpts', 'devtmpfs', 'efivarfs',
...            'fuse.gvfsd-fuse', 'fusectl', 'hugetlbfs', 'mqueue', 'proc',
...            'pstore', 'securityfs', 'sysfs', 'tmpfs',
...        )
...    ],
...    column_names=["Mount point", "Device", "Type"],
... ))
---------------------------------------------------
| Mount point  | Device                    | Type |
---------------------------------------------------
| /            | /dev/mapper/internal-root | ext4 |
| /boot        | /dev/sda5                 | ext4 |
| /boot/efi    | /dev/sda1                 | vfat |
| /mnt/backups | /dev/mapper/backups       | ext4 |
---------------------------------------------------
linux_utils.fstab.parse_fstab(filename='/etc/fstab', context=None)[source]

Parse the Linux configuration file /etc/fstab.

Parameters:
Returns:

A generator of FileSystemEntry objects.

Here’s an example:

>>> from linux_utils.fstab import parse_fstab
>>> next(e for e in parse_fstab() if e.mount_point == '/')
FileSystemEntry(
    configuration_file='/etc/fstab',
    line_number=8,
    device_file='UUID=7801a1c2-7ad7-4c0b-9fbb-2a47ae802f71',
    mount_point='/',
    vfs_type='ext4',
    options=['errors=remount-ro'],
    dump_frequency=0,
    check_order=1,
)
class linux_utils.fstab.FileSystemEntry(**kw)[source]

An entry parsed from /etc/fstab.

Each entry in the fstab file has six fields, these are mapped to the following properties:

  1. device
  2. mount_point
  3. vfs_type
  4. options
  5. dump_frequency
  6. check_order

Refer to the fstab man page for more information about the meaning of each of these fields. The values of the following properties are computed based on the six fields above:

check_order[source]

The order in which the filesystem should be checked at boot time (an integer number, defaults to 0).

Note

The check_order property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

device

The block special device or remote filesystem to be mounted (a string).

The value of this property may be a UUID=... expression.

device_file[source]

The block special device to be mounted (a string).

The value of this property is computed by passing device to coerce_device_file().

Note

The device_file property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

dump_frequency[source]

The dump frequency for the filesystem (an integer number, defaults to 0).

Note

The dump_frequency property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

mount_point[source]

The mount point for the filesystem (a string).

Each occurrence of the escape sequence \040 is replaced by a space.

Note

The mount_point property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

nfs_directory[source]

The directory on the NFS server (a string or None).

When vfs_type is nfs or nfs4 and device is of the form <server>:<directory> the value of nfs_directory will be the part after the colon (:).

Note

The nfs_directory property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

nfs_server[source]

The host name or IP address of the NFS server (a string or None).

When vfs_type is nfs or nfs4 and device is of the form <server>:<directory> the value of nfs_server will be the part before the colon (:).

Note

The nfs_server property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

options[source]

The mount options for the filesystem (a list of strings).

Note

The options property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

vfs_type

The type of filesystem (a string like ‘ext4’ or ‘xfs’).

linux_utils.luks

Python API for cryptsetup to control LUKS full disk encryption.

The functions in this module serve two distinct purposes:

Low level Python API for cryptsetup

The following functions and class provide a low level Python API for the basic functionality of cryptsetup:

This functionality make it easier for me to write test suites for Python projects involving full disk encryption, for example crypto-drive-manager and rsync-system-backup.

Python implementation of cryptdisks_start and cryptdisks_stop

The command line programs cryptdisks_start and cryptdisks_stop are easy to use wrappers for cryptsetup that parse /etc/crypttab to find the information they need.

The nice thing about /etc/crypttab is that it provides a central place to configure the names of encrypted filesystems, so that you can refer to a symbolic name instead of having to constantly repeat all of the necessary information (the target name, source device, key file and encryption options).

A not so nice thing about cryptdisks_start and cryptdisks_stop is that these programs (and the whole /etc/crypttab convention) appear to be specific to the Debian ecosystem.

The functions cryptdisks_start() and cryptdisks_stop() emulate the behavior of the command line programs when needed so that Linux distributions that don’t offer these programs can still be supported by projects like crypto-drive-manager and rsync-system-backup.

linux_utils.luks.DEFAULT_KEY_SIZE = 2048

The default size (in bytes) of key files generated by generate_key_file() (a number).

linux_utils.luks.create_image_file(filename, size, context=None)[source]

Create an image file filled with bytes containing zero (\0).

Parameters:
Raises:

ValueError when size is invalid, ExternalCommandFailed when the command fails.

linux_utils.luks.generate_key_file(filename, size=2048, context=None)[source]

Generate a file with random contents that can be used as a key file.

Parameters:
Raises:

ExternalCommandFailed when the command fails.

linux_utils.luks.create_encrypted_filesystem(device_file, key_file=None, context=None)[source]

Create an encrypted LUKS filesystem.

Parameters:
  • device_file – The pathname of the block special device or file (a string).
  • key_file – The pathname of the key file used to encrypt the filesystem (a string or None).
  • context – An execution context created by executor.contexts (coerced using coerce_context()).
Raises:

ExternalCommandFailed when the command fails.

If no key_file is given the operator is prompted to choose a password.

linux_utils.luks.unlock_filesystem(device_file, target, key_file=None, options=None, context=None)[source]

Unlock an encrypted LUKS filesystem.

Parameters:
  • device_file – The pathname of the block special device or file (a string).
  • target – The mapped device name (a string).
  • key_file – The pathname of the key file used to encrypt the filesystem (a string or None).
  • options – An iterable of strings with encryption options or None (in which case the default options are used). Currently ‘discard’, ‘readonly’ and ‘tries’ are the only supported options (other options are silently ignored).
  • context – An execution context created by executor.contexts (coerced using coerce_context()).
Raises:

ExternalCommandFailed when the command fails.

If no key_file is given the operator is prompted to enter a password.

linux_utils.luks.lock_filesystem(target, context=None)[source]

Lock a currently unlocked LUKS filesystem.

Parameters:
Raises:

ExternalCommandFailed when the command fails.

linux_utils.luks.cryptdisks_start(target, context=None)[source]

Execute cryptdisks_start or emulate its functionality.

Parameters:
Raises:

ExternalCommandFailed when a command fails, ValueError when no entry in /etc/crypttab matches target.

linux_utils.luks.cryptdisks_stop(target, context=None)[source]

Execute cryptdisks_stop or emulate its functionality.

Parameters:
Raises:

ExternalCommandFailed when a command fails, ValueError when no entry in /etc/crypttab matches target.

class linux_utils.luks.TemporaryKeyFile(filename, size=2048, context=None)[source]

Context manager that makes it easier to work with temporary key files.

__init__(filename, size=2048, context=None)[source]

Initialize a TemporaryKeyFile object.

Refer to generate_key_file() for details about argument handling.

__enter__()[source]

Generate the temporary key file.

__exit__(exc_type=None, exc_value=None, traceback=None)[source]

Delete the temporary key file.

linux_utils.tabfile

Generic parsing of Linux configuration files like /etc/fstab and /etc/crypttab.

linux_utils.tabfile.parse_tab_file(filename, context=None, encoding='UTF-8')[source]

Parse a Linux configuration file like /etc/fstab or /etc/crypttab.

Parameters:
  • filename – The absolute pathname of the file to parse (a string).
  • context – An execution context created by executor.contexts (coerced using coerce_context()).
  • encoding – The name of the text encoding of the file (a string).
Returns:

A generator of TabFileEntry objects.

This function strips comments (the character # until the end of the line) and splits each line into tokens separated by whitespace.

class linux_utils.tabfile.TabFileEntry(**kw)[source]

Container for the results of parse_tab_file().

context[source]

The execution context from which the configuration file was retrieved.

Note

The context property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

configuration_file[source]

The name of the configuration file from which this entry was parsed (a string).

Note

The configuration_file property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

line_number[source]

The line number from which this entry was parsed (an integer).

Note

The line_number property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

tokens[source]

The tokens split on whitespace (a nonempty list of strings).

Note

The tokens property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().