menuimage

Description

Usage

Example

Options

Notes

Kudos

License

Changelog

Download

Donation

My_Other_Sites

My_Programs

Comments

lv-convert2cache v0.1.3 [12 Nov 2024] by Dominic

Description

TL;DR: lv-convert2cache.sh supercharges a Logical Volume.

lv-convert2cache.sh is a bash wrapper script for LVM functions which convert an existing LVM logical volume (LV) into a read/write 'cache LV'. A typical scenario is that an existing LV (in an existing volume group (VG)) is currently stored on one (or more) physical volume(s) (PV(s)) which are on one (or more) slower devices (e.g. spinning disks), and we want to make this LV faster for read and write by (optionally creating and) using a new PV on a smaller but faster device (e.g. NVMe SSD) as a 'cache pool'. For more info see 'man 7 lvmcache'.

After some sanity tests, lv-convert2cache.sh asks you (once) to confirm before any action is taken.

When lv-convert2cache.sh has completed, the new cache LV will have the same name as, and be usable as, the original LV. There will also be two hidden LVs - the data LV on the original slower drive (named as the original LV with '_corig' suffixed) and a cache pool LV on the faster device (named as the original LV with '_CacheVol_cvol' suffixed). Internally the cache pool LV will hold both the cached data and the metadata for managing the cache. All LVs can be seen with: lvs --all --options +devices

lv-convert2cache.sh has been tested under LVM 2.03.11(2) (2021-01-08).

Usage

lv-convert2cache.sh [options] existing_VG/existing_LV PV_for_cache ["further_options_for_lvconvert_(at_cache_creation)"]

Example

# chunksize parameter may be needed with a large PV_for_cache (e.g. 200GB)
lv-convert2cache.sh timedicer/home /dev/sda1 "--chunksize 512"

Options

-b use writeback cachemode setting (faster writes, but risk of data loss)
-h see this help
-l see changelog
-n do not show program title
-v merely show existing cache LVs, if any

Notes

The device (typically a drive partition) to be used for the new PV (PV_for_cache i.e. the cache pool) must exist, but need not already be a PV. It should be much faster than the PV(s) which hold the existing_LV (or the conversion is pointless), and its contents (if any) must be expendable - they will be lost.

I have not seen any guidance on the recommended size ratio between the cache pool device (PV_for_cache) and the underlying data LV (existing_LV). Most examples use small cache pools (e.g. 10GB), maybe because they were written when larger SSDs were expensive. I have used a 200GB cache pool (for a 2TB source LV) without problems, except having to specify "--chunksize 512" when creating it.

- Of 2 possible cache methods, lv-convert2cache.sh uses 'dm-cache' (alternative 'dm-writecache' only speeds up writing not reading)
- Of 2 possible cache options, lv-convert2cache.sh uses 'cachevol' (the cached copies of data blocks and the metadata for managing the cache are both stored in the same (hidden) LV on the same (fast) device; the more complex alternative option 'cachepool' allows having cache data & metadata on different (LVs and) devices [which could make the cached LV operate marginally faster])
- Unsurprisingly, to make changes lv-convert2cache.sh must be run by superuser
- I made this utility to help me and you, but you use it at your own risk

Normally lv-convert2cache.sh uses the default and recommended 'writethrough' cachemode setting, but this means that caching gives no speed gain when writing, and may even be slower. Option '-b' (untested at time of writing [22 Dec 2023]) forces use of cachemode 'writeback' - this should bring write speed gains but at the risk of data loss if the cache pool LV (cachevol) is corrupted or lost.

Note for versions of LVM earlier than 2.03.12 it is not possible to resize logical volumes of cache type, including those created by lv-convert2cache.sh. To workaround this limitation, split and then rejoin the data and cache LVs (but you will lose the cache data which will have to be rebuilt over time):

lvs -a -o +cache_mode,cache_settings,cache_policy,chunk_size # show LVs
# example: VG 'timedicer', cache LV 'home', non-standard chunksize '512'
lvconvert --splitcache timedicer/home # split cache LV
lvs -a -o +cache_mode,cache_settings,cache_policy # show LVs
lvextend -L+100G -r timedicer/home # extend LV and underlying FS
# restore cache LV
lvconvert --type cache --cachevol timedicer/home_CacheVol timedicer/home --chunksize 512
lvs -a -o +cache_mode,cache_settings,cache_policy,chunk_size # show LVs

With the above lvconvert command it is possible to specify --cachemode writeback (instead of the default --cachemode writethrough) for faster writes, however as of LVM tools 2.03.11(2) this is regarded as unsafe and you will have to ignore/override warnings saying 'repairing a damaged cachevol is not yet possible' and 'cache mode writethrough is suggested for safe operation'. lv-convert2cache.sh (except with option '-b') sticks with the default setting.

Another approach which works equally well but may be a little slower (because --uncache is slower than --splitcache) involves removing the cache pool LV, something like this:

lvconvert --uncache timedicer/home # may take some time
lvextend -L+100G -r timedicer/home # extend LV and the underlying FS
lv-convert2cache.sh timedicer/home /dev/sda1 "--chunksize 512"


If you lose the device on which your cache pool is stored, you will have problems, especially because you will also have lost the metadata for the whole LV (so you will not be able to access the underlying LV even though all your data should still be there). See the first listed page at Kudos below; if this is insufficient, recovery should be possible by using a backup of LVM metadata to recreate the PV, VG and cache pool LV - for guidance try: https://www.golinuxcloud.com/recover-lvm2-partition-restore-vg-pv-metadata/#How_to_manually_delete_LVM_metadata_in_Linux or, in case it has disappeared, then as pdf at: https://www.timedicer.co.uk/programs/helpinfo/Recover_LVM2_partition_PV_VG_LVM_metadata.pdf

Kudos

The following pages were helpful in devising this utility and writing the help text -
https://blog.delouw.ch/2020/01/29/using-lvm-cache-for-storage-tiering/
https://manpages.ubuntu.com/manpages/focal/man7/lvmcache.7.html
https://docs.redhat.com/fr/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_logical_volumes/enabling-caching-to-improve-logical-volume-performance_configuring-and-managing-logical-volumes
https://www.stderr.nl/Blog/Software/Linux/LvmResizeCachedLv.html

License

Copyright © 2024 Dominic Raferd. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Changelog

0.1.3 [12 Nov 2024] - Updated link in kudos section
0.1.2 [20 Dec 2023] - Add -b option (enable writeback cachemode) (untested)
0.1.1 [19 Dec 2023] - Help text changes: add info re backup pdf page for data recovery + re splitting cached LV
0.1.0 [14 Dec 2022] - Help text changes, and add hints on what to do if cache pool device goes missing
0.0.6 [16 Nov 2022] - Help text clarifications, bugfix -v option, add -n option
0.0.5 [08 Apr 2022] - Help text clarifications
0.0.4 [06 Apr 2022] - Add help about how to resize/extend a cache type LV, add -v option
0.0.3 [01 Apr 2022] - Help text clarifications
0.0.2 [11 Feb 2022] - Help text clarifications
0.0.1 [02 Feb 2022] - Help text changes, bug fix, check if existing LV is already cached
0.0.0 [11 Jan 2022] - Initial version

Download lv-convert2cache.sh

Donation

I have provided this software free gratis and for nothing. If you would like to thank me with a contribution, please let me know and I will send you a link. Thank you!

My Other Sites

My Programs

Here is a selection of some (other) programs I have written, most of which run under GNU/Linux from the command line (CLI), are freely available and can be obtained by clicking on the links. Dependencies are shown and while in most cases written and tested on an x86-based Linux server, they should run on a Raspberry Pi, and many can run under Windows using Windows Subsystem for Linux (WSL) or Cygwin. Email me if you have problems or questions, or if you think I could help with a programming requirement.

Backup Utilities

Debian/Ubuntu kernel and LVM Utilities

Miscellaneous Programs

Comments

This section is closed. If you have a question, please submit it by email, thank you.

No comments yet for '/programs/help/lv-convert2cache.sh.php'