Linux


How to backup Linux VM’s with large data disks (>4 TB)

Recently I ran into an issue on a project I’m working on. The customer has a Linux virtual machine running on Azure with a large data disk (20 TB). I knew – but forgot to remember – that Azure Backup doesn’t support disks larger then 4 TB (more info here). The specific drive is used for logging, so for a moment I thought that Azure files could be a solution but the specific Linux version (RHEL 6.7) isn’t supported for secure transportation towards Azure files. So I found another solution.

I decided to add multiple drives to this virtual machine (and split the needed size by the number of drives). In my example I added 3 disks to this virtual machine.

largedisk1.png

Now logon to the CLI of that specific virtual machine. The 3 datadisks were made available using /dev/sdc, /dev/sdd and /dev/sde.

First we have to create physical volumes on top of /sdc, /sdd and /sde using the following command :

pvcreate /dev/sdc /dev/sdd /dev/sde

You can check this using the following command :

pvs

of for detailed information:

pvdisplay /dev/sdc

Now we are going to create a volume group named logging using the 3 physical volumes with this command :

vgcreate logging /dev/sdc /dev/sdd /dev/sde

Now we create a logical volume using the following command:

lvcreate -n logs -l 100%FREE logging

Now let’s format the volume

mkfs.ext4 /dev/logging/logs default 0 0

Now we have to edit the /etc/fstab file. In my (demo) case I add the following line:

/dev/logging/logs /var/logging ext4 defaults 0 0

My fstab file looks als follows:

largedisk2

After rebooting the new volume is available on /var/logging (in my demo case)

As you can see there is one 9 TB disk (in my demo) which I can access :

largedisk3

Now we are able to use Azure Backup to backup this machine:

largedisk4

Thanks to BM for the feedback! #TheManWithTheSleeve


How to add a data disk to your Azure Linux VM the right way 2   Recently updated !

In this blogpost I shall describe how you add an extra data disk to your Linux VM running on Azure.

Step 1 Add a new disk to your Linux VM using the Azure Portal

2019-06-08 07_55_11-Microsoft Edge.png

Choose Add data disk to add an extra data disk and choose create data disk.

2019-06-08 07_57_12-Microsoft Edge.png

For demo purposes I quickly entered a default 20 GiB HD, nothing fancy 🙂

Don’t forget to save your changes!

2019-06-08 07_58_27-Microsoft Edge.png

Step 2 Connect to your VM using SSH or use the Serial console on your VM Pane in the Azure Portal

2019-06-08 07_53_04-Microsoft Edge.png

Use the following command to find all your data drives :

You see all your drives and the newly created drive

2019-06-08 08_00_53-Microsoft Edge.png

Here, sdc is the newly added disk. Let’s continue.

Now we have to partition the added disk using the following command :

Use the n command to add a new partition. In this example, we also choose p for a primary partition and accept the rest of the default values. The output will be similar to the following example:

2019-06-08 08_04_33-Microsoft Edge.png

Now we are going to write a partition (format) the newly added disk using the following command :

2019-06-08 08_06_03-Microsoft Edge.png

Now we are going to mount the formatted drive using the following commands :

2019-06-08 08_07_45-Microsoft Edge.png

You see your newly created disk :

2019-06-08 08_08_55-Microsoft Edge.png

To ensure that the drive is remounted automatically after a reboot, it must be added to the /etc/fstab file. Herefore I’m going to use the blkid utility:

2019-06-08 08_10_11-VMLIN01 - Serial console - Microsoft Azure ‎- Microsoft Edge.png

Now copy the UUID :

2019-06-08 08_11_15-VMLIN01 - Serial console - Microsoft Azure ‎- Microsoft Edge.png

Now we are going to add the UUID to the /etc/fstab. You can use VI or (like I prefer to use) nano.

The format is als follows :

UUID=<YourUUID> /<YourMountPoint> ext4 defaults,nofail 1 2

In my case it looks as follows :

2019-06-08 08_13_41-Microsoft Edge.png

Ok, let’s reboot the VM and check if the drive still exists..

2019-06-08 08_15_22-Microsoft Edge.png

Use the following command to check

As you can see the /MyDataDrive is still available after reboot.


How to fix PFSense libssl.so.8 not found error(s) 2

Recently I ran into a problem with PFSense. After rebooting I noticed several errors during the boot proces.

pfsense_error.PNG

I was able to boot into the single user environment and after some investigating I found a solution that worked for me.

After rebooting I noticed that there were some (file) lockes and I had to update the kernel (otherwise PFSense keeps on booting in a loop) using the 13 option in the menu (type exit to go to the menu)

After this I had to reboot the PFSense and after spinning up I was able to open the Web interface and found out that everything was working again.


How to set the right timezone in Ubuntu

Did you notice that your timezone was not correct? This is how to configure the right timezone :

You can check your date/time settings :

The timedatectl is installed by default, but when you did a minimal install you can fix this with the following command :

Tip : You can check out this website to verify your timezone : https://www.timeanddate.com/worldclock/netherlands/amsterdam


Howto fix “dpkg was interrupted, you must manually run sudo dpkg –configure -a to correct the problem” error

When you install a package using dpkg and somehow you kill the proces, you may encouter the above error. There is an easy solution for that :

Remove the lock files using the following command :

 


Howto install (multicore) PAR2 on Ubuntu

Use the following command to install PAR2 support on your Linux host :

If your device has more than 1 core available, you can significantly speed up verification and repair by using all cores in your system. Installing these multicore versions of Par2 is easy for many platforms :

 

 

 


How to configure PFsense to PXE boot using FOGProject

For testing purposes I’m using PFsense to allow my clients to PXE boot using the FOG TFTP server.

This are the settings which are working :

PropertyValue
Next serverIP Address of FOG Server
Default BIOS file nameundionly.kpxe
UEFI 32 bit file nameipxe32.efi
UEFI 64 bit file nameipxe.efi

 

You can find these settings in the PFsense DHCP service menu.


Installing Plesk on CentOS Linux

In this post I’m going to describe the simple steps to install the latest release of the Plesk software on CentOS.

First make sure you are root:

Make sure you start with a clean starting point, remove existing instances of Apache, MySql, Cyrus and PHP:

Make sure your CentOS distribution is fully patched:

Let’s disable the Security Enhanced Linux feature:

Go to the installation folder:

Download and execute the installer script:

Finally configure the Plesk configuration using an internet browser : https://ip address of server:8443


How to troubleshoot your Linux VM running on Microsoft Azure

Many people are running Linux in a virtual machine on Azure. But what if a Linux virtual machine refuses to start?

Go to the Azure portal and open the virtual machine properties. First check out the CPU, network and disk utilization. Is CPU constantly peaking at 100%? Then you know that you must investigate that first. You see absolutely no utilization at all? Then your virtual machine might be down or doing nothing at all. When your virtual machine is slowly but online, maybe you have choosen the wrong virtual machine type and do you require more resources.

Ok… let’s choose the troubleshoot option. (The screendumps are from the dutch Azure website)

When you choose the troubleshoot option, you see the current resource status. A green sign means that there should be no problems with the Azure platform resources you are running on. In my case I see a green sign, so that’s a good! You also see the latest issues and activities. Did someone recently restart your virtual machine? You should see a notice of that. Remember how important it is to take security in mind. Are you and your co-workers all using the same account? Then it can be difficult to identify who rebooted the server.

You also see most common issues regarding your type of virtual machine. Just click on a problem and Microsoft gives you advice. You directly have the option to check for the tips that Microsoft gives you.

Console session

Most system administrators first instinct is to check the console screen. Unfortunately there is no live console screen which you can use. So you can’t monitor the boot process (and see the errors occurring) realtime. But there are ways to monitor it with a alternative method. Let’s go to the first option and click the first link:

After you’ve selected the first option you notice the follow screen:

You notice the latest boot process. You can scroll down this window. Notice the options to download the logfile, and to take a screendump and download it. You can’t see a live screen of the console but you’re able to download a screendump of the console. Not ideally but it can provide you with some interesting info.

Reset password

Sometimes there is a problem with your password.  Maybe you forgot your password!? You can use CLI or Powershell to change it.  You can find more info here and here. When you have full access to azure and the virtual machine you can reset your root password without knowing the current password.

Check for a pending reboot

Maybe some actions required a reboot and for that reason some services are not running. Check if the file /var/run/reboot-required exists or not. If it exists then you first have to reboot your Linux virtual machine before further troubleshooting.

Restart your virtual machine

There could be a resource problem or a hanging process. Choose to restart your virtual machine. Click on restart virtual machine to restart it. Use the console and boot information mentioned earlier to check the progress.

Reset the SSH connection creds

Sometimes there could be an issue with your SSH keys. Choose this option to recreate your SSH keys. (Option 4)

Migrate your Virtual machine to another host

You have the option the migrate (move) your virtual machine to another host. Sometimes there could be a problem with a specific region or host Use this option to make sure that this doesn’t apply to you.

Consider the use of premium storage

Check your number of IO’s. Do you have a application which requires a lot of IO? Consider the use of premium storage. Microsoft Azure Premium Storage delivers high-performance, low-latency disk support for virtual machines running I/O-intensive workloads. VM disks that use Premium Storage store data on solid state drives. You can migrate your application’s VM disk to Azure Premium Storage to take advantage of the speed and performance of these disks. But be aware of the costs! If your disks does not require high IOPS, you can limit costs by maintaining it in Standard Storage, which stores virtual machine disk data on Hard Disk Drives insteads of SSD’s. More info here.

Revert or fallback to your latest snapshot/backup

Sometimes it’s easier not to troubleshoot but to restore your latest backup and/or snapshot. Especially if you have a working (and tested!) backup and are able to restore

Conclusion

Microsoft provides more and more support for Linux virtual machines. The not real time console session is a bummer but Microsoft offers a lot of tips for you to take a clooser look at. I hope that this post will provide you with a good place to start your investigation. Make sure you have a working (and tested!) back-up plan in order. Everyone needs a restore or one point or another. 🙂 Microsoft also provides support plans, costs are $ 250 monthly with a minimum term of 6 months. You can always fallback on Microsoft’s Linux team which has advanced knowledge but for a price..


File level restore on Azure

Making backups of virtual machines running on Azure using snapshot technology is a nice feature. But sometimes you don’t want to revert the whole snapshot but only want to restore a single file. Now this is possible. It uses the same backup/Snapshot technology you probably are already using.

Azure_file_level_restore_1

Open the virtual machine properties in the all resources tab. Choose the Back-up option.

Azure_file_level_restore_2

Go to the file level restore option. (more/upper right)

Azure_file_level_restore_3

Select the back-up set containing the file(s) you wish to restore. Then choose to download the script. Upload that script to your Virtual Machine. (winscp, copy/paste into nano/vi or any other way you choose to). It takes approx. 1 minute to generate and download the script.

Execute the script using bash <filename.sh>. First time the VM adds support for the iSCSI service which is required for mounting the back-upset. Choose Y for installation the iSCSI drivers and wait a few seconds. You see that entire back-upset is mounted. Now you can copy all the necessary files you need.

After you are ready, go to the Azure portal and choose to unmount the back-upset. Now you are all finished!