Troubleshooting FortiSOAR™

Troubleshooting FortiSOAR™ Issues

This section covers issues that you might face while deploying FortiSOAR™, upgrading FortiSOAR™, or while using FortiSOAR™.

Issues occurring in FortiSOAR™ due to insufficient space

You can face deployment or upgrade failures due to insufficient space. If you have limited partition size for /dev/mapper/vgapp-csapps (earlier /dev/mapper/cybersponse-cs--apps), then FortiSOAR™ upgrade might fail. Therefore, before you upgrade your FortiSOAR™ system, you must ensure that you increase the partition size to a minimum of 4 GB for /dev/mapper/vgapp-csapps to prevent potential loss of backups.

You might also experience any of the following symptoms when the disk space of the database on which FortiSOAR™ is running gets full:

  • Users are unable to log into FortiSOAR™.
  • All FortiSOAR™ services might stop working, as they cannot write to their respective log files.
    For example, the PostgreSQL service fails to start when the PostgreSQL database disk is full.

Insufficient space in FortiSOAR™ can be caused due to a number of reasons, some of them are as follows:

  • Increase in the number of log files in /var/log/ and /var/log/cyops
  • /home drive is full
  • Increase in the data in the database

You can fix this issue using the following methods:

  • Extend the disk space by adding a new disk and then extending the logical volume in the new disk.
  • Extend the logical volume by using the free space that is already available in the volume group.
  • Extend the logical volume on the existing disk without adding a new disk, if sufficient unallocated space is available on the existing disk.

Resolution 1

Perform the following steps to extend your disk space by adding a new disk and then extending the logical volume (LVM) in the new disk:

  1. Stop all FortiSOAR™ services using the following command:
    # csadm services --stop
  2. Add the new disk drive with the required size on the instance.
  3. Run the following command to check the size of the newly added and unpartitioned disk.
    # lsblk
    This command displays the size of newly added and unpartitioned disk. In this example, sde is the newly added disk:
    NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
    sda 8:0 0 8G 0 disk
    '-sda1 8:1 0 8G 0 part /
    sdb 8:16 0 100G 0 disk
    '-sdb1 8:17 0 100G 0 part
    sdc 8:32 0 300G 0 disk
    '-sdc1 8:33 0 300G 0 part
    '-vg_repo-lvol0 253:0 0 500G 0 lvm /repos
    sdd 8:48 0 200G 0 disk
    '-vg_repo-lvol0 253:0 0 500G 0 lvm /repos
    sde 8:64 0 200G 0 disk #This is the new attached partition
    sr0 11:0 1 1024M 0 rom
    Note: In case the disk is not reflected in the VM, you can run the following command:
    ## echo "- - -" > /sys/class/scsi_host/<host#>/scan
    (Replace host# with the correct host number)
    Then run the # lsblk command again to verify the newly added disk.
    If even after running the above command the newly added disk is NOT visible under lsblk, then reboot the appliance using reboot command.
  4. Run the following command to create the physical volume by specifying the name for the newly added disk:
    # pvcreate /dev/<disk_name>
  5. Run the following command to check the name of the volume group:
    # vgs
    The column VG corresponds to the volume group name (cyops) as seen in the following sample output:
    VG #PV #LV #SN Attr VSize VFree
    cyops 1 13 0 wz--n- <243.65g 8.00g
  6. Run the following command with the volume group name (from VG column in step 5) and disk_name as specified in step 4, to extend the volume group size:
    # vgextend <lvm_group_name> /dev/<disk_name>
  7. Run the following command to check the size of the extended volume group:
    # vgs
    The column VFree corresponds to the volume group size (20.00g) as seen in the following sample output:
    VG #PV #LV #SN Attr VSize VFree
    cyops 1 13 0 wz--n- <243.65g 20.00g
  8. Run the following command to extend the logical volume by the size you specify:
    # lvextend -L +<disk_size>G <LVM_name>
    The following example shows that the logical volume is being extended by 18GB:
    lvextend -L +18G /dev/mapper/<LVM_name>
    Note: You must extend the disk size to less than or equal to the total size of the volume group.
  9. Run the following command to resize the file system for the disk that has been extended in step 8:
    # xfs_growfs /dev/mapper/<LVM_name>
  10. Run the following command to check if the volume size is extended:
    # df -h /dev/mapper/<LVM_name>
  11. Start all FortiSOAR™ services using the following command:
    # csadm services --start

Resolution 2

Perform the following steps to extend the logical volume by using the free space that is already available in the volume group:

  1. Stop all FortiSOAR™ services using the following command:
    # csadm services --stop
  2. Run the following command to check the free size available in volume group:
    # vgs
    The column VFree corresponds to the volume group size (20.00g) as seen in the following sample output:
    VG #PV #LV #SN Attr VSize VFree
    cyops 1 13 0 wz--n- <243.65g 20.00g
  3. Run the following command to extend the logical volume by the size you specify:
    # lvextend -L +<disk_size>G <LVM_name>
    The following example shows that the logical volume is being extended by 18GB:
    lvextend -L +18G /dev/mapper/<LVM_name>
    Note: You must extend the disk size to less than or equal to the total size of the volume group.
  4. Run the following command to resize the file system for the disk that has been extended in step 3:
    # xfs_growfs /dev/mapper/<LVM_name>
  5. Run the following command to check if the volume size is extended:
    # df -h /dev/mapper/<LVM_name>
  6. Start all FortiSOAR™ services using the following command:
    # csadm services --start

Resolution 3

Perform the following steps to extend the logical volume on the existing disk without adding a new disk, if sufficient unallocated space is available on the existing disk. Before proceeding further, it is recommended that you find out whether sufficient unallocated space is available on existing disk using the following command:
# parted /dev/<disk_name> print free

For example, running the # parted /dev/sdb print free command will display the following:

Model: VMware Virtual disk (scsi)
Disk /dev/sdb: 215GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number  Start   End     Size    Type     File system  Flags

        32.3kB  1049kB  1016kB  Free Space
 1      1049kB  161GB   161GB   primary               lvm
        161GB   215GB   53.7GB  Free Space

The “Free Space” in above the output refers to unallocated space. If the mentioned size is sufficient, then perform the following steps:

  1. Stop all FortiSOAR™ services using the following command:
    # csadm services --stop
  2. Create a partition on the existing disk using the following command:
    # fdisk /dev/<disk_name>
  3. Enter n to create a new partition.
  4. Enter p to choose the newly created partition as the primary partition.
  5. Select the Partition number. The operating system will calculate this on its own, you just require to press Enter.
  6. Select defaults for the First Cylinder/Sector and Last Cylinder/Sector values.
  7. Enter w to write the changes to the partition table.
  8. Restart the VM.
  9. After the VM restarts, stop all FortiSOAR™ services using the following command:
    # csadm services --stop
  10. Run the following command to see the new partition:
    # fdisk -l /dev/<disk_name>
    This command displays the properties of the newly attached partition. For this example, /dev/sda3 is the newly added partition.
  11. Create a physical volume in the new partition using the following command:
    # pvcreate
    For example, # pvcreate /dev/sda3
  12. Extend the volume group (cyops) using the following command:
    # vgextend
    For example, # vgextend cyops /dev/sda3
  13. Extend the logical volume by the size you specify using the following command:
    # lvextend -L+<size>G <LVM_name>
    For example, to extend the logical volume by 20 G use the following command:
    lvextend -L+20G /dev/mapper/cyops-relations
  14. Increase the file system size using the following command:
    # xfs_growfs
    For example, # xfs_growfs /dev/mapper/cyops-relations
  15. Start all FortiSOAR™ services using the following command:
    # csadm services --start

Troubleshooting Deployment Issues

The FortiSOAR™ Virtual Appliance deployment on ESX is failing

Resolution:

  1. Verify that FortiSOAR™ Virtual Appliance file that you have downloaded is not corrupted by running a # md5sum command for the FortiSOAR™ Virtual Appliance.
  2. Check that the ESX server has fulfilled all prerequisites specified for the VM. Refer to the Planning section for details.
  3. If both points 1 and 2 are ok, contact VMWare support.

Cannot access the FortiSOAR™ portal

Resolution:

  1. Check the ESX network to which FortiSOAR™ VM is connected.
  2. Check if the IP address is assigned to your FortiSOAR™ VM, in the case of DHCP or static IP addresses.
    Refer to the Editing the VM configuration section for more information on Setting a static IP and Determining your DHCP IP address.

Cannot login to the FortiSOAR™ platform

Resolution:

  1. Check if you are using the correct credentials that have been provided to you by FortiSOAR™ Customer Support.

  2. ssh to the VM where you have deployed FortiSOAR™ to check the status of cyops-auth service. The cyops-auth service must be running.

  3. If both points 1 and 2 are ok, and the cyops-auth service is running, contact FortiSOAR™ support.

Getting a 502 error when you click on the Reports tab

Resolution:

  1. ssh to the VM where you have installed FortiSOAR™.
  2. Log in using the ssh credentials.
  3. Run the sudo su command.
    Enter your FortiSOAR™ password.
  4. Run the $ sudo systemctl restart tomcat command.
  5. Run the $ sudo systemctl restart nginx command.

If the issue yet does not get resolved, contact FortiSOAR™ support.

Troubleshooting Upgrade Issues

Post license renewal you cannot log into FortiSOAR™

If you have requested for a license from FortiSOAR™ with lesser number of users than your existing users, you cannot log onto FortiSOAR™ post upgrade.

For example, when you had requested a license from FortiSOAR™, you had requested for a license for 10 users; however, you have 15 users existing in your system, you will not be able to log onto FortiSOAR™ after upgrading to versions 4.10 and later.

So, it is very important for you to provide the correct number of users while requesting for a license from FortiSOAR™.

Resolution:

Contact your FortiSOAR™ Customer Success (CS) representative to generate a new license for you with the correct number of users.

Failure to upgrade FortiSOAR™

In case you face a failure while trying to upgrade FortiSOAR™, then perform the following steps:

Resolution:

  1. To gather logs and send them to your FortiSOAR™ Customer Success (CS) representative do the following:
    1. ssh to machine as a root user and type the following command:
      # csadm log --collect
    2. Specify the path where you want to collect the logs. By default, the logs are collected in the /tmp/ folder.
      A file named cyops-logs.tar.gz.gpg gets generated in the path you have specified. Send this file to your CS representative.
  2. Revert the snapshot of your system to the latest working state. You must take a snapshot of your system before you attempt to upgrade FortiSOAR™ on your system.
    The process for taking a snapshot and reverting to the current snapshot is detailed in the Take a snapshot on the vSphere Web Client and Revert to a snapshot on the vSphere Web Client articles present on the Fortinet Support Site. You must log onto the support site to view the information.

Post-upgrade your playbooks fail to execute, and the playbooks are also not listed in the executed playbooks log

If you have not cleaned the workflow history prior to upgrading your FortiSOAR™ version, and if you have a large number of records in the workflow history (> 50000), then the overall upgrade time would increase, and this issue might occur. For the method to clean the workflow execution run history, see the "Setting up auto-cleanup of workflow execution history" topic in the Debugging and Optimizing Playbooks chapter in the "Playbooks Guide."

Resolution

Perform the following steps:

  1. Check the install logs to see if errors such as the following are present:
    psycopg2.InternalError: missing chunk number 0 for toast value 1502832 in pg_toast_17046
  2. Connect to the postgres db and run the following command:
    # REINDEX table pg_toast.pg_toast_XXXXX;
    (where XXXXX is the number from the error message present in step 1).
  3. Run the following commands:
    # cd /opt/cyops-workflow/sealab
    # sudo -u nginx /opt/cyops-workflow/.env/bin/python3 manage.py migrate
    Note: If manage.py fails again for pg_toast value (error mentioned in step 1), then you must execute steps 2 and 3 again with the new pg_toast value mentioned in the error, till the manage.py executes successfully.

Login and logout events are not audited after you have upgraded your FortiSOAR™ version

After you have upgraded FortiSOAR™ on your system, you observe the following error in the auditlog log file located at /var/log/cyops/cyops-gateway/auditlog.log :

ERROR c.c.a.service.RecordLogService.processRecordLogs - 500 Internal Server Error, and you also do not see any Login and Logout events in Audit Logs on FortiSOAR™ UI, then perform the steps mentioned in the resolution.

Resolution

To resolve this issue and include login and login events in audit logs, run the following commands on your FortiSOAR™ VM:

yes | cp /opt/cyops-workflow/sealab/.envdir/APPLIANCE_PUBLIC_KEY /etc/cyops/APPLIANCE_PUBLIC_KEY
yes | cp /opt/cyops-workflow/sealab/.envdir/APPLIANCE_PRIVATE_KEY /etc/cyops/APPLIANCE_PRIVATE_KEY
chmod -R 644 /etc/cyops/APPLIANCE_*_KEY
chown  tomcat:tomcat /etc/cyops/APPLIANCE_*_KEY
sudo systemctl restart tomcat