Overview
Administrators can use various logs that FortiSOAR™ generates to troubleshoot FortiSOAR™ issues. This chapter lists the key FortiSOAR™ services and processes and also provides some troubleshooting tips. This chapter also provides some additional configuration settings so that you can tune the results that get displayed by FortiSOAR™ for record similarity and field prediction. For more information on record similarity and field prediction, see the Working with Modules - Alerts & Incidents chapter in the "User Guide."
If you face any issues while deploying or upgrading FortiSOAR™, see the Troubleshooting FortiSOAR™ chapter in the "Deployment Guide." If you face deployment or upgrade failures due to insufficient space, or if you face issues while using FortiSOAR™ that might be caused due to insufficient space, like you are unable to log into FortiSOAR™ or FortiSOAR™ services stop working, then see the Issues occurring in FortiSOAR™ due to insufficient space
section in the Troubleshooting FortiSOAR™ chapter in the "Deployment Guide."
List of logs used for troubleshooting FortiSOAR™
FortiSOAR™ log files are stored in the following location: /var/log/cyops
. You will find the following directories in the /var/log/cyops
location:
Log Name | Purpose |
---|---|
cyops-api/ssl_cyops_api_access.log | Used for troubleshooting web (nginx) UI or API access issues. |
cyops-api/ssl_cyops_api_error.log | Used for troubleshooting API errors. |
cyops-api/prod.log | Used for troubleshooting FortiSOAR™ PHP related issues. |
cyops-auth/das.log | Used for troubleshooting FortiSOAR™ authentication issues. |
cyops-gateway/auditlog.log | Used for troubleshooting FortiSOAR™ audit log issues. |
cyops-gateway/etl.log | Used for troubleshooting FortiSOAR™ application and system-level issues. |
cyops-gateway/saml.log | Used for troubleshooting FortiSOAR™ SAML issues. |
cyops-search/falcon.log | Used for troubleshooting FortiSOAR™ Search issues. If you get any error when you are indexing or searching for a record in FortiSOAR™, you can use the falcon.log file to troubleshoot ElasticSearch issues. This log is also used for checking the status of ElasticSearch indexing. |
cyops-gateway/gateway.log | Used for troubleshooting FortiSOAR™ Gateway issues such as, audit log page failing to load or the SSO configuration page failing to load. |
cyops-notifier/notifier.log | Used for troubleshooting FortiSOAR™ Web Socket issues. |
cyops-workflow/beat.log | Used for troubleshooting issues of the FortiSOAR™ Scheduler. |
cyops-workflow/celeryd.log | Used for troubleshooting FortiSOAR™ playbook runtime issues. |
cyops-workflow/sealab.log | Used for troubleshooting FortiSOAR™ playbook framework issues. |
cyops-workflow/ssl_cyops_workflow_access.log | Used for troubleshooting playbook access issues. |
cyops-workflow/ssl_cyops_workflow_error.log | Used for troubleshooting playbook errors. |
cyops-workflow/uwsgi.log | Used for troubleshooting FortiSOAR™ playbook and connector issues. |
cyops-integrations/connectors.log | Used for troubleshooting FortiSOAR™ connector related issues. |
cyops-integrations/integrations.log | Used for troubleshooting FortiSOAR™ connector framework issues. |
cyops-integrations/integrations/imap/listener.log | Used for troubleshooting the IMAP connector issues. |
cyops-integrations/ssl_cyops_integrations_access.log | Used for troubleshooting connector access issues. |
cyops-integrations/ssl_cyops_integrations_error.log | Used for troubleshooting connector errors. |
install For example, 5.0.0-855.log . | Used for troubleshooting FortiSOAR™ installation issues. Install logs are named according to the FortiSOAR™ version and build number. |
install/connectors.log | Used for troubleshooting connector installation issues. |
upgrade_cyops_<version_number>-<timestamp>.log For example, upgrade_cyops_5.0.0-2019-05-23-1558604373.log | Stores upgrade console log and you can use it to troubleshoot FortiSOAR™ upgrade issues. |
install/config-vm-<timestamp>.log For example, install/config-vm-27_Nov_2018_18h_14m_34s.log | Used for troubleshooting FortiSOAR Configuration Wizard issues. |
For troubleshooting FortiSOAR™ audit log issues use the tomcat.log
located at cyops/tomcat.log
.
For Centos OS level errors, use the Messages
logs located at /var/log/messages
.
Logging Levels
You can set the following logging levels in the log files:
- DEBUG: Low-level system information for debugging purposes.
- INFO: General system information.
- WARNING: Information describing a minor problem that has occurred.
- ERROR: Information describing a major problem that has occurred.
- CRITICAL: Information describing a critical problem that has occurred.
Changing the logging levels
- For sealab or workflow:
- Open the
/opt/cyops-workflow/sealab/sealab/config.ini
file and set theWORKFLOW_LOG_LEVEL
parameter to the required logging level. For example,WORKFLOW_LOG_LEVEL = 'INFO'
- Restart the
uwsgi
service.
- Open the
- For integrations:
- Open the
/opt/cyops-integrations/integrations/configs/config.ini
file and set theconnector_logger_level
parameter to the required logging level. For example,connector_logger_level= 'INFO'
- Restart the
uwsgi
service.
- Open the
- For celeryd:
- Open the
/etc/celery/celeryd.conf
file and set theCELERYD_LOG_LEVEL
parameter to the required logging level. For example,CELERYD_LOG_LEVEL = 'INFO'
- Restart the
celeryd
service.
- Open the
- For nginx (UI), API, or php:
- Open the
/opt/cyops-api/app/config/config_prod.yml
file and set thelevel
parameter to the required logging level. For example,level = 'INFO'
- Run the
# systemctl restart php-fpm nginx
command.
- Open the
List of key FortiSOAR™ services and processes
Name of Services/Processes | Description |
---|---|
redis | Caching service used by the playbook engine. To know the status of this service, use the # systemctl status redis command. |
postgresql-12 | Service for all application data stored in postgresql DB. To know the status of this service, use the # systemctl status postgresql-12 command. |
elasticsearch | Service to bring up the elasticsearch service. To know the status of this service, use the # systemctl status elasticsearch command. |
php-fpm | Service for PHP FastCGI implementation. To know the status of this service, use the # systemctl status php-fpm command. |
uwsgi | Software application that aims at developing a full stack for building hosting services. uWSGI is named after the Web Server Gateway Interface. We host our playbook execution engine application and connector integrations applications on a uWSGI server. To know the status of uwsgi use the # systemctl status uwsgi command. |
celeryd | celeryd is used to run the playbooks asynchronously in the FortiSOAR™ playbook execution engine. To know the status of celeryd use the # systemctl status celeryd command. |
celerybeatd | celerybeatd is a playbook scheduler; used to kick off tasks at regular intervals, that are then executed by available worker nodes in the cluster. To know the status of use the # systemctl status celerybeatd command. |
cyops-auth | Service used for FortiSOAR™ authentications. To know the status of this services, use the # systemctl status cyops-auth command. |
cyops-tomcat | Service used for SSO, auditing, and websocket. To know the status of this services, use the # systemctl status tomcat command. |
cyops-search | Service used for full-text searching, finding similar records and predicting the value of fields based on similarity. |
cyops-ha | Service is responsible for setting up High Availability in the FortiSOAR™ environment |
cyops-postman | Service is responsible for setting up and managing FortiSOAR™'s distributed multi-tenant setup. |
rabbitmq-server | Service is responsible to send audit and live sync notifications. This service is also responsible for data transfer in a distributed environment. |
nginx | Service used for Web UI. To know the status of this services, use the # systemctl status nginx command. |
If you want to restart, start, or stop all the services together, use FortiSOAR™ Admin CLI (csadm
). For more information on csadm
, see the FortiSOAR™ Admin CLI chapter in the "Administration Guide."
You can run the csadm
command on any FortiSOAR™ machine using any terminal. Any user who has root
or sudo
permissions can run the csadm
command.
To restart FortiSOAR™ services, type: # csadm services --restart
To start FortiSOAR™ services, type: # csadm services --start
To stop FortiSOAR™ services, type: # csadm services --stop
To know the status of all FortiSOAR™ services type: # csadm services --status
To view the status of individual FortiSOAR™ services use the # systemctl status <service_name>
command. For example, to see the status of the nginx
service, use the # systemctl status nginx
command.
When you run # csadm services --status
command the status of FortiSOAR™ services are displayed with a background color so that you can quickly and easily identify which services are running and which are not running. The status of services that are running are displayed in a Green background, and the status of services that are not running are displayed in a Red background.
Following image displays how the statuses of FortiSOAR™ services are displayed when some services are running, and some are not running:
Additional settings for record similarity and field predictions
FortiSOAR™ 6.0.0 introduces "Record Similarity" i.e., FortiSOAR™ displays records that are similar to the record on which you are working. Version 6.0.0 also introduces "Record Field Value Prediction" i.e., FortiSOAR™ predicts values of fields of your choice within a record from the values of fields of existing records based on the criteria you have defined, making it easier for analysts to make informed decisions. For more information, see the Working with Modules - Alerts & Incidents chapter in the "User Guide."
This section provides information on how you can tune the results that are displayed by FortiSOAR™ for record similarity and field predictions using the following parameters in the /opt/cyops/config/cyops-search/config.yml
file:
minimum_should_match
:<percentageValue>
: This setting defines that a record will be considered similar only if there is a match of at least the percentage value that you have specified on the related fields. This is especially true for similarity based on related records. For example, if you set this parameter asminimum_should_match
:10%
(default), then if you have defined similarity for alerts based on related indicators, then FortiSOAR™ will display only those records as similar that match a minimum of 10% of the indicators. Therefore, for an alert that has 10 related indicators, FortiSOAR™ similarity results will display alerts that even have one common indicator; but if an alert has 20 related indicators, then FortiSOAR™ similarity results will display only those alerts that have at least 2 indicators in common.max_query_terms
:<numberOfItems>
: This setting defines how many terms of the parent record will be looked up for similarity in other records. Continuing the same example as above, if you set this parameter asmax_query_terms
:25
(default), then if an alert has more than 25 indicators, only 25 of them will be checked for similarity in other records. Note that increasing the value of this setting will increase the time FortiSOAR™ takes to return similarity and suggestion results.
For more information on the above parameters and other parameters, refer to the ElastiSearch reference at: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html
Troubleshooting Tips
Your Workflow data size has increased
Increase in your Workflow data can cause performance bottlenecks.
Resolution:
You can purge Executed Playbook Logs using the Purge Logs button on the top-right of the Executed Playbook Logs
dialog. For more information on purging, see the Debugging and Optimizing Playbooks chapter in the "Playbooks Guide."
Change the default value of some of the user profile parameters
An administrator with CRUD permissions on the Security
module can change the default value of the following user-profile related parameters:
Parameter Name | Description | Default Value |
---|---|---|
max_reset_attempts | Maximum number of times users' can click the Reset Password link before actually resetting their password. If the user exceeds the value set in this parameter, then users' will not get a new link to reset their password based on the number of hours specified in the reset_locktime parameter. By default, the max_reset_attempts is set to 10 times and the reset_locktime is set to 12 hours, therefore, if a user clicks the Reset Password 11 times without actually resetting their password, then the user will not get a new link to reset their password for 12 hours. | 10 times |
reset_locktime | Number of hours that users' will not get a new link to reset their password if they exceed the value set in the max_reset_attempts parameter. | 12 hours |
max_failed | Number of times that users' can enter an incorrect password, while logging into FortiSOAR™, before their account gets locked. If the user exceeds the value set in this parameter, then the user will get locked out based on the number of minutes specified in the lock_minutes parameter. By default, the max_failed is set to 5 times and the lock_minutes is set to 30 mins, therefore, if a user enters an incorrect password 6 times, then their account gets locked for 30 mins. | 5 times |
lock_minutes | Number of minutes that users' account gets locked if they exceed the value set in the max_failed parameter. | 30 mins |
To change the value of the max_reset_attempts
parameter, the administrator should run the following curl command on their FortiSOAR™ system:
curl -X PUT \ https://<CyOPs™ HOSTNAME/IP>/api/auth/config \ -H 'Authorization: <Bearer Token>' \ -d '{ "option":"max_reset_attempts", "value":5 }'
The above command changes the number of times users' can click the Reset Password link to 5 times, i.e., a user can click the Reset Password link 5 times without actually setting the new password. However, if the user clicks the Reset Password link for the 6th time, the user will be blocked.
Similarly, to change the value of the reset_locktime
, max_failed
, and lock_minutes
parameters, the administrator should run the same curl command on their FortiSOAR™ system, but after changing the option and value parameter values:
curl -X PUT \ https://<CyOPs™ HOSTNAME/IP>/api/auth/config \ -H 'Authorization: <Bearer Token>' \ -d '{ "option":"reset_locktime", "value":2 }'
The above command changes the number of hours that users' will not get a new link to reset their password to 2 hours if they have exceeded the value set in the max_reset_attempts
parameter.
curl -X PUT \ https://<CyOPs™ HOSTNAME/IP>/api/auth/config \ -H 'Authorization: <Bearer Token>' \ -d '{ "option":"max_failed", "value":3 }'
curl -X PUT \ https://<CyOPs™ HOSTNAME/IP>/api/auth/config \ -H 'Authorization: <Bearer Token>' \ -d '{ "option":"lock_minutes", "value":15 }'
max_reset_attempts
parameter. Error displayed while performing a search operation in FortiSOAR™
Resolution:
If you get any error while performing a global search in FortiSOAR™, check that the elasticsearch.service
and the cyops-search.service
are running.
If these are not running, then start these services using the following commands:
# systemctl start elasticsearch
# systemctl start cyops-search
For more information, see the FortiSOAR™ Search Errors
topic in the Elasticsearch Configuration chapter.
Reindexing FortiSOAR™ modules for search
Partial indexing of a module, or when a module does not get indexed, can lead to errors in FortiSOAR™ search. You can manually reindex any skipped or unsuccessfully indexed modules. For more information, see the FortiSOAR™ Search Errors
topic in the Elasticsearch Configuration chapter.
Resolution:
To reindex all the FortiSOAR™ modules, run the following command:
$ sudo -u nginx php /opt/cyops-api/app/console cybersponse:elastic:create --env="prod"
To reindex specific FortiSOAR™ modules, run the following command:
$ sudo -u nginx php /opt/cyops-api/app/console cybersponse:elastic:create --env="prod" --index='{"type":"type of the module(s)"}'
For example:
$ sudo -u nginx php /opt/cyops-api/app/console cybersponse:elastic:create --env="prod" --index='{"type":"indicators", "tasks"}