During the installation of Splunk on a Linux guest I encountered an error where mongod was not able to start. But first an overview of the environment:
After the startup of Splunk I could see the following error messages in the splunk logfile:
12-29-2024 17:35:04.217 +0100 ERROR SidecarThread [33800 MainThread] - reading standard input
12-29-2024 17:35:04.219 +0100 ERROR SidecarThread [33800 MainThread] - 2024/12/29 17:35:04 Supervisor logs printed at : /opt/splunk/var/log/splunk
12-29-2024 17:35:04.369 +0100 ERROR MongodRunner [34134 MongodLogThread] - mongod exited abnormally (exit code 4, status: PID 34135 killed by signal 4: Illegal instruction) - look at mongod.log to investigate.
12-29-2024 17:35:04.369 +0100 ERROR KVStoreBulletinBoardManager [34134 MongodLogThread] - KV Store process terminated abnormally (exit code 4, status PID 34135 killed by signal 4: Illegal instruction). See mongod.log and splunkd.log for details.
12-29-2024 17:35:04.369 +0100 WARN KVStoreConfigurationProvider [34134 MongodLogThread] - Action scheduled, but event loop is not ready yet
12-29-2024 17:35:04.369 +0100 ERROR KVStoreBulletinBoardManager [34134 MongodLogThread] - KV Store changed status to failed. KVStore process terminated..
The important bit is:
mongod exited abnormally (exit code 4, status: PID 34135 killed by signal 4: Illegal instruction)
Also starting mongod as root failed (core dumped). After some investigation it looked like the linux guest was missing AVX (can be checked by looking at the flags under cat /proc/cpuinfo) which is needed by mongod.
The solution was to disable Hyper-V as described here:
in Windows start a Command Prompt with Administrator privileges
Disable Hypervisor Launch: bcdedit /set hypervisorlaunchtype off
Disable Microsoft Hyper-V: DISM /Online /Disable-Feature:Microsoft-Hyper-V
The last point required a reboot of Windows. After restarting Windows and the Linux guest AVX was available and mongod started successfully:
These are the steps to install the 60 days trial version of Splunk Enterprise 9.4.0 on RedHat Enterprise Linux 8.1. I used Oracle VirtualBox as the virtualization software running on Windows 10. First we create an account on splunk.com and download the Linux rpm file via wget. As the root user we install the rpm file as follows:
rpm -i splunk-9.4.0.x86_64.rpm
[root@lin Downloads]# rpm -i splunk-9.4.0.x86_64.rpm
warning: splunk-9.4.0.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID b3cd4420: NOKEY
no need to run the pre-install check
complete
[root@lin Downloads]#
The installation created a user and a group named splunk. We change the password of the splunk user as follows:
passwd splunk << EOF
splunk
splunk
EOF
Changing password for user splunk.
New password: BAD PASSWORD: The password is shorter than 8 characters
Retype new password: passwd: all authentication tokens updated successfully.
[root@lin ~]#
Now we start splunk for the first time. Enter admin as the username and choose a password:
su - splunk
cd /opt/splunk/bin/
./splunk start --accept-license
[root@lin Downloads]# su - splunk
[splunk@lin ~]$ cd /opt/splunk/bin/
[splunk@lin bin]$ ./splunk start --accept-license
This appears to be your first time running this version of Splunk.
Splunk software must create an administrator account during startup. Otherwise, you cannot log in.
Create credentials for the administrator account.
Characters do not appear on the screen when you type in credentials.
Please enter an administrator username: admin
Password must contain at least:
* 8 total printable ASCII character(s).
Please enter a new password:
Please confirm new password:
Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.
Generating RSA private key, 2048 bit long modulus
..+++++
...........................................................+++++
e is 65537 (0x10001)
writing RSA key
Generating RSA private key, 2048 bit long modulus
.......+++++
.................................+++++
e is 65537 (0x10001)
writing RSA key
Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.
Splunk> Now with more code!
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Creating: /opt/splunk/var/lib/splunk
Creating: /opt/splunk/var/run/splunk
Creating: /opt/splunk/var/run/splunk/appserver/i18n
Creating: /opt/splunk/var/run/splunk/appserver/modules/static/css
Creating: /opt/splunk/var/run/splunk/upload
Creating: /opt/splunk/var/run/splunk/search_telemetry
Creating: /opt/splunk/var/run/splunk/search_log
Creating: /opt/splunk/var/spool/splunk
Creating: /opt/splunk/var/spool/dirmoncache
Creating: /opt/splunk/var/lib/splunk/authDb
Creating: /opt/splunk/var/lib/splunk/hashDb
Creating: /opt/splunk/var/run/splunk/collect
Creating: /opt/splunk/var/run/splunk/sessions
New certs have been generated in '/opt/splunk/etc/auth'.
Checking critical directories... Done
Checking indexes...
Validated: _audit _configtracker _dsappevent _dsclient _dsphonehome _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket history main summary
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-9.4.0-6b4ebe426ca6-linux-amd64-manifest'
All installed files intact.
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Generating a RSA private key
......+++++
..............................+++++
writing new private key to 'privKeySecure.pem'
-----
Signature ok
subject=/CN=lin.fritz.box/O=SplunkUser
Getting CA Private Key
writing RSA key
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
Done
[ OK ]
Waiting for web server at http://127.0.0.1:8000 to be available..................... Done
If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com
The Splunk web interface is at http://lin.fritz.box:8000
[splunk@lin bin]$
Now we activate systemd to start splunk at boot time:
[splunk@lin bin]$ /opt/splunk/bin/splunk stop
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
... [ OK ]
Stopping splunk helpers...
[ OK ]
Done.
[splunk@lin bin]$ exit
logout
[root@lin Downloads]# /opt/splunk/bin/splunk enable boot-start -systemd-managed 1 -user splunk
Systemd unit file installed at /etc/systemd/system/Splunkd.service.
Configured as systemd managed service.
[root@lin Downloads]# init 6
After the server restart we see that the Splunkd service is running:
systemctl status Splunkd
[root@lin ~]# systemctl status Splunkd
● Splunkd.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
Loaded: loaded (/etc/systemd/system/Splunkd.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2024-12-29 17:55:30 CET; 5min ago
Process: 1138 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/Splunkd.service (code=exited, status=0/SUCCESS)
Process: 1055 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/Splunkd.service (code=exited, status=0/SUCCESS)
Main PID: 1054 (splunkd)
Tasks: 166 (limit: 74402)
Memory: 1.2G (max: 11.4G)
CGroup: /system.slice/Splunkd.service
├─1054 splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd
├─2790 [splunkd pid=1054] splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd [process-runner]
├─3972 compsup daemon
├─4023 /opt/splunk/var/run/supervisor/pkg-run/pkg-identity4241153237/identity
├─4027 /opt/splunk/var/run/supervisor/pkg-run/pkg-agent-manager2026604619/agent-manager
├─4070 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore
├─4125 /opt/splunk/bin/python3.9 /opt/splunk/etc/apps/splunk_secure_gateway/bin/ssg_alerts_ttl_modular_input.py
├─4137 /opt/splunk/bin/python3.9 -O /opt/splunk/lib/python3.9/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
└─4139 /opt/splunk/bin/python3.9 /opt/splunk/etc/apps/splunk_secure_gateway/bin/ssg_enable_modular_input.py
Dec 29 17:55:38 lin.fritz.box splunk[2446]: Validated: _audit _configtracker _dsappevent _dsclient _dsphonehome _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket history main summary
Dec 29 17:55:38 lin.fritz.box splunk[2446]: Done
Dec 29 17:55:41 lin.fritz.box splunk[1054]: Checking filesystem compatibility... Done
Dec 29 17:55:41 lin.fritz.box splunk[1054]: Checking conf files for problems...
Dec 29 17:55:41 lin.fritz.box splunk[1054]: Done
Dec 29 17:55:41 lin.fritz.box splunk[1054]: Checking default conf files for edits...
Dec 29 17:55:41 lin.fritz.box splunk[1054]: Validating installed files against hashes from '/opt/splunk/splunk-9.4.0-6b4ebe426ca6-linux-amd64-manifest'
Dec 29 17:55:42 lin.fritz.box splunk[1054]: PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
Dec 29 17:55:42 lin.fritz.box splunk[1054]: 2024-12-29 17:55:42.694 +0100 splunkd started (build 6b4ebe426ca6) pid=1054
At that point I hit an error where the mongod process was not able to start and the log showed:
mongod exited abnormally (exit code 4, status: PID 34135 killed by signal 4: Illegal instruction)
I solved the problem by applying the steps described here. After applying the fix the service starts without error. The desktop / mobile web interface can be accessed with the URL: http://<hostname>:8000 using the username: admin and the password specified during the installation:
Installation of Splunk Mobile
If you want to use the mobile app Splunk Mobile you need to perform some additional tasks. First on the Apps section on the main page select “Splunk Secure Gateway”. I selected the defaults and choose Europe (Central) as the Spacebridge Location as this had the fastest response time for my location. Then select splunk mobile and click next:
A QR code shows up. Now install Splunk Mobile on your mobile and select Sign In and click on the + next to Private Instances. In the Code tab you can select Open Camera to scan this QR code. Thats it. We now have a basic splunk installation running on Linux and the possibility to login with our Desktop or with the mobile app.
