To troubleshoot the issue using a Python debugger with remote debugging, we will need a local virtual environment with debugger package installed which will act as a customer and will try to connect to a debugger session opened inside Splunk’s python script using the same debugger. Below steps will guide you through this environment setup in both Windows and Linux Splunk environments.
/ $ mkdir debug
/ $ virtualenv debug
….
/ $ source debug/bin/activate
(debug) $
C:\> mkdir debug
C:\> virtualenv debug
….
C:\> .\debug\Scripts\activate.bat
(debug) C:\>
After activating your virtual environment, we will use “pip” to install the debugger packages in our virtual environments.
(debug) / $ pip install epdb
….
(debug) / $ pip show epdb
….
Location: /debug/lib/python2.7/site-packages
….
(debug) C:\> pip install rpdb
….
(debug) C:\> pip show epdb
….
Location: c:\debug\lib\site-packages
….
Download the Netcat utility from https://joncraton.org/blog/46/netcat-for-windows/ and unzip the package in a directory.
Verify the utility works by viewing its help:
C:\nc111nt> nc -h
….
As we want to troubleshoot what’s happening when the python script is called, we will have to use the python debuggers inside our python script by starting a debug session and putting a breakpoint where we want the script to halt. Use the below code snippet to insert breakpoints in your python scripts.
Note: Please make sure you set proper file permissions so that you can edit and save the file
# Modified Code - Start
import sys
sys.path.append("/debug/lib/python2.7/site-packages")
import epdb
epdb.serve(port=12345)
# Modified Code - End
# Modified Code - Start
import sys
sys.path.append('c:\debug\lib\site-packages')
import rpdb
debugger = rpdb.Rpdb(port=12345)
debugger.set_trace()
# Modified Code - End
As seen in the above code, we first imported the “sys” module and added our folder where debugger packages are available as a module path. This way our script can now import modules from that folder. Then we imported the debugger modules “epdb” and “rpdb” in Linux and Windows respectively and started the debugging sessions on port 12345. Once the Splunk’s python interpreter executes this line, it halts the program, listens on this port and waits for the debugger customers to connect to this port and continue to debug the program.
(debug) C:\> pip install rpdb
….
(debug) C:\> pip show epdb
….
Location: c:\debug\lib\site-packages
….
(debug) / $ python
Python 2.7.16 (default, Apr 12 2019, 15:32:40)
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import epdb
>>> epdb.connect(port=12345)
….
(Epdb)
In Windows, we will use the “netcat” command-line utility instead of Python virtual environment as below:
(debug) C:\nc111nt> nc localhost 12345
….
(Pdb)
Ishan Gajera is a Technical Lead at Crest Data Systems. With 9+ years of experience in software development, he is a seasoned technical leader and project manager. At Crest, Ishan has worked with multiple Enterprise customers in the areas of Data Analytics, DevOps, and Cloud technologies. Before joining Crest, Ishan worked for 5 years at Tata Consultancy Services developing large Enterprise and Web applications. He is a graduate of Bachelor of Engineering in Information Technology from Dharmsinh Desai University, India.