Segmentation fault: No such process (src/thread.cpp:147) with Alpine/3.8
Hello,
salt-call sometimes crashes with a enigmatic “No such process (src/thread.cpp:147) Segmentation fault” on Alpine/3.8 and Alpine/edge.
I suspect this problem’s cause lies with python or musl. I opened a ticket for saltstack as well: https://github.com/saltstack/salt/issues/50598
This problem appears with salt versions 2017.7.8 and 2018.3.3, both from
the Alpine repositories and (for Alpine/edge) when installing via
bootstrap.saltstack.com.
The problem does not occur with Alpine 3.7 or when running
masterless (file_client: local)
To test this with Alpine 3.8 and 3.7 versions from bootstrap.saltstack.com (includes a workaround for some trouble with m2crypto and futures):
mkdir -p -m 1777 /dev/shm # https://bugs.alpinelinux.org/issues/6407
apk add curl strace build-base py2-pip python-dev openssl-dev swig; pip install m2crypto
pip install six futures # fixes ImportError: No module named concurrent
curl -Ls https://bootstrap.saltstack.com | sed 's/py2-m2crypto//g'|sh /dev/stdin -FXPMDd git v2018.3.3
echo master: 127.0.0.1 > /etc/salt/minion
service salt-minion stop; service salt-minion start; service salt-master start
salt-call test.ping ; salt-key -Ay
while salt-call saltutil.sync_all; do true; done
After a (sometimes short, sometimes long) while this gives: No such
process (src/thread.cpp:147) Segmentation fault.
The problem occurs faster when memory is tight: lxc config set a38
limits.memory.swap false ; lxc config set container_name limits.memory
340MB
Strace shows its caused by a call to clone.
mmap(NULL, 94208, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd950018000
mprotect(0x7fd95001a000, 86016, PROT_READ|PROT_WRITE) = 0
clone(child_stack=0x7fd95002e648, flags=CLONE_VM|CLONE_FS
No such process (src/thread.cpp:147)
|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tidptr=0x7fd95002eb20, tls=0x7fd95002eae8, child_tidptr=0x7fd95002eb20) = 3693
+++ killed by SIGSEGV +++
Segmentation fault
To reproduce on Alpine edge (lxd container, 512MB RAM) image:
mkdir -p -m 1777 /dev/shm # https://bugs.alpinelinux.org/issues/6407
apk update; apk add strace salt-master salt-minion
echo master: 127.0.0.1 > /etc/salt/minion
service salt-minion stop; service salt-minion start; service salt-master start
salt-call test.ping ; salt-key -Ay
while salt-call saltutil.sync_all; do free -h; done
(from redmine: issue id 9675, created on 2018-11-21)