Glewlwyd ERROR: Glewlwyd - Received segmentation fault signal #101
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
KittenSquad/glewlwyd#101
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Describe the issue
Under stress glewlwyd docker container stop responding, sometimes it logs "Glewlwyd ERROR: Glewlwyd - Received segmentation fault signal" (when I set DEBUG log level).
I observed this behavior in my kubernetes deployment, when number of connections with token request with ESTABLISHED status was ~40-50 (I checked with netstat inside running container, with simple script
while true; do date; netstat -nt| awk '{ print $6 }' | grep -v Foreign | grep -v ^$ | sort | uniq -c; sleep 1; doneTo Reproduce
Deploy glewlwyd in Kubernetes, generate > 50 connections to container with token request.
I discovered another way to to achieve this undesirable condition (maybe the reason is the same?):
a) start official docker image (I belive this is 2.1.1 vesion):
docker run --rm -it -p 4593:4593 babelouest/glewlwydb) configure oauth2 plugin, I set only "Name", "Display Name" as "oauth2", and add private and public key. I didn't configure any client or scope.
c) generate as much token request as you can. I use twice - on two terminals - that script:
After some time (less then minute) glewlwed is stucked, but container is still running - you can't get token or login to web interface.
Expected behavior
glewlwyd should handle greater load/more connections
System (please complete the following information):
Docker logs
Hello @rurus9 ,
I've ran some tests to try to reproduce your problem but so far without success.
I've tried with the following parameters:
Each combination of those parameters were tested, but I haven't been able to reproduce any problem you mention.
Although, my test script is executed from another machine than the one running Glewlwyd, because the test itself can affect the program tested in terms of performance.
To be sure, I ran tests using apache ab, using the same parameters combination as above with the compiled source instance, but not with the docker image.
Each tests were completed successfully (i.e. no segfault),
Here is the type of ab command I use:
The file post.txt contains the following:
And the result looks like this:
I suggest you check the configuration in your infrastructure running the docker images, but so far I couldn't find any problem in Glewlwyd itself that could lead in segfault in case of heavy load.
Today we tested glewlwyd official docker image on few Linux servers and workstations, every time a large number of requests caused the application to hang.
This time we used ab too, with parameters like yours:
post.txt:
We tested this way
1. First we started docker image:
docker run -it -p 4593:4593 babelouest/glewlwyd2. Then we used from another station
ab- without any glewlwyd configuration. Every test was fine:3. Then we only added oauth2 plugin (we set: "Name" and "Display Name":
glwd, Tokens signature:JWT type, Key size:512 bitsand "Secret key" and "Public key").Now, when we ran ab, glewlwyd stucked... Not every time with
Glewlwyd ERROR: Glewlwyd - Received segmentation fault signal, but whenever you can't get web login interface and working token endpoint.ab logs:
Although glewlwyd does not work, docker image is still running, we can log into with
docker exec -it CONTAINER_ID bashand execute commands:How can I help debug this strange docker image behavior?
Hello @rurus9 ,
So I could finally reproduce the crash with the docker image.
For what I've seen, the problem lies in Alpine Linux, and more specifically with the architecture x86_64, because I've run heavy load tests with the following docker platforms:
The only platform with the crash problem is
Alpine Linux 3.11 x86_64, which unfortunately is the platform underneath Glewlwyd docker image.I'm gonna ask to the Alpine team if there's a way to fix that, it may be a config problem or a bug on their side, but I'll also provide at least 2 other docker images for Glewlwyd: one with Debian Buster and one with Ubuntu LTS. And until the problem in Alpine Linux is fixed (if it can be), I suggest to use the other images if performance is an issue.
The new docker images will be available soon.
@babelouest - this is both good and bad news.
Could you share your Dockerfile for Debian [x86_64] flavor?
Hello @rurus9 ,
Can you try to build the docker image with the current master branch? It looks like
github.com/babelouest/glewlwyd@d5948972affixed it for us.I was testing today and it's ok now.
Thank you.