Error reading modules directories with ZFS #150
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
KittenSquad/glewlwyd#150
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Describe the issue
Glewlwyd gives wrong diagnostics on missing data. ERROR: load_client_module_instance_list
To Reproduce
Install Glewlwyd using your favorite method.
Initialize sqlite3
sqlite3 $db < init.sqlite3.sqlTry to start Glewlwyd:
`2020-06-26T15:33:24Z - Glewlwyd INFO: Starting Glewlwyd SSO authentication service
2020-06-26T15:33:24Z - Glewlwyd ERROR: load_user_module_instance_list - Error module database not found
2020-06-26T15:33:24Z - Glewlwyd ERROR: load_client_module_instance_list - Error module database not found
Screenshots
N/A
System (please complete the following information):
cat /etc/os-release gives
Additional context
First, and most important, THANK YOU for this project. We hope PRs like this will help to strengthen the project.
Now, the interesting part is we are able to start Glewlwyd if we overlay init-core.sqlite3.sql
Resulting in >
2020-06-26T15:51:12Z - Glewlwyd INFO: Starting Glewlwyd SSO authentication service
2020-06-26T15:51:12Z - Glewlwyd INFO: Glewlwyd started on port 4593, prefix: api, secure: false, bind address: 192.168.0.77
However, one gets invalid credentials on initial admin login.
Do we still need to run init-core.sqlite3.sql? If not, what are we missing? strace does not show any missing library dependency.
Thanks again.
init-core.sqlite3.sqlonly creates the core tables, not the modules tables, neither initializes the data, so you shouldn't use it to initialize your environment.It looks like you don't have the modules installed in their directory. Can you check the presence of the *so files in the
*_module_pathvalues referenced in the config file?Also when you say
Install Glewlwyd using your favorite method., what method is it?On June 26, 2020, at 12:25 PM, Nicolas Mora notifications@github.com wrote:
The modules are in the module path. We double checked. We have the full path names, and confirmed the the user (an admin) has full access to the path hierarchies.
Downloaded the debian-buster.tar.gz file, extracted the contents, and dpkg -i the .deb files.
On June 26, 2020, at 12:25 PM, Nicolas Mora notifications@github.com wrote:
Oh no, please do ;)
In the
init.sqlite3.sqlyes, which is the file mentioned in the Initialise database section in the install documentation, but theinit-core.sqlite3.sqlscript only creates the core tables.So by initializing your database with the
-core-*sqlscript you won't have the modules client-database nor user-database, and no user available, that's why you won't have error messages in the logs but won't be able to log in either.The problem lies where this error message comes from:
For some reason, your glewlwyd instance doesn't loads the modules properly. Can you post your full glelwwyd log?
Hi,
We followed the instructions give here Install Glewlwyd on Debian Buster but changed the version numbers to match the latest release v2.3.1.
The full installation log, and the glewlwyd console log is attached.
requested.log
We are using sqlite3 for this test; so we recreated the database using init.sqlite3.sql only.
You do see that glewlwyd started with no diagnostics but login as admin meets with
2020-06-26T20:59:09Z - Glewlwyd WARNING: Security - Authorization invalid for username admin at IP Address 192.168.0.57which makes it impossible to configure glewlwyd.Any ideas?
It seems that you follow the install procedure correctly, yet something's not right, I still don't know what yet.
In your logs, the interesting part is missing. The glerwlwyd logs you display are those:
I need what's before that in the glewlwyd.log file to try to understand what's wrong. In your first message, you mention these error messages:
That's the kind of error message I need
This is why I asked if there is an order to how sql files are installed.
IF we load the sql files in this order
then we get this
If this order,
or
we install ** init.sqlite3.sql only** then admin cannot login.
PS: We are happily enjoying v2.2.0 -Works Great! Thank you.
This is a clean VM serving as Glewlwyd TEST environment for the 2.3.1 upgrade.
PPS: We have tried running as root, thinking it might be related to permissions, but it makes no difference.
There is no order to run the sql files because there is only one sql file to run, in your case it's the file docs/database/init.sqlite3.sql.
What I see in your log output is that glewlwyd loads no plugin at all.
Normally, your log output should look like that:
Can you check that the module paths in your config file are correctly filled, and point to a directory where the module files are present are accessible to the user running glewlwyd?
Thanks @babelouest.
We have tried those two options - Triple checked the module path, and run as root to eliminate permissions issue.
We'll keep thinking about this one. Please leave the ticket open for a couple of days while we troubleshoot the install.
Can you please point me to the routine that attempts to load the plugin modules, or walk me through the process of loading the modules from the database? strace is showing that the .so files are there but read from the database?
Thank you.
More information.
Should those not return the list of modules after a fresh installation, or are we barking up the wrong tree?
We are getting warmer - we think.
We pointed
We got an error -
So, glewlwyd is probing the directory but is refusing to load the files. Strace does not show glewlwyd trying to read the directory entries. In fact chmod 0000 on the files does not make a difference to glewlwyd.
Eureka! (We think)
We use SmartOS internally for various functions and reasons. If you look carefully at the man pages for readdir
Currently, only some filesystems (among them: Btrfs, ext2, ext3, and ext4) have full support for returning the file type aka DT_REG in d_type. All applications must properly handle a return of DT_UNKNOWN.Unfortunately, ZFS does not return DT_REG. All is not lost,
man readdirand posix mandates that all applications must handle a return of DT_UNKNOWN.It would be wonderful if lines 1626 , 1886, 2135, and 2397 in glewlwyd.c are modified to also test for DT_UNKNOWN.
I am only a C program reader (not a programmer) so the fix could be a simple DT_REG || .. DT_UNKNOWN test. However, according to this SO question. If d_type is DT_UNKNOWN you need stat to learn anything about whether the directory entry is itself a directory or whatever. I do not believe that this use case is that complex.
It will be my pleasure to help with testing when you are ready.
Thanks again for this wonderful project.
Ah, that's very interesting.
Indeed I only test
if (in_file->d_type == DT_REG). Thanks for pointing out the problem with exotic filesystems (yes, ZFS is exotic, change my mind ;) ).I'll add an else case with DT_UNKNOWN according to the documentation, I'll let you know when there will be a version to tests.
Ah, the beauty of open source software:)
Awesome project. More grease to your elbow.
Thanks @babelouest.
A quick search through glewlwyd code base shows DT_REG is used in glewlwyd.c only. My conclusion after a simple analysis of that file makes me believe that a simple
in_file->d_type == DT_REG || in_file->d_type == DT_UNKNOWNfix in the four places in that file would be sufficient to close the PR. There is no need for an else case.in_file->d_type == DT_REG || in_file->d_type == DT_UNKNOWNThat may work in your case, but if the
in_fileis not a regular file, there may be bigger problems. I will stat the path pointed byin_fileand useS_ISREG()to verify it's a regular file instead.You are right. Of course you are right - Reason I did not want to argue with the creator:)
We made a quick, one liners, half-has fix to the code and rebuilt for testing. Here is the feedback.
The two directories "." and ".." proved to be a source of annoyance but the modules are loaded and we can carry on with our testing. We will patiently wait for a fix by a true professional.
Thank you.
Hello @a1730 ,
I've added a second check on module files with
statwhenin_file->d_type == DT_UNKNOWN, can you test the last commitgithub.com/babelouest/glewlwyd@13aefd6e3dto verify if the fix actually fixes your problem?My manual tests pass but I don't have a ZFS filesystem at the moment to be 100% sure.
Works like a charm.
We will run this version until your next release.
Thank you.
All right then, release 2.3.2 will be out soon I guess, I have a couple more of little updates to add and it'll be ready. Until then, if you have other bugs, don't hesitate.
Thanks for the bug report @a1730 !