[09:25:35]
<Yunohost Git/Infra notifications> [yunohost] Axolotle pushed to enh-pydantic: fix: validation: typing ([b0b0abdb](https://github.com/YunoHost/yunohost/commit/b0b0abdb9bb78af0212947468bf2c40380ab5b55))
[09:25:35]
<Yunohost Git/Infra notifications> [yunohost] Axolotle pushed to enh-pydantic: refactor: validation: add YunohostCustomError helper to translate custom errors ([f123bc1c](https://github.com/YunoHost/yunohost/commit/f123bc1c321781c7cddaecdbed6bc0e13ff6a4c7))
[10:19:22]
<thatoo> Bonjour,
Ici, https://github.com/YunoHost-Apps/dolibarr_ynh/blob/master/hooks/post_user_create
on vérifie si la variable $DISABLE_HOOK est true et si c'est le cas, alors le hook post_user_create est annulé.
Dans ce script, https://github.com/YunoHost-Apps/dolibarr_ynh/blob/acddd2131dc62ed2aa2bdc0f3291028510af6f56/conf/syncyunohost.sh#L44 , la variable DISABLE_HOOK est "exporté" avec la valeur true.
Pourtant, dans le hook post_user_create se lance et si je regarde dans les logs, en effet, je trouve
```
2025-12-29 10:44:26,517: DEBUG - + sudo -E '[' '' == true ']'
```
[10:21:54]
<thatoo> Au préalable, la ligne
`` if sudo -E [ "$DISABLE_HOOK" == "true" ]; then ``
était
`` sudo -E if [ "$DISABLE_HOOK" == "true" ]; then ``
voir ce commit : https://github.com/YunoHost-Apps/dolibarr_ynh/commit/756dc1e7452455f6fa44dca3db0221142c5c4174
mais ça crée une erreur
```
2025-12-29 10:49:35,237: WARNING - ./50-dolibarr: line 4: syntax error near unexpected token `then'
2025-12-29 10:49:35,238: WARNING - ./50-dolibarr: line 4: `sudo -E if [ "$DISABLE_HOOK" == "true" ]; then'
2025-12-29 10:49:35,341: ERROR - Could not run script: /etc/yunohost/hooks.d/post_user_create/50-dolibarr
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/yunohost/hook.py", line 302, in hook_callback
hook_return = hook_exec(
^^^^^^^^^^
File "/usr/lib/python3/dist-packages/yunohost/hook.py", line 428, in hook_exec
raise YunohostError("hook_exec_failed", path=path)
yunohost.utils.error.YunohostError: Could not run script: /etc/yunohost/hooks.d/post_user_create/50-dolibarr
```
[10:22:35]
<thatoo> L'avantage de l'erreur est qu'elle interrompt bien le hook post_user_create mais elle le fait dans tous les cas, et ça ce n'est pas cool
[11:53:24]
<thatoo> I tried something like
```
sudo -E bash -c 'if [ "$DISABLE_HOOK" = "true" ]; then
echo "post_user_create hook is disabled."
exit 0
fi'
```
but sync_users_ldap2dolibarr.php is still launched so it means that the hook post_user_create isn't canceled
https://paste.yunohost.org/raw/bihatujano
Why $DISABLE_HOOK doesn't seem to be replaced by its value? Any help?
[12:09:10]
<Aleks (he/him/il/lui)> Uuuurh but why would you want to run a 'if' as sudo o.o
[12:10:36]
<Aleks (he/him/il/lui)> In that context the $VAR probably isnt replaced because the whole thing is wrapped in simple quotes preventing bash from expanding the $ syntax
[12:11:53]
<Aleks (he/him/il/lui)> But again i dont know why you would want this to be run inside a sudo... Testing if $var == somevalue and calling echo is going to yield the same thing indeopendently on the user you run the code with
[12:13:50]
<Salamandar> sudo echo is very useful
[12:14:16]
<Salamandar> it allows you to add a potential security vuln
[12:14:44]
<Aleks (he/him/il/lui)> wat 😬
[12:15:33]
<Salamandar> j/k
[13:06:09]
<thatoo> Good question.
I guess it's more the -E than sudo which is targeted here to get the value of $DISABLE_HOOK that has been exported within the app environment but I might be wrong.
I'm a bit lost who is exporting the value and who is checking it.
[13:07:59]
<thatoo> I tried
```
if [ "$DISABLE_HOOK" = "true" ]; then
echo "post_user_create hook is disabled."
exit 0
fi
```
It doesn't work.
I get
```
2025-12-29 13:59:56,822: DEBUG - + '[' '' = true ']'
```
in logs and sync_users_ldap2dolibarr.php is still launched.
[13:21:50]
<thatoo> I guess the issue is more coming from the export command : https://github.com/YunoHost-Apps/dolibarr_ynh/blob/acddd2131dc62ed2aa2bdc0f3291028510af6f56/conf/syncyunohost.sh#L44
[13:40:45]
<thatoo> I even tried
```
sudo -u "$app" -E bash -c "if [ "$DISABLE_HOOK" = "true" ]; then
echo 'post_user_create hook is disabled.'
exit 0
fi"
```
and got in logs (sync_users_ldap2dolibarr.php is still launched)
```
2025-12-29 14:18:08,378: DEBUG - + sudo -u dolibarr -E bash -c 'if [ "" = "true" ]; then
2025-12-29 14:18:08,379: DEBUG - echo '''post_user_create hook is disabled.'''
2025-12-29 14:18:08,379: DEBUG - exit 0
2025-12-29 14:18:08,380: DEBUG - fi'
```
[13:44:19]
<Aleks (he/him/il/lui)> uuuuh but exporting the variable is only relevant when calling other commands / programs from your script ... i mean ... you don't need `sudo` everytime your run an `if`, i dont get it x_x
[13:44:53]
<Aleks (he/him/il/lui)> if the regular `if` "doesn't work" then we need to understand why / what
[13:45:16]
<Aleks (he/him/il/lui)> by "I get `2025-12-29 13:59:56,822: DEBUG - + '[' '' = true ']'`", you mean that the variable shouldn't be empty string ?
[13:45:29]
<Aleks (he/him/il/lui)> where is that variable supposed to be set initially ?
[13:46:03]
<Aleks (he/him/il/lui)> line 56 there's an `unset DISABLE_HOOK` : https://github.com/YunoHost-Apps/dolibarr_ynh/blob/acddd2131dc62ed2aa2bdc0f3291028510af6f56/conf/syncyunohost.sh#L56
[13:47:56]
<thatoo> https://github.com/YunoHost-Apps/dolibarr_ynh/blob/acddd2131dc62ed2aa2bdc0f3291028510af6f56/conf/syncyunohost.sh#L44
[13:49:01]
<Aleks (he/him/il/lui)> ok i suppose i understand the context now ... :
1) you set `DISABLE_HOOK` to `true` from `syncyunohost.sh` then
2) call `yunohost user create`
3) expect that `DISABLE_HOOK` is propagated to the `post_user_create` hook
[13:50:09]
<Aleks (he/him/il/lui)> first thought is, i think the issue is at step 2, because `yunohost user create` is called with sudo, but not with `-E` so it doesnt propagate the exported variable
[13:50:49]
<Aleks (he/him/il/lui)> (but then ofc you don't need `sudo` inside the `post_user_create` because that's unrelated, you just need the variable to be propagated to the `post_user_create` hook script call)
[13:50:54]
<thatoo> indeed,
first it is set
`` export DISABLE_HOOK=true ``
then the command is launch
```
sudo yunohost user create "$USERNAME"
-p "$password"
-F "$fullname"
-d "$PARAM4"
```
then, the hook post_user_create is run (whereas it shouldn't because of DISABLE_HOOK=true)
then it adds mail forward (and I can see it does it after the hook)
and finally, it unset the variable in case the hook is needed later
[13:51:39]
<Aleks (he/him/il/lui)> try `sudo -E yunohost [...]` instead of `sudo yunohost [...]`
[13:52:53]
<thatoo> ok, and in the post_user_create, just a simple if, without sudo?
[13:56:27]
<Aleks (he/him/il/lui)> yes
[13:56:58]
<thatoo> ```
sudo -E yunohost user create "$USERNAME"
-p "$password"
-F "$fullname"
-d "$PARAM4"
```
simply doesn't work
[13:59:09]
<Aleks (he/him/il/lui)> if it doesn't work then please explain why / how it doesn't work
[13:59:20]
<m606> Hi Aleks, considering the parallelization example you've shown me, I wonder whether I can somehow pass variable by reference so that my current loop works? cf. the attached file for a simplified related case
[13:59:37]
<m606> https://aria.im/_bifrost/v1/media/download/AYcCAtSd7J8D-C1gCq7hhQRzl_sq8kdE5R8tzY79wHBcE-Z8_U5JO-BX7898VEicYTh-oINsW1Jwvj4yFdDsAtxCebb46jSAAG1hdHJpeC5vcmcvcmZYTmZMdlBycWZVbXVuT2VySnJObkVD
[14:00:16]
<thatoo> well the user isn't create, the script fails.
I don't have logs.
[14:02:24]
<m606> https://aria.im/_bifrost/v1/media/download/Ae6_Tt1OhaTfUUfUL8kyr7GYgKjjsWKd6io_7Owl9wYdIRvq0HJ-SpGDydBLT00Dcm8hXcow0L8LINW9XwoQjGVCebb5Eu3AAG1hdHJpeC5vcmcvZ2hzSk5TbUFBclRWbmJwVHhyY29aZ2FK
[14:03:17]
<Aleks (he/him/il/lui)> you mean you want to be able to modify the "counter" value in your example ? I realize that the underlying question may be "how do I return results from each work/job to the main thread" ... which in fact was not really an issue in my case of the logo fetching because the "result" was just saving the fetched files locally
[14:04:31]
<m606> yes I understand you didn't have this issue in the logo loop
[14:04:50]
<Aleks (he/him/il/lui)> supposedly the results are just, what the ThreadPool.imap call returns ?
[14:05:08]
<Aleks (he/him/il/lui)> (e.g. `results = ThreadPool(8).imap_unordered(fetch_logo, logos_to_download)` )
[14:06:13]
<Aleks (he/him/il/lui)> but how do you know it fails, isnt there at least an error message @_@
[14:06:24]
<m606> but no that's not exactly my question. it's more than from what I read here and there, `imap` is taking function + list as args
[14:06:57]
<m606> and more over that python (at least of int variable) does not support immutable vars.
[14:06:59]
<thatoo> I guess it is because the user dolibarr has very strict sudo ability set in
/etc/sudoers.d/dolibarr_syncyunohost
```
dolibarr ALL=(ALL) NOPASSWD: /usr/bin/yunohost user list --output-as json, /usr/bin/yunohost user create * -p * -F * -d *, /usr/bin/yunohost user update * --add-mailforward *, /usr/bin/yunohost user update * --remove-mailforward *, /usr/bin/yunohost user update * -F *, /usr/bin/yunohost user update * -p *, /usr/bin/yunohost user delete *, /usr/bin/yunohost user group add * *, !/usr/bin/yunohost user group add admins *, /usr/bin/yunohost user group remove * *, !/usr/bin/yunohost user group remove admins *
```
[14:08:09]
<m606> (one sec i will clarify my question)
[14:08:17]
<Aleks (he/him/il/lui)> hmmm okay but it would be curious that this forbids from calling those commands with sudo -E o.O Or maybe that's a thing O.o
[14:10:58]
<Aleks (he/him/il/lui)> hmm apparently there's an "env_reset" thingy that is enabled by default in /etc/sudoers
[14:11:11]
<Aleks (he/him/il/lui)> we can maybe work around this with another way to call the command with sudo
[14:12:03]
<Aleks (he/him/il/lui)> thatoo: can you try, instead of `sudo yunohost [...]` -> `sudo env DISABLE_HOOK=true yunohost [...]` ?
[14:12:31]
<Aleks (he/him/il/lui)> (NB when I write `[ ... ]` i mean "the rest of the arguments, not actually writing `[ ... ]` ...)
[14:13:01]
<Aleks (he/him/il/lui)> ah but you would need to enable the "env" command in sudo but that's like a big nono
[14:13:03]
<Aleks (he/him/il/lui)> urghfmpfngbn
[14:13:40]
<thatoo> didn't work
[14:14:18]
<Aleks (he/him/il/lui)> what about `sudo DISABLE_HOOK=true yunohost` ?
[14:14:35]
<Aleks (he/him/il/lui)> (without the intermediate `env` command)
[14:15:37]
<thatoo> neither
[14:16:09]
<Aleks (he/him/il/lui)> @_@
[14:16:18]
<Aleks (he/him/il/lui)> what happens exactly, how do you know it fails
[14:16:53]
<thatoo> the member is created in Dolibarr but the user isn't created in yunohost.
[14:17:31]
<Aleks (he/him/il/lui)> but you have no way to debug what's happening in the `syncyunohost.sh` script ?
[14:19:54]
<thatoo> well, I don't know how to do that in script but I tyried directly in the terminal :
```
$ sudo yunohost user create "usertest" -p "123usertest123" -F "User Test" -d "zyuno.local"
Vous êtes sur le point de définir un nouveau mot de passe. Le mot de passe doit comporter au moins 8 caractères, bien qu'il soit recommandé d'utiliser un mot de passe plus long (c'est-à-dire une phrase secrète) et/ou une combinaison de caractères (majuscules, minuscules, chiffres et caractères spéciaux).
Info : Le compte 'usertest' sera ajouté au groupe 'all_users'
Attention : PHP Warning: Undefined property: stdClass::$LDAP_FIELD_LOGIN_SAMBA in /var/www/dolibarr/scripts/user/sync_users_ldap2dolibarr.php on line 84
Attention : PHP Warning: Undefined property: stdClass::$LDAP_FIELD_PASSWORD in /var/www/dolibarr/scripts/user/sync_users_ldap2dolibarr.php on line 85
Attention : PHP Warning: Undefined property: stdClass::$LDAP_FIELD_PASSWORD_CRYPTED in /var/www/dolibarr/scripts/user/sync_users_ldap2dolibarr.php on line 86
Attention : PHP Warning: Undefined property: stdClass::$LDAP_FIELD_TITLE in /var/www/dolibarr/scripts/user/sync_users_ldap2dolibarr.php on line 95
Attention : PHP Warning: Undefined property: stdClass::$LDAP_FIELD_DESCRIPTION in /var/www/dolibarr/scripts/user/sync_users_ldap2dolibarr.php on line 96
Attention : PHP Warning: Undefined property: stdClass::$LDAP_FIELD_SID in /var/www/dolibarr/scripts/user/sync_users_ldap2dolibarr.php on line 97
Attention : PHP Warning: Undefined array key "telephonenumber" in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Trying to access array offset on null in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Undefined array key "facsimiletelephonenumber" in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Trying to access array offset on null in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Undefined array key "mobile" in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Trying to access array offset on null in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Undefined array key "telephonenumber" in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Trying to access array offset on null in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Undefined array key "facsimiletelephonenumber" in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Trying to access array offset on null in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Undefined array key "mobile" in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Trying to access array offset on null in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Succès ! Le compte a été créé
fullname: User Test
mail: usertest@zyuno.local
username: usertest
$ sudo yunohost user create "usertest" -p "123usertest123" -F "User Test" -d "^C
$ sudo DISABLE_HOOK=true yunohost user create "usertest" -p "123usertest123" -F "User Test" -d "zyuno.local"
sudo: désolé, vous n'êtes pas autorisé à définir ces variables d'environnement : DISABLE_HOOK
```
[14:21:16]
<thatoo> By the way, all these PHP Warniung are due to sync_users_ldap2dolibarr.php launched by the post_user_create hook that I'm trying to disable :-)
[14:22:21]
<thatoo> ```
$ sudo env DISABLE_HOOK=true yunohost user create "usertest" -p "123usertest123" -F "User Test" -d "zyuno.local"
[sudo] Mot de passe de dolibarr :
```
with env, it asks for the password indeed
[14:22:36]
<Aleks (he/him/il/lui)> hmmmokay so let's try to see how we may allow to explicitly pass the variable when calling sudo ...
[14:22:50]
<thatoo> ```
$ sudo -E yunohost user create "usertest" -p "123usertest123" -F "User Test" -d "zyuno.local"
sudo: désolé, vous n'êtes pas autorisé à conserver l'environnement
```
[14:23:00]
<Aleks (he/him/il/lui)> or actually it's unclear to me why this variable is needed in the first place (in case there's a different way to achieve this)
[14:23:49]
<m606> so I don't know what it returns (not tested yet) but I don't mind, what I want it to use the dict populated withing the loop. Here is a better example:
[14:24:08]
<m606> https://aria.im/_bifrost/v1/media/download/AQtiuz6zJlLP7LAum9T_jt7xHanq1LVH5kuwbI4I0uLIgWVE81C9Mffhq2kNSj58PSztFtENaMHQN0UzlX57241Cebb6UWUAAG1hdHJpeC5vcmcvaVBGdlV6S2VLbUpaQVVLQWxMR2R5c1h5
[14:26:47]
<thatoo> just to disable the post_user_create hook.
This hook sync dolibarr's user from ynh LDAP when a user is created in ynh
But as with this module, we are creating ynh user from dolibarr's member, we don't need this hook to run and as it takes about 1 minute to run (the hook), just to say that all users are already created, it's useless and time consuming.
[14:28:35]
<m606> there are again a few typos
[14:28:42]
<m606> i'm improving it
[14:29:33]
<thatoo> https://superuser.com/questions/1646926/sudo-sorry-you-are-not-allowed-to-set-the-following-environment-variables
Could help?
```
ALL=NOPASSWD:SETENV:
```
[14:30:39]
<thatoo> it works!
[14:31:37]
<thatoo> well, not sure, there are all these PHP Warning. I need to check in real situation
```
$ sudo DISABLE_HOOK=true yunohost user create "usertest" -p "123usertest123" -F "User Test" -d "zyuno.local"
Vous êtes sur le point de définir un nouveau mot de passe. Le mot de passe doit comporter au moins 8 caractères, bien qu'il soit recommandé d'utiliser un mot de passe plus long (c'est-à-dire une phrase secrète) et/ou une combinaison de caractères (majuscules, minuscules, chiffres et caractères spéciaux).
Info : Le compte 'usertest' sera ajouté au groupe 'all_users'
Attention : PHP Warning: Undefined property: stdClass::$LDAP_FIELD_LOGIN_SAMBA in /var/www/dolibarr/scripts/user/sync_users_ldap2dolibarr.php on line 84
Attention : PHP Warning: Undefined property: stdClass::$LDAP_FIELD_PASSWORD in /var/www/dolibarr/scripts/user/sync_users_ldap2dolibarr.php on line 85
Attention : PHP Warning: Undefined property: stdClass::$LDAP_FIELD_PASSWORD_CRYPTED in /var/www/dolibarr/scripts/user/sync_users_ldap2dolibarr.php on line 86
Attention : PHP Warning: Undefined property: stdClass::$LDAP_FIELD_TITLE in /var/www/dolibarr/scripts/user/sync_users_ldap2dolibarr.php on line 95
Attention : PHP Warning: Undefined property: stdClass::$LDAP_FIELD_DESCRIPTION in /var/www/dolibarr/scripts/user/sync_users_ldap2dolibarr.php on line 96
Attention : PHP Warning: Undefined property: stdClass::$LDAP_FIELD_SID in /var/www/dolibarr/scripts/user/sync_users_ldap2dolibarr.php on line 97
Attention : PHP Warning: Undefined array key "telephonenumber" in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Trying to access array offset on null in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Undefined array key "facsimiletelephonenumber" in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Trying to access array offset on null in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Undefined array key "mobile" in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Trying to access array offset on null in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Undefined array key "telephonenumber" in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Trying to access array offset on null in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Undefined array key "facsimiletelephonenumber" in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Trying to access array offset on null in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Undefined array key "mobile" in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Attention : PHP Warning: Trying to access array offset on null in /var/www/dolibarr/htdocs/core/class/ldap.class.php on line 1254
Succès ! Le compte a été créé
fullname: User Test
mail: usertest@zyuno.local
username: usertest
```
[14:31:49]
<Aleks (he/him/il/lui)> m606: urgh i'm lost at what `var1` etc is supposed to correspond to in the context of the security.toml thingy ... to me, i would imagine something like this (but i have not been working on this topic as much as you so i don't know the entire context) :
```python
def query_nist_api(cpe: str):
# (I don't know the actual URL / query args / HTTP headers but whatev)
r = requests.get(f"https://api.nist.gouv/whatever?cpe={cpe}")
assert r.return_code == 200 # Or whatever relevant thing allows to validate that we have a propre result
return (cpe, r.json()) # if this is the relevant thing to do
cpe_for_which_to_fetch_infos = ["cpe:1:2:3:nextcloud:nextcloud:*", "cpe:1:2:3:wordpress:wordpress:*", ...]
results = ThreadPool(8).imap(query_nist_api, cpe_for_which_to_fetch_infos)
# "results" will be a *list* of tuple with the cpe id + the json result from the queries,
# maybe you want to convert it to a dict if that's more useful idk :
nist_api_result_per_cpe = {}
for cpe, api_result in results:
nist_api_result_per_cpe[cpe] = api_result
```
[14:32:07]
<Yunohost Git/Infra notifications> [issues] grossermensch [commented](https://github.com/YunoHost/issues/issues/2737#issuecomment-3696687399) on [issue #2737](https://github.com/YunoHost/issues/issues/2737) Discuss about postgresql migration on debian major upgrade: I am currently running into exactly this issue when updating from yunohost 12 to 13 (Bookworm to Trixie) which comes wit...
[14:33:38]
<thatoo> which is normal, I didn't change yet the post_user_create hook. I'm trying and will let you know
[14:33:49]
<Aleks (he/him/il/lui)> (I'm supposed to work on $dayjob folks so might be less responsive)
[14:34:23]
<thatoo> Bingo!
https://paste.yunohost.org/raw/otogaguyab
[14:34:30]
<thatoo> Thank you <a data-mention-type="user" href="https://matrix.to/#/@Alekswag:matrix.org" contenteditable="false">Aleks (he/him/il/lui)</a>
[14:39:06]
<Aleks (he/him/il/lui)> m606: i'm not sure exactly but my understanding is that you were confused by how to organize the code because maybe you wanted to update the secuirty.toml datastructure directly from the function from each worker/thread but you should definitely not do that, you want to have one step "fetch all the infos you need" (which is I/O intensive), and then use all the results fetched to update the security.toml
[14:39:24]
<m606> ah you imagined that at the level of the API.
basically the script calls 2 APIs and populates a dictionnary with it.
I thought of implementing Threadpooling at in a loop which would include both API and appending data to dictionnary
[14:40:04]
<m606> yes I guess that's it
[14:40:14]
<Aleks (he/him/il/lui)> yeah but clearly having multiple threads trying to modify a common datastructure is ... one of the hardest problem of computer science, so if it's not necessary you should just avoid it
[14:40:44]
<m606> but ok no rush, I'll first publish a version without multiprocessing, we'll see then
[14:41:58]
<Aleks (he/him/il/lui)> thread/processes sharing common read-writable data is a hug pitfall, because a thread A may want to write the thing at the same time as thread B, while maybe a thread C wants to read it etc, you need "locks" (mutex / semaphores(?)) etc to prevent race conditions, it's just a complex topic, but we don't need this here
[14:45:33]
<m606> I mean no thread would modify existing data in the dict
[14:46:18]
<m606> hmm not sure how it works exactly because each thread would write in a different key of the dictionnary, but would the final dictionnary contain all the updates of the various thread, or are they working on copy of the original dict, and due to race conditions, then not all updates would appear in the end?
[14:46:38]
<Aleks (he/him/il/lui)> yeah in this case it may feel "simple enough that it could work" but i wouldn't even be sure that if both thread A and B try to add entries to a dictionnary D, I wouldn't bet that "D" is actually "shared" between threads instead of each threads having its own copy ... would need to check the multiprocess/multihreading doc for this
[14:47:15]
<m606> ok yes that's the point. thanks!
[14:48:35]
<Aleks (he/him/il/lui)> and generally speaking, you may know as the developer that "each thread will only want to update keys that other threads won't want to edit", but the program doesn't know that, thread A and B could very well try to update the same key "at the same time" (well, not really because in Python, multi-threading is sort of an illusion because of the GIL... but in other languages, or in post-GIL python, that would be an actual issue I suppose)
[14:51:14]
<Aleks (he/him/il/lui)> anyway, when using multiprocessing / multithreading, if you can avoid getting lost in existential questions of which data are shared and how exactly, etc, it's just way simpler so the idea here is to limit to "only perform the HTTP queries in parallel, and keep the actual logic of "update the security.toml" as a single-thread thing because it's not going to be CPU nor IO intensive once you have all the queried info available
[14:53:36]
<Aleks (he/him/il/lui)> the key aspect here being that each thread job should be purely independent from each other, which is the case if you stick to "it just performs the HTTP query". But they are not independent anymore if you decide that "it performs the HTTP query AND update a common data structure"
[16:48:40]
<Yunohost Git/Infra notifications> [yunohost] Axolotle pushed to enh-pydantic: refactor: validation: return not dumped form ([0b0b6c6c](https://github.com/YunoHost/yunohost/commit/0b0b6c6c85d58ee7d33c49884a70cb87e628fd58))
[16:48:40]
<Yunohost Git/Infra notifications> [yunohost] Axolotle pushed to enh-pydantic: refactor: app_install: rename app install arg + simplify env dict building ([c8d1a2aa](https://github.com/YunoHost/yunohost/commit/c8d1a2aa88f74e5be56a387d4a8cb81ef46a1c39))
[16:48:41]
<Yunohost Git/Infra notifications> [yunohost] Axolotle pushed to enh-pydantic: refactor: test: update test_questions calls to ask_questions_and_parse_answers ([2495029e](https://github.com/YunoHost/yunohost/commit/2495029e84c0d96f4f93a4d9382ef4eea8e2774b))
[16:52:21]
<Yunohost Git/Infra notifications> [yunohost] Axolotle pushed to enh-pydantic: refactor: remove app install legacy env args YNH_APP_ARG* ([202a7dbc](https://github.com/YunoHost/yunohost/commit/202a7dbc8e2a31487cf269c9d75fd1d8ee7e00e6))
[16:54:00]
<Yunohost Git/Infra notifications> [yunohost] Axolotle edited [pull request #2005](https://github.com/YunoHost/yunohost/pull/2005): [Trixie] Pydantic v2
[16:54:15]
<Yunohost Git/Infra notifications> [yunohost] Axolotle edited [pull request #2005](https://github.com/YunoHost/yunohost/pull/2005): [Trixie] Pydantic v2
[21:24:59]
<m606> So based on your suggestion I have reworked the script to separate IO & datastructure.
Here is now that very loop which could be added multiprocessing capabilities (currently takes between 1h and 1h20 without API key for 642 apps, should be less with a key):
```py
retrieved_vulnerabilities = {}
apps_counter = 0
for app_name, props in catalog.items():
# @TODO? Add multiprocessing to this loop - cf. https://github.com/YunoHost/yunohost/blob/dev/src/app_catalog.py#L253-L272
# Lookup only for app names passed as function argument
if 'all' in selected_apps or app_name in selected_apps:
apps_counter += 1
logging.info('-------------------------------------------------')
app_url = props['url']
api_results = check_app_security(cache_path, app_name, app_url, from_date_source, apps_counter, apps_number)
if api_results:
retrieved_vulnerabilities[app_name] = api_results
progress_bar.update(1) #that's really the loop that's takes 99.7% of exec time, so let it account for 100% on the progress bar
```
`check_app_security` is the function which calls the 2 APIs and format their output on a common template. Now contrary to what I showed you earlier today, not any of the variables need to be passed by reference to check_app_security.
[21:26:05]
<m606> but I will push it this evening as it works well in the current condition
[21:44:24]
<Yunohost Git/Infra notifications> [issues] SveDec [commented](https://github.com/YunoHost/issues/issues/2112#issuecomment-3697605362) on [issue #2112](https://github.com/YunoHost/issues/issues/2112) Allow restricting services to remote IPs not only allow or deny: ### Update to my january comment
Since aprils [migration n°32](https://github.com/YunoHost/yunohost/blob/dev/debian/cha...