[07:04:54]
<Yunohost Git/Infra notifications> [yunohost] 🔴 Pipeline [#1291302335](https://gitlab.com/YunoHost/yunohost/-/pipelines/1291302335) failed on branch dev
[08:49:33]
<Guillaume Bouzige> Hello there, https://github.com/YunoHost/yunohost/pull/1194 is this lexicon lib implemented or this is just preparatory work ?
[08:49:55]
<Aleks (he/him/il/lui)> yes that was done a while ago
[08:50:21]
<Guillaume Bouzige> so it mean the auto dns conf works with multiple dns providers ?
[08:50:27]
<Aleks (he/him/il/lui)> that corresponds to the "DNS" stuff in the webadmin > Domains > any 'top' domain > config panels stuff
[08:50:41]
<Aleks (he/him/il/lui)> yeah assuming you fill the appropriate credentials
[08:51:01]
<Aleks (he/him/il/lui)> but in practice only OVH and idk, Gandi iirc? were properly tested
[08:51:30]
<Guillaume Bouzige> this part "YunoHost automatically detected that this domain is handled by the registrar gandi."
[08:51:34]
<Aleks (he/him/il/lui)> the annoying part is that lexicon ain't really "flat", ie there are still specificities related to some providers and you can't guarantee stuff really works without actually testing each provider (or registrar I mean)
[08:51:49]
<Guillaume Bouzige> I see
[08:51:53]
<Guillaume Bouzige> but promising
[08:52:03]
<Guillaume Bouzige> if one want to test and repare one provider...
[08:52:11]
<Aleks (he/him/il/lui)> yeah
[08:52:23]
<Guillaume Bouzige> I always thoughts this was limited to gandi only
[08:52:31]
<Guillaume Bouzige> so it is great to know
[08:53:26]
<Guillaume Bouzige> so this one https://github.com/YunoHost/yunohost/pull/1820 is only related to the proposed or auto-applyied conf in case of correct api key and implementation..;
[08:55:44]
<Aleks (he/him/il/lui)> uuuh well yunohost also generally-speaking generates a recommended DNS configuration
[08:55:58]
<Aleks (he/him/il/lui)> either you apply it "manually" either via the lexicon integration
[08:56:08]
<Aleks (he/him/il/lui)> (or nsupdate / dyndns for the nohost.me / noho.st etc)
[08:56:15]
<Guillaume Bouzige> yep that is what I understood
[08:56:38]
<Aleks (he/him/il/lui)> but anyway i'm seeing in the code that so far our version of lexicon doesnt support CAA record for some reason
[08:56:52]
<Aleks (he/him/il/lui)> https://github.com/YunoHost/yunohost/blob/dev/src/dns.py#L707
[08:57:02]
<Aleks (he/him/il/lui)> ah, or it works with Gandi but not OVH ? idk
[08:57:25]
<Guillaume Bouzige> yeah with gandi it does
[08:57:31]
<Aleks (he/him/il/lui)> i spent hours testing several providers and tweaking stuff it was kind of a pain
[08:57:49]
<Guillaume Bouzige> yeah dns is pain innit
[08:58:08]
<Guillaume Bouzige> I was trying to understand this comment https://github.com/YunoHost/yunohost/pull/1820#issuecomment-2081560782
[09:00:32]
<Aleks (he/him/il/lui)> it's related to the fact that the DNS configuration is pretty generic and is "for every domain", but reality is more complex and there are cases where you *want* every subdomain to be handled by this server, but some other times, that may not be the case yet this may lead to DNS resolution issues etc
[09:01:33]
<Aleks (he/him/il/lui)> e.g. having the wildcard record (`*` in A and AAAA records) means that if anybody does a typo in the domain name, or reach "bl0g.domain.tld" instead of "blog.domain.tld", the DNS will still answer with the IP of the server, but nginx may be configured to answer for blog.domain.tld and not bl0g.domain.tld
[09:01:48]
<Aleks (he/him/il/lui)> which will result in the user seeing a spooky invalid cert warning in their browser
[09:02:55]
<Aleks (he/him/il/lui)> though we can also configure nginx to handle wildcard but that may have other implications etc
[09:03:09]
<Guillaume Bouzige> I have personally never applied the wildcard to a domain unless it is absolute necessity but understand that from a simplistic admin perspective it helped to smooth things a little
[09:04:05]
<Aleks (he/him/il/lui)> yeah, one the one hand you want the wildcard to avoid adding an A/AAAA record everytime you add a subdomain, but on the other hand, nginx ain't configure to handle a wildcard an people hitting a typo will see an invalid cert
[09:10:28]
<Aleks (he/him/il/lui)> so back to the issue, i think titus was proposing to have a setting for this
[09:10:30]
<Aleks (he/him/il/lui)> but on the other hand one must be careful with design and adding too much settings can slowly turn yunohost into an epic mess and loose the kiss aspect of yunohost
[09:10:36]
<Aleks (he/him/il/lui)> a *per-domain* setting i mean (i think)
[09:10:37]
<Guillaume Bouzige> most peoples will happily copy and paste at install and that will be it
[09:10:39]
<Guillaume Bouzige> I guess
[09:10:48]
<Émy - OniriCorpe> > <@Alekswag:matrix.org> it's related to the fact that the DNS configuration is pretty generic and is "for every domain", but reality is more complex and there are cases where you *want* every subdomain to be handled by this server, but some other times, that may not be the case yet this may lead to DNS resolution issues etc
you forgot the case you want all subdomains of a domain to be handled by a specific package (so other packages are forbidden to use them), like for adguardhome \\:D/
[09:11:46]
<Émy - OniriCorpe> > <@oniricorpe:im.emelyne.eu> you forgot the case you want all subdomains of a domain to be handled by a specific package (so other packages are forbidden to use them), like for adguardhome \\:D/
so we also have to do something in the manifest for that case
[09:11:49]
<Émy - OniriCorpe> yay more work
[10:39:06]
<Yunohost Git/Infra notifications> [issues] alexAubin opened [issue #2391](https://github.com/YunoHost/issues/issues/2391): bookworm: add a post ssowatconf hook ?
[10:39:06]
<Yunohost Git/Infra notifications> [issues] alexAubin labeled :birthday: feature on [issue #2391](https://github.com/YunoHost/issues/issues/2391): bookworm: add a post ssowatconf hook ?
[11:55:41]
<Yunohost Git/Infra notifications> [issues] selfhoster1312 opened [issue #2392](https://github.com/YunoHost/issues/issues/2392): regenconf is super slow
[11:55:41]
<Yunohost Git/Infra notifications> [issues] selfhoster1312 labeled :space_invader: bug on [issue #2392](https://github.com/YunoHost/issues/issues/2392): regenconf is super slow
[11:56:46]
<selfhoster1312> i'll keep digging for optimization by rewriting it to better understand the code :)
[11:59:10]
<Yunohost Git/Infra notifications> [issues] alexAubin [commented](https://github.com/YunoHost/issues/issues/2392#issuecomment-2115043568) on [issue #2392](https://github.com/YunoHost/issues/issues/2392) regenconf is super slow: >md5 is slow... Honestly I very much doubt that md5 hashing is the issue ... just look at the call to yunohost setting...
[12:01:18]
<Aleks (he/him/il/lui)> also i wouldn't call "0.2s" "unbearably slow ... but yes on a RPi, regen conf can take something like 30s
[12:01:46]
<Yunohost Git/Infra notifications> [issues] selfhoster1312 edited [issue #2392](https://github.com/YunoHost/issues/issues/2392): regenconf is super slow
[12:01:50]
<selfhoster1312> well that's just for small steps the whole regen-conf takes a lot longer
[12:02:06]
<Yunohost Git/Infra notifications> [issues] OniriCorpe labeled :cake: enhancement on [issue #2392](https://github.com/YunoHost/issues/issues/2392): regenconf is super slow
[12:02:26]
<selfhoster1312> i'm not sure md5 is the bottleneck... it could be on a slow CPU, but IO is always going to be a bottleneck which is why using timestamps instead of md5 sum could be a lot faster
[12:03:07]
<Émy - OniriCorpe> > rewrite it in rust for blazing fast correctness and fearless concurrency? 8)
yes and now almost no one here can read it :D
[12:04:00]
<selfhoster1312> it's not very different from python, and not very hard to learn if you treat it as high-level language and don't go for every last drop of performance :)
[12:04:31]
<selfhoster1312> but well Rust is the best language out there ® but i think even in python we could have some optimizations like avoid to read the same file many times in the same operation :-/
[12:07:20]
<selfhoster1312> reading settings.py, is there no locking in place? how do we prevent corrupting the settings when reading/writing at the same time?
[12:12:20]
<Bram> yunohost has a global lock mechanism for those situations
[12:13:11]
<Bram> and there are no best languages, there are good languages for specific situations and I'm already tired thinking about having to argue about the details for yunohost
[12:13:38]
<Bram> also, iirc, regen-conf is calling a lot of other services and so on which can take a lot of time
[12:15:25]
<selfhoster1312> Bram, the "best language" comment was obviously tongue in cheek 😅️
[12:16:28]
<selfhoster1312> but yes i noticed calling external commands is taking a lot of time in the hooks themselves
[12:16:37]
<Bram> > <selfhoster1312> Bram, the "best language" comment was obviously tongue in cheek 😅️
since the beginning of the project we keep getting "you should use {this language} {this distro} {this packaging system}" all the time, I kinda admit I'm a bit tired of that ^^'
[12:17:23]
<Bram> tbh if you really want to do rust, please rewrite this whole mess of slapd in rust with a real documentation and good UX and the whole world will thanks you 🥲
[12:17:59]
<selfhoster1312> oh noes not slapd 😭️
[12:18:22]
<Bram> yeah, exactly
[12:19:57]
<selfhoster1312> but tbh i do feel like Rust would be a good addition to the yunohost toolkit... it's fast, correct, easy to write/read, and easy to transition to from python because it has full interop to/from python3
and it's not just the language du jour that some people find fancy, it's increasingly being taught and relied upon as a foundational language to replace C/C++
[12:20:20]
<selfhoster1312> it's not like i'm proposing Elixir or Nushell :) :)
[12:20:47]
<selfhoster1312> but i get the fatigue from "this magic tool will save us"
[12:21:33]
<Bram> it's already hard for us to get people to write python for the core, I can't imagine finding people to write rust honestly, in addition to having python interoperability and the whole building toolchain and modifying our .deb generation tools (and I say that while loving rust)
[12:21:36]
<selfhoster1312> in this case i was just using it as a benchmark of wtf are we doing wrong for it to be so slow
[12:21:38]
<Bram> it's just not suited for our current needs
[12:23:16]
<selfhoster1312> Bram, personal anecdata as a total amateur programmer: i find it MUCH EASIER to write Rust than Python, because of cargo docs, and rustc warnings/errors... every time i write a line of python i spend 1 hour debugging whatever i broke on the other side of the source code due to side effects... in 99% of cases in Rust if it compiles it works
[12:23:24]
<selfhoster1312> am i weird? i know other people like me...
[12:24:05]
<Bram> no, you are just used to write a constrained language, it's not the same skill set than writing a dynamic programming language with less seat belts
[12:24:41]
<Salamandar> > <@Bram_:matrix.org> yunohost has a global lock mechanism for those situations
kinda like a GIL :D
[12:24:48]
<selfhoster1312> i don't think so? i wrote amateur php/python for 10 years before learning rust and was used to it... but it always remained a nightmare? maybe my brain is broken 🥸️
[12:25:22]
<Bram> on the other hand most yunohost contributors aren't coming from a developper background but mostly a sysadmin or "I'm hacking my way into linux" people and learning python is way easier for them
[12:25:58]
<selfhoster1312> makes sense
[12:26:29]
<selfhoster1312> anyway i'll keep exploring why regen-conf is so slow by rewriting it in rust :P
[12:26:49]
<Bram> have you already tried to launch a profiler on it ^^'?
[12:26:55]
<selfhoster1312> Bram, maybe you have an idea about my second point on the issue... why are we using hashing at all and not timestamps?
[12:27:08]
<selfhoster1312> nah i'm gonna do that next :) #FlamegraphFTW
[12:27:37]
<Bram> > <selfhoster1312> nah i'm gonna do that next :) #FlamegraphFTW
that's generally the first step because our intuition is nearly always wrong in optimizing
[12:27:58]
<Salamandar> > <selfhoster1312> Bram, maybe you have an idea about my second point on the issue... why are we using hashing at all and not timestamps?
cuz:
* you might be able to edit a file without changing its timestamp (yeah yeah OS allows that)
* you might have the SAME CONTENT with a changed timestamp (restore a backup, touch file, etc)
[12:28:19]
<selfhoster1312> so true... i started by rewriting a few hooks in Rust and figured i only gained like -25% execution time... so that's why i started looking at the regenconf code itself :D
[12:29:05]
<selfhoster1312> Salamandar, how common is the first case?
[12:29:18]
<selfhoster1312> i know it's technically possible but i would place it on the "you shot yourself in the foot" shelf
[12:29:24]
<Émy - OniriCorpe> regenconf is really the last thing i would complain on the slowness subject x)
[12:29:35]
<selfhoster1312> Émy - OniriCorpe, feel free to suggest other areas i should investigate :p
[12:30:17]
<Bram> from the last time I did profiling the building or the argparse madness to parse the CLI argument was one of the slowest part of yunohost
[12:30:23]
<Bram> also ... slapd ™
[12:30:36]
<Bram> this stuff is so slow to answer I never knew why and I hate it
[12:30:50]
<Bram> also: avoid looking at its source code, it's like 90's style C
[12:31:13]
<selfhoster1312> from what i could see slapd is really fast except the modify_ext_s query?
[12:31:21]
<selfhoster1312> i think we could work around it
[12:31:44]
<selfhoster1312> the argparse would be moulinette?
[12:31:53]
<Bram> yes
[12:32:32]
<Bram> https://github.com/YunoHost/moulinette/blob/dev/moulinette/actionsmap.py this whole madness
[12:33:13]
<selfhoster1312> buuuut about LDAP i would really like if we could stop using it as a source of truth and consider it an export instead
[12:33:43]
<selfhoster1312> last week i was thinking, maybe it's a terrible idea, but maybe we could have a redis instance to store yunohost global state in RAM and allow access from multiple processes
[12:34:08]
<Émy - OniriCorpe> > <selfhoster1312> Émy - OniriCorpe, feel free to suggest other areas i should investigate :p
by order of importance (for me): opening in the webadmin:
- the log list (which becomes exponentially slower as log files accumulate)
- the application list (whish is slow with my ~40 installed apps)
- same for the services list
- a domain name management page (not the domain name list, which is fast)
[12:34:18]
<kayou> I remember when i wrote unit test for moulinette, that argparse gaves me such a headache
[12:34:50]
<selfhoster1312> Bram, do you have an example code/command i could run without yunohost installed to benchmark the actionsmap.py ? or should i write one?
[12:35:08]
<Bram> > <@kayou:matrix.org> I remember when i wrote unit test for moulinette, that argparse gaves me such a headache
I often think we would be better and faster writing our own parser but oh gosh I'm not sure we want to do that
[12:35:09]
<kayou> > <@oniricorpe:im.emelyne.eu> by order of importance (for me): opening in the webadmin:
> - the log list (which becomes exponentially slower as log files accumulate)
> - the application list (whish is slow with my ~40 installed apps)
> - same for the services list
> - a domain name management page (not the domain name list, which is fast)
I think it's because of the webadmin, not the yunohost core system
[12:35:45]
<kayou> > <selfhoster1312> Bram, do you have an example code/command i could run without yunohost installed to benchmark the actionsmap.py ? or should i write one?
https://github.com/YunoHost/moulinette/blob/dev/test/test_actionsmap.py
[12:35:47]
<kayou> ¯\_(ツ)_/¯
[12:36:06]
<Émy - OniriCorpe> > <@kayou:matrix.org> I think it's because of the webadmin, not the yunohost core system
webadmin is core :P
[12:37:14]
<kayou> I see it like a kind of an app "above" the core
[12:37:48]
<kayou> which is not 100% true 🤡
[12:42:22]
<selfhoster1312> thx
[12:43:32]
<selfhoster1312> can we maybe get an "Optimization" label on github issues? so i don't label things as bug/featurerequest
[12:45:13]
<Émy - OniriCorpe> enhancements? refactor?
[12:45:52]
<Émy - OniriCorpe> ```
time yunohost log list -l 999999
...
real 0m44,728s
user 0m8,680s
sys 0m0,966s
```
:"3
[12:46:12]
<selfhoster1312> ouch, how is that even possible
[12:47:03]
<kayou> we need a profiler :)
[12:47:13]
<Émy - OniriCorpe> logs are not automatically purged
[12:47:43]
<Bram> I used to work on a "vaccum" command for logs but I lost the code in a hard drive crash I think >_>
[12:47:48]
<selfhoster1312> Émy - OniriCorpe, i think i found the "bug"
[12:47:53]
<Bram> and now I'm too tired by $day_job to work on it again :<
[12:48:01]
<selfhoster1312> it seems metadata is parsed from disk even if details are not requested
[12:48:10]
<Émy - OniriCorpe> > <@oniricorpe:im.emelyne.eu> logs are not automatically purged
i removed by hand all my logs before 2024-01-01 and right now i have exactly 736 log files
[12:48:10]
<Yunohost Git/Infra notifications> [issues] chri2 labeled :space_invader: bug on [issue #2393](https://github.com/YunoHost/issues/issues/2393): if an account app exists app will not install
[12:48:13]
<Yunohost Git/Infra notifications> [issues] chri2 opened [issue #2393](https://github.com/YunoHost/issues/issues/2393): if an account app exists app will not install
[12:49:37]
<selfhoster1312> 0m6,737s vs 0m0,396s here
[12:50:20]
<selfhoster1312> Émy - OniriCorpe, can you try adding this before `try: metadata =` in /usr/lib/python3/dist-packages/yunohost/log.py:
```
if not with_details and not with_suboperations:
operations[base_filename] = entry
continue
```
[12:50:25]
<selfhoster1312> on line 112 here
[12:51:10]
<Émy - OniriCorpe> luckily, the webadmin cap the number of requested logs to 25 https://github.com/YunoHost/yunohost-admin/blob/4aec50c0b9feeb442a8b9e34eb526be7c5ce00f5/app/src/views/tool/ToolLogs.vue#L36
[12:51:50]
<selfhoster1312> then time it again 😎️
[12:52:04]
<Émy - OniriCorpe> "try: metadata =" not found
[12:52:13]
<selfhoster1312> nah it's not on the same line
[12:52:17]
<selfhoster1312> it was a visual cue :P
[12:52:43]
<Bram> (it would be a good contribution to add a `--profile` CLI argument to the whole YunoHost)
[12:53:27]
<Émy - OniriCorpe> https://aria.im/_matrix/media/v1/download/im.emelyne.eu/HCkVNVoUOdUdDgoXcaghRIXQ
[12:53:29]
<Émy - OniriCorpe> like that?
[12:53:58]
<selfhoster1312> yes in the log_list function
[12:54:26]
<Émy - OniriCorpe> hmmmm yeah
real 0m1,037s
user 0m0,808s
sys 0m0,202s
[12:54:31]
<selfhoster1312> that's better :)
[12:54:34]
<selfhoster1312> still SUPER slow
[12:54:37]
<selfhoster1312> but not NIGHTMARE slow
[12:54:49]
<selfhoster1312> Émy - OniriCorpe, wanna make the PR?
[12:54:53]
<Émy - OniriCorpe> BUUUUT
[12:55:06]
<selfhoster1312> but?
[12:55:21]
<Émy - OniriCorpe> before i had 736 listed log files
[12:55:26]
<Émy - OniriCorpe> with your edit i have 1838
[12:55:35]
<kayou> this doesn't handle [this subcase](https://github.com/YunoHost/yunohost/blob/498b8d9c43fc4782e2a009070ba5c2d731710be4/src/log.py#L129), but idk if it's important
[12:55:54]
<kayou> ha, it seems important :p
[12:56:16]
<selfhoster1312> ah, correct :)
[12:56:31]
<Émy - OniriCorpe> ah, its duplicating each log by 3
[12:56:34]
<Émy - OniriCorpe> https://aria.im/_matrix/media/v1/download/im.emelyne.eu/gXoECCsOMYqBIjHGLpRarJpm
[12:58:14]
<kayou> which one is duplicate? path are not the same
[12:58:23]
<kayou> > <@kayou:matrix.org> this doesn't handle [this subcase](https://github.com/YunoHost/yunohost/blob/498b8d9c43fc4782e2a009070ba5c2d731710be4/src/log.py#L129), but idk if it's important
but it's probably because of that^
[12:58:56]
<selfhoster1312> yes it's definitely related
[12:59:05]
<selfhoster1312> too bad we have to parse an entire yml file just to get a reference to a parent
[12:59:08]
<selfhoster1312> can't we use a symlink?
[12:59:58]
<kayou> a way to speedup this would be to store metatada into a database, but I can already hear Alek s screaming :)
[13:00:12]
<Émy - OniriCorpe> sorry i have the cognition and attention span of a gravel stone for now, i can't process all those lines and understand wich one are the good ones
[13:00:58]
<Émy - OniriCorpe> or automatically purge logs and keep only like the 100 last ones
[13:01:05]
<Émy - OniriCorpe> :"3
[13:02:03]
<selfhoster1312> kayou, a way to speedup this is to optimize the data structure so we don't need disk reads for nothing, and use a fast programming language like Rust, but i can already hear everyone screaming :)
[13:02:11]
<selfhoster1312> i'll open an issue about this
[13:02:36]
<Émy - OniriCorpe> No parsing 1000 logs issue if there is no 1000 logs
[13:02:45]
<kayou> +1 for the data structure, -1 for the language^^
[13:03:00]
<selfhoster1312> Émy - OniriCorpe, that's like a bandaid on a cancer
[13:03:03]
<selfhoster1312> computers are FAST
[13:03:05]
<selfhoster1312> programs are slow
[13:03:41]
<Émy - OniriCorpe> Absolutely no one needs an app upgrade log from last year 🙂↕️
[13:03:42]
<selfhoster1312> your computer can display billions of polygons with applied shaders at 60fps but can't list 1000 files in 1s ???? u_u
[13:05:17]
<Émy - OniriCorpe> yes, certainly, but "done is better than perfect" and i'm still right about the irrelevant nature of having thousands old logs in stock x)
[13:05:35]
<selfhoster1312> i'm not saying you're wrong ^^
[13:06:11]
<selfhoster1312> i'm just shocked at how slow software has become over the years despite computers being so fucking fast
[13:07:24]
<Yunohost Git/Infra notifications> [issues] selfhoster1312 labeled :birthday: feature on [issue #2394](https://github.com/YunoHost/issues/issues/2394): yunohost log list is super slow
[13:07:24]
<Yunohost Git/Infra notifications> [issues] selfhoster1312 opened [issue #2394](https://github.com/YunoHost/issues/issues/2394): yunohost log list is super slow
[13:07:38]
<Émy - OniriCorpe> the software isn't slow, that's my 5200 rpm HDD that is slow lmao
[13:08:05]
<selfhoster1312> 20 years ago 5200rpm was plenty fast :)
[13:08:11]
<selfhoster1312> the HDD hasn't gotten slower :P
[13:09:13]
<selfhoster1312> and it was taking >6s on my freaking fast SSD u_u
[13:11:11]
<Aleks (he/him/il/lui)> > <@Bram_:matrix.org> I often think we would be better and faster writing our own parser but oh gosh I'm not sure we want to do that
we should just have the CLI be a client of the webapi .... (assuming we properly design what happens when the webapi is down for some reason..)
[13:12:07]
<selfhoster1312> ^ that sounds like a great idea that would fix a lot of problems of sync/lock/cache
[13:12:14]
<kayou> > <selfhoster1312> and it was taking >6s on my freaking fast SSD u_u
what if you spin your SSD at 7200rpm?
[13:12:45]
<selfhoster1312> no sorry it's python not spinning *rust* ;) ;) ;)
[13:13:58]
<selfhoster1312> sooo please open issues if stuff is slow, i'd be glad to take a look... i'm a bad programmer but i'm not a bad algorithmician ;)
[13:14:29]
<Aleks (he/him/il/lui)> https://imgs.xkcd.com/comics/is_it_worth_the_time_2x.png for future reference
[13:15:16]
<selfhoster1312> 🥰️
[13:16:34]
<Bram> > <@Alekswag:matrix.org> we should just have the CLI be a client of the webapi .... (assuming we properly design what happens when the webapi is down for some reason..)
you still need to parse the arguments somehow, this won't change it :/
[13:17:33]
<Aleks (he/him/il/lui)> > <selfhoster1312> sooo please open issues if stuff is slow, i'd be glad to take a look... i'm a bad programmer but i'm not a bad algorithmician ;)
to me one of the biggest thing is the regenconf, but please don't try to optimize/profile small subparts like "list pending" or whatever, you should look at the whole global regen conf, ideally in a context like a RPi, where you'll see it takes O(30s), and overall the biggest stuff is about running `yunohost settings get` or similar stuff, who itself take most of their time indeed from argparse, and also from loading python libs, and the long-term foreseen solution is having CLI-as-an-API-client
[13:17:57]
<Aleks (he/him/il/lui)> > <@Bram_:matrix.org> you still need to parse the arguments somehow, this won't change it :/
yeah but you don't have to start python, load lib, load the actionsmap cache, call argparse etc
[13:18:35]
<Aleks (he/him/il/lui)> > <@Alekswag:matrix.org> to me one of the biggest thing is the regenconf, but please don't try to optimize/profile small subparts like "list pending" or whatever, you should look at the whole global regen conf, ideally in a context like a RPi, where you'll see it takes O(30s), and overall the biggest stuff is about running `yunohost settings get` or similar stuff, who itself take most of their time indeed from argparse, and also from loading python libs, and the long-term foreseen solution is having CLI-as-an-API-client
also there are already some optimization in bookworm
[13:19:05]
<selfhoster1312> Aleks (he/him/il/lui), yeah i noted about `yunohost settings get` i'll get to it tonight or tomorrow... but can you confirm my feeling that there's no locking mechanism in there?
[13:19:25]
<Bram> > <@Alekswag:matrix.org> yeah but you don't have to start python, load lib, load the actionsmap cache, call argparse etc
I don't see how you won't have to do that to generate the CLI arguments list to parse '-' (but you'll remove a lot of loading yes)
[13:19:31]
<Aleks (he/him/il/lui)> https://github.com/YunoHost/yunohost/commit/16391d7374def89fbe5fb336bf79b4c184ac3fb1
[13:20:05]
<Aleks (he/him/il/lui)> i think there's no lock mechanism anymore for this because that's a read operation but i'm not sure what you're referring to exactly
[13:20:15]
<Aleks (he/him/il/lui)> > <@Bram_:matrix.org> I don't see how you won't have to do that to generate the CLI arguments list to parse '-' (but you'll remove a lot of loading yes)
ah yes i see what you mean
[13:20:21]
<selfhoster1312> `<<< "$YNH_SETTINGS"` sounds better already :)
[13:20:24]
<Aleks (he/him/il/lui)> hmmm
[13:20:41]
<selfhoster1312> well i'd like to at least read the settings but i don't see how to make sure it's not being overwritten
[13:21:20]
<Aleks (he/him/il/lui)> still not seeing what you mean x_x
[13:21:48]
<selfhoster1312> how do i know if yunohost is currently writing into /etc/yunohost/settings.yml ?
[13:22:20]
<Aleks (he/him/il/lui)> you assume it aint :')
[13:22:29]
<selfhoster1312> ok that's what i thought ^^
[13:22:49]
<Aleks (he/him/il/lui)> it's not like there are a gazillion yunohost commands running at the same time and you need to think about concurrency
[13:22:57]
<Aleks (he/him/il/lui)> you're thinking too low-level
[13:23:04]
<Aleks (he/him/il/lui)> yunohost is essentially high-level glue
[13:24:48]
<Aleks (he/him/il/lui)> i mean yunohost is supposed to be something that only does stuff when the admin wants to perform admin task, it's not like it needs to be optimized to be fast at machine scale, it just need to be fast at human scale, e.g. it would be just fine if regen conf took 5ish second instead of 30
[13:26:11]
<Aleks (he/him/il/lui)> i mean it's not like we're developping Dota II where clicking your character taking 50ms instead of 60ms is a big deal
[13:26:33]
<selfhoster1312> well i'm thinking low-level because i'm reimplementing small bits at a time to see what optimization it amounts to... but i have hopes in the big picture it makes enough of a difference that we can run tests in a few minutes and not a few hours :P
[13:26:37]
<selfhoster1312> maybe i'm too optimistic ^^
[13:28:24]
<Aleks (he/him/il/lui)> i mean i expect switching from md5 to mtime to effectively have 0 measurable impact really
[13:28:56]
<selfhoster1312> really? is this a bet? 🍿️
[13:29:22]
<selfhoster1312> especially on a raspberry pi where disk IO is super slow i bet we can do x10 speed
[13:29:37]
<Aleks (he/him/il/lui)> no, because i know you can probably find a stupid command + context that never happens in real life where you'll see whatever stuff happening
[13:29:41]
<selfhoster1312> because md5 requires to read ENTIRE file
[13:29:46]
<Salamandar> md5 is really really fast and we're talking about very small files
[13:30:02]
<Salamandar> yeah but small files
[13:30:28]
<Aleks (he/him/il/lui)> just setup a damn ynh on a RPi on bookworm, add 5-ish domain, run `yunohost tools regen-conf --debug` and honestly md5 vs mtime is going to be the least of your concern
[13:30:45]
<selfhoster1312> Aleks (he/him/il/lui), yes that's part of my plan i'm just not there yet ^^
[13:30:51]
<Aleks (he/him/il/lui)> and even `tools regen-conf` is not like something that is run super often either
[13:30:52]
<selfhoster1312> (i do have a raspberry pi specially for this purpose)
[13:30:53]
<Salamandar> https://aria.im/_matrix/media/v1/download/matrix.org/cExgWjGKoobmXgbqFzFcJuyK
[13:32:08]
<Aleks (he/him/il/lui)> cf the benefit of optimizing it compared to, for example, not having to compile apps locally etc
[13:33:45]
<selfhoster1312> i add users more often than i compile new versions :)
but we should optimize both :P
[13:34:16]
<selfhoster1312> Salamandar, not a raspi ?
[13:48:05]
<Aleks (he/him/il/lui)> well sometimes "having to compile locally" even means you cant install the app because it needs e.g. 2 or 4GB of RAM to npm build and you don't have that RAM
[15:16:50]
<Salamandar> > <selfhoster1312> Salamandar, not a raspi ?
Indeed its not, but still
[15:50:42]
<selfhoster1312> on raspberry pi:
```
# time yunohost log list
operation:
description: Postinstall your YunoHost server
name: 20231105-135436-tools_postinstall-raspberrypi.local
path: /var/log/yunohost/categories/operation/20231105-135436-tools_postinstall-raspberrypi.local.yml
started_at: 2023-11-05 15:54:36
real 0m9.188s
```
[15:50:50]
<selfhoster1312> yes that's 9s for listing 1 log entry
[15:52:45]
<selfhoster1312> https://xmpp-upload.kl.netlib.re/upload/URgIgJe4IOvQ4764/3c503b7e-2627-404d-9b0d-7108b587f1a4.png
[16:12:26]
<selfhoster1312> https://xmpp-upload.kl.netlib.re/upload/qm302qALmU6w5946/4cea7556-7c06-4c93-bb4b-6085aaabdc97.png
[16:13:43]
<Yunohost Git/Infra notifications> [issues] selfhoster1312 [commented](https://github.com/YunoHost/issues/issues/2394#issuecomment-2115655666) on [issue #2394](https://github.com/YunoHost/issues/issues/2394) yunohost log list is super slow: Tested on a raspberry pi with 1 single log entry: root@raspberrypi:~# time yunohost log list operation: 0: de...
[16:16:14]
<Yunohost Git/Infra notifications> [issues] selfhoster1312 edited [issue #2392](https://github.com/YunoHost/issues/issues/2392): regenconf is super slow
[16:17:00]
<Yunohost Git/Infra notifications> [issues] selfhoster1312 [commented](https://github.com/YunoHost/issues/issues/2392#issuecomment-2115663056) on [issue #2392](https://github.com/YunoHost/issues/issues/2392) regenconf is super slow: Added benchmarks from an actual Raspberry pi running the two versions (Python/Rust).
[16:18:47]
<selfhoster1312> so on a SSD that version is 50-100x faster, but on a raspi microSD it's 200-500x faster 😎️
[16:19:24]
<Aleks (he/him/il/lui)> https://i.imgflip.com/8qa56w.jpg
[16:21:32]
<selfhoster1312> sorry i'm tempted to laugh but am i actually bothering you with this "stuff is slow let's make it fast" ?
[16:21:57]
<selfhoster1312> i could just stop talking about it until i have an actual PR with huge wins ^^"
[16:26:50]
<Aleks (he/him/il/lui)> i don't know i'm just confused because every command you try is super slow like "yeah it takes X seconds to run any yunohost command" and i'm not really surprised but then let's either do actual profiling and optimize stuff (which I do from time to time, at least obvious stuff like lazy-loading libs which are only important in specific cases)
but at the same time i'm seeing "i've recoded the regenconf using rust", which i highly doubt because the regenconf is mostly bash commands so wtf
[16:26:59]
<Aleks (he/him/il/lui)> i don't even know what it means "recoded in rust"
[16:27:16]
<Aleks (he/him/il/lui)> i mean we just ain't gonna include rust for the reason bram mentionned, period x_x
[16:28:24]
<Aleks (he/him/il/lui)> >Two orders of magnitude of slowness (x100) is quite a big deal! :(
as i was trying to say, what matters is what it feels like for a human ... for a human, there's no difference between 0.1s and 0.001s ...
[16:28:41]
<selfhoster1312> i haven't finished remaking regenconf yet (just small parts), but i'll have code online soon enough to give you an idea
[16:28:48]
<Aleks (he/him/il/lui)> it's not like we're gonna do a `for i in range(1, 1000000): hook_list()`
[16:31:29]
<selfhoster1312> sure, but the small wins do add up in the grand scheme of things...
[16:34:23]
<selfhoster1312> i mean i'm approaching this from a viewpoint of "WOW IT'S INCREDIBLE HOW FAST WE CAN BE" but if you're not amazed but just annoyed at some point please let me know
[16:36:54]
<Aleks (he/him/il/lui)> to me when you want to work on perf optimization, one should profile the biggest picture and work on where the gain is the most obvious
i mean great if you can improve $randompartofthecode from 0.1 to 0.001s but if loading the libs is taking 3s this is a 3% improvement vs if we can go from 3s to 2s loading libs, this is a 30% improvement
[18:18:15]
<Navan> Silly idea but is there an OpenAPI definition for the YunoHost web API?
[18:40:10]
<Aleks (he/him/il/lui)> hmyes
[18:42:30]
<Aleks (he/him/il/lui)> you can run this script https://github.com/YunoHost/yunohost/blob/dev/doc/generate_api_doc.py
[18:42:40]
<Aleks (he/him/il/lui)> i thought it was in the public doc somewhere somehow
[18:46:17]
<Navan> https://aria.im/_matrix/media/v1/download/local.beeper.com/navan_g7O8kzIM7zDnuyi7PEPu043wg6k5jqefn9eOPQ2gwl5LpmyRETmptrRmXQYD9Wtw
[18:46:54]
<Navan> might have to patch this before I start working on a client
[18:47:46]
<Aleks (he/him/il/lui)> what's "this" ..?
[18:48:23]
<Navan> the scripts being used to generate openapi.json
https://github.com/YunoHost/yunohost/tree/dev/doc
[18:50:19]
<Navan> If I am already going to be working on fixing this, should I try and migrate from 3.0.3 to 3.1?
[18:54:47]
<Aleks (he/him/il/lui)> i don't know how you managed to get this, it took me 5 min to realize one should run `./ynh-dev rebuild-api-doc`
[18:54:55]
<Aleks (he/him/il/lui)> and i don't reproduce your screenshot
[18:55:33]
<Aleks (he/him/il/lui)> https://github.com/YunoHost/ynh-dev/blob/master/ynh-dev#L570
[18:56:23]
<Aleks (he/him/il/lui)> > <@Alekswag:matrix.org> i don't know how you managed to get this, it took me 5 min to realize one should run `./ynh-dev rebuild-api-doc`
i mean 10* and i'm the one who wrote it so uh idk what you ran x_x
[18:58:04]
<Navan> I just cloned YunoHost/yunohost, created a virtualenv, installed pyyaml and ran py generate_api_doc.py
[18:58:25]
<Navan> because the python script is reading a static yaml file I didn't think of setting up a dev server
[19:01:19]
<Navan> I then copied the generated openapi.json to an online validator
[19:02:04]
<Navan> Looking at the bash function, it just downloads a copy of swagger-ui so I can use api.html to browse, but it doesn't validate the generated json
[19:02:16]
<Aleks (he/him/il/lui)> ok ?
[19:02:25]
<Aleks (he/him/il/lui)> but how did you "validate" the json ?
[19:02:44]
<Navan> https://editor.swagger.io
[19:02:50]
<Navan> This is an online validator
[19:04:35]
<Navan> https://aria.im/_matrix/media/v1/download/local.beeper.com/navan_jgOKIM3Rv217DWk5EX6nAMumjrO5505vVp9iM3uGb7EYRQfP4dIUc6rmfNdBAGt2
[19:09:24]
<Aleks (he/him/il/lui)> hmkay but how bad is this ? what's good about having a real OpenAPI-whatever compared to the swagger doc ?
[19:10:22]
<Navan> Only benefit is being able to automatically generate clients for any language
[19:10:42]
<Navan> most code generators mess up if the api spec isn't fully valid
[19:11:03]
<Aleks (he/him/il/lui)> oh i see
[19:11:26]
<Aleks (he/him/il/lui)> but doesnt it need like more info about the data structures etc
[19:11:29]
<Aleks (he/him/il/lui)> or entities idk
[19:12:38]
<Navan> I think you can just define all the different data types it might accept
[19:12:50]
<Navan> There is also AnyType
[19:14:45]
<Navan> This isn't high priority at all. How many people even want a native client for the YunoHost api in a language of their choice? The Swagger doc is also more than enough to understand what is generally happening
[19:29:03]
<Yunohost Git/Infra notifications> [yunohost] 🔴 Pipeline [#1291302335](https://gitlab.com/YunoHost/yunohost/-/pipelines/1291302335) failed on branch dev
[19:36:49]
<Yunohost Git/Infra notifications> [yunohost] 🔴 Pipeline [#1289318919](https://gitlab.com/YunoHost/yunohost/-/pipelines/1289318919) failed on branch dev
[19:38:38]
<Yunohost Git/Infra notifications> [issues] tituspijean labeled :birthday: feature on [issue #2395](https://github.com/YunoHost/issues/issues/2395): Add a diagnosis check for infamous "Wi-Fi is currently blocked by rfkill." on Raspbian
[19:38:38]
<Yunohost Git/Infra notifications> [issues] tituspijean opened [issue #2395](https://github.com/YunoHost/issues/issues/2395): Add a diagnosis check for infamous "Wi-Fi is currently blocked by rfkill." on Raspbian
[19:38:38]
<Yunohost Git/Infra notifications> [issues] tituspijean assigned tituspijean on [issue #2395](https://github.com/YunoHost/issues/issues/2395): Add a diagnosis check for infamous "Wi-Fi is currently blocked by rfkill." on Raspbian
[19:46:07]
<Yunohost Git/Infra notifications> [yunohost] 🔴 Pipeline [#1286976675](https://gitlab.com/YunoHost/yunohost/-/pipelines/1286976675) failed on branch dev
[19:47:30]
<Yunohost Git/Infra notifications> [yunohost] 🔴 Pipeline [#1286877487](https://gitlab.com/YunoHost/yunohost/-/pipelines/1286877487) failed on branch handle-metronome-as-an-app
[19:56:21]
<Yunohost Git/Infra notifications> [yunohost] 🔴 Pipeline [#1282591241](https://gitlab.com/YunoHost/yunohost/-/pipelines/1282591241) failed on branch dev
[20:12:32]
<Yunohost Git/Infra notifications> Thatoo forked yunohost to [Thatoo/yunohost](https://github.com/Thatoo/yunohost)
[20:16:06]
<Yunohost Git/Infra notifications> [yunohost] Thatoo opened [pull request #1839](https://github.com/YunoHost/yunohost/pull/1839): Update ynh_mysql_dump_db in mysql helper
[20:35:36]
<Yunohost Git/Infra notifications> [yunohost] Thatoo opened [pull request #1840](https://github.com/YunoHost/yunohost/pull/1840): Update ynh_mysql_connect_as in mysql helper
[21:00:19]
<Yunohost Git/Infra notifications> [yunohost] tituspijean created new branch add_rfkill_diagnosis
[21:00:20]
<Yunohost Git/Infra notifications> [yunohost] tituspijean pushed 1 commit to add_rfkill_diagnosis: Add diagnoser about rfkill blocking Wi-Fi card ([b681b1eb](https://github.com/YunoHost/yunohost/commit/b681b1eb62839f3538aca0b2b7b80f06c3f6f98a))
[21:00:26]
<Yunohost Git/Infra notifications> [yunohost] 🔴 Pipeline [#1286916112](https://gitlab.com/YunoHost/yunohost/-/pipelines/1286916112) failed on branch dev, add_rfkill_diagnosis
[21:09:39]
<Yunohost Git/Infra notifications> [yunohost] tituspijean pushed 1 commit to add_rfkill_diagnosis: Enhance strings for rfkill diagnoser ([a139c3c0](https://github.com/YunoHost/yunohost/commit/a139c3c00a0ae46d1ca7fc1bfac672079212114c))
[21:11:55]
<Yunohost Git/Infra notifications> [yunohost] 🔴 Pipeline [#1282586191](https://gitlab.com/YunoHost/yunohost/-/pipelines/1282586191) failed on branch OniriCorpe-patch-1
[21:13:38]
<Yunohost Git/Infra notifications> [yunohost] tituspijean opened [pull request #1841](https://github.com/YunoHost/yunohost/pull/1841): Add diagnoser about rfkill blocking Wi-Fi card
[21:16:25]
<Yunohost Git/Infra notifications> [yunohost] github-advanced-security[bot] [commented](https://github.com/YunoHost/yunohost/pull/1841#discussion_r1604051088) on pull request #1841 Add diagnoser about rfkill blocking Wi-Fi card: ## Explicit returns mixed with implicit (fall through) returns
Mixing implicit and explicit returns may indicate an err...
[21:22:12]
<Yunohost Git/Infra notifications> [yunohost] tituspijean pushed 1 commit to add_rfkill_diagnosis: Appease linter ([e77d6694](https://github.com/YunoHost/yunohost/commit/e77d6694ea9e6645c7368f6236b3929f3fc9d3f8))
[21:23:16]
<Yunohost Git/Infra notifications> [yunohost] 🔴 Pipeline [#1287508625](https://gitlab.com/YunoHost/yunohost/-/pipelines/1287508625) failed on branch dev, add_rfkill_diagnosis
[21:23:17]
<Yunohost Git/Infra notifications> [yunohost] ✖️ Pipeline [#1294168000](https://gitlab.com/YunoHost/yunohost/-/pipelines/1294168000) canceled on branch add_rfkill_diagnosis
[22:28:17]
<Yunohost Git/Infra notifications> [yunohost] 🔴 Pipeline [#1286979018](https://gitlab.com/YunoHost/yunohost/-/pipelines/1286979018) failed on branch dev, add_rfkill_diagnosis
[22:39:44]
<Yunohost Git/Infra notifications> [yunohost] 🔴 Pipeline [#1286251327](https://gitlab.com/YunoHost/yunohost/-/pipelines/1286251327) failed on branch dev, add_rfkill_diagnosis
[22:49:18]
<Yunohost Git/Infra notifications> [yunohost] 🔴 Pipeline [#1283083146](https://gitlab.com/YunoHost/yunohost/-/pipelines/1283083146) failed on branch dev, add_rfkill_diagnosis
[23:57:55]
<Yunohost Git/Infra notifications> [yunohost] 🔴 Pipeline [#1294176383](https://gitlab.com/YunoHost/yunohost/-/pipelines/1294176383) failed on branch add_rfkill_diagnosis