Logs: liberachat/#haskell
| 2026-03-23 11:24:17 | <Freakie> | in other words, calculating the data size somehow reduces the live data from 1-2 gb to 20 mb (at the cost of runtimes that increased a thousanfold or something), while the native GC functions seemed to do anything |
| 2026-03-23 11:24:41 | <[exa]> | yap that confirms it :D you force it, the thunks collapse |
| 2026-03-23 11:25:17 | <[exa]> | cf. lazy foldl vs strict foldl' |
| 2026-03-23 11:25:21 | <Freakie> | well I've tried strictifying (even strictifying everything) and it didn't do much |
| 2026-03-23 11:25:24 | → | arandombit joins (~arandombi@2a02:2455:8656:7100:8944:14a8:1bde:2184) |
| 2026-03-23 11:25:24 | × | arandombit quits (~arandombi@2a02:2455:8656:7100:8944:14a8:1bde:2184) (Changing host) |
| 2026-03-23 11:25:24 | → | arandombit joins (~arandombi@user/arandombit) |
| 2026-03-23 11:25:39 | <[exa]> | try running with a profiler to see where the allocations happen |
| 2026-03-23 11:25:45 | <Freakie> | I already know what and where |
| 2026-03-23 11:25:47 | <Freakie> | just not why |
| 2026-03-23 11:25:52 | <[exa]> | ah |
| 2026-03-23 11:26:03 | <[exa]> | pastebin? |
| 2026-03-23 11:26:07 | <Freakie> | the constructor tag is called CONSTR_NOCAF but I couldn't find anything on what makes it different from other types |
| 2026-03-23 11:26:24 | <[exa]> | (not sure if the code is public-ish :D) |
| 2026-03-23 11:26:33 | <Freakie> | well the repo is private |
| 2026-03-23 11:26:39 | <Freakie> | do you want the code snippets or profiling? |
| 2026-03-23 11:27:03 | <[exa]> | if you know which snippet causes it and you can paste it, it might help |
| 2026-03-23 11:27:13 | <Freakie> | on it |
| 2026-03-23 11:30:07 | <[exa]> | btw CONSTR_NOCAF is iirc "some constructor that didn't fit in the specialized tiny cases" so it's basically "just any data" |
| 2026-03-23 11:30:47 | × | weary-traveler quits (~user@user/user363627) (Remote host closed the connection) |
| 2026-03-23 11:30:58 | <Freakie> | just didn't know if the tag communicated something that might be used to troubleshoot or not |
| 2026-03-23 11:31:02 | <Freakie> | https://pastebin.com/uB36SXJ2 hope this is enough context |
| 2026-03-23 11:31:37 | <Freakie> | the function in question is called at the end of a loop iteration to transfer relevant information from a priority queue to the output |
| 2026-03-23 11:31:50 | × | traxex quits (traxex@user/traxex) (Ping timeout: 245 seconds) |
| 2026-03-23 11:31:59 | <[exa]> | ahhh that's this code, I see |
| 2026-03-23 11:32:59 | <Freakie> | the let bindings were an attempt to eliminate thunks but i dont think they do anything |
| 2026-03-23 11:33:51 | <Freakie> | if you give me a sec i can show some profiling as well |
| 2026-03-23 11:34:03 | <[exa]> | that seems flawless; maybe add a ! in front of `x` there becasue that's the only thing that might get thunkificated |
| 2026-03-23 11:35:01 | <Freakie> | about 90% of the live data from heap profiling is `UnreducedArc`s though |
| 2026-03-23 11:35:05 | <[exa]> | also you can use strict tuples (Data.Strict.Tuple) instd of the (# #) primitives, it's probably going to be the same but the strict tuples are somewhat less syntactically invasive |
| 2026-03-23 11:35:34 | <Freakie> | the unboxed tuples are hell yeah but it's less the strictness and more the allocations themselves that bothered me |
| 2026-03-23 11:35:45 | <Freakie> | i've been pretty conservative with doing it though |
| 2026-03-23 11:36:19 | <Freakie> | I guess in theory the argument could be the issue |
| 2026-03-23 11:36:25 | → | fp joins (~Thunderbi@130.233.70.34) |
| 2026-03-23 11:37:54 | <[exa]> | oh no PtrNode actually forces the `x`, shouldn't be an issue |
| 2026-03-23 11:39:20 | <Freakie> | any easy to way share png/svgs? I guess just imgur |
| 2026-03-23 11:39:48 | <[exa]> | imgur is ok |
| 2026-03-23 11:40:04 | <[exa]> | not sure if there are any better nowadays |
| 2026-03-23 11:40:30 | <Freakie> | https://imgur.com/a/yiZl0CN |
| 2026-03-23 11:40:51 | ← | oskarw parts (~user@user/oskarw) (ERC 5.6.1 (IRC client for GNU Emacs 30.2)) |
| 2026-03-23 11:41:42 | <Freakie> | not sure what the STACK is unless it's just a plain old program stack |
| 2026-03-23 11:42:50 | <Freakie> | i'm running a slightly larger input that shows the behavior more clearly |
| 2026-03-23 11:43:03 | <[exa]> | continuation stack ("list of function invocations waiting for forced results") |
| 2026-03-23 11:43:06 | → | CiaoSen joins (~Jura@p549cb690.dip0.t-ipconnect.de) |
| 2026-03-23 11:43:25 | <[exa]> | kinda hints that there's really an actual very deep force at the end |
| 2026-03-23 11:43:35 | <Freakie> | https://imgur.com/a/e0uZSwU |
| 2026-03-23 11:43:55 | × | karenw quits (~karenw@user/karenw) (Ping timeout: 245 seconds) |
| 2026-03-23 11:44:13 | <Freakie> | the first image had about 50 mb max residency profiled, this one has 900 mb |
| 2026-03-23 11:44:36 | <Freakie> | so it looks like the stack gets kinda irrelevant on bigger data |
| 2026-03-23 11:45:00 | <[exa]> | this looks like it might actually be exploding algorithmically |
| 2026-03-23 11:45:07 | <[exa]> | as in, you're doing BFS there basically, right? |
| 2026-03-23 11:45:23 | <Freakie> | no, a product combination of two lists |
| 2026-03-23 11:45:36 | → | DragonMaus joins (~DragonMau@user/dragonmaus) |
| 2026-03-23 11:45:36 | <Freakie> | they contain nodes ordered by level |
| 2026-03-23 11:45:59 | <[exa]> | can you try tracing out the length of the LPQueue? Just to be super sure. |
| 2026-03-23 11:46:10 | <Freakie> | I have before, it's not that suspicious |
| 2026-03-23 11:46:16 | <[exa]> | ok |
| 2026-03-23 11:46:25 | <Freakie> | it contains `Arc1`s btw |
| 2026-03-23 11:46:37 | <[exa]> | might be that it copies them too much or something |
| 2026-03-23 11:47:07 | <Freakie> | the peaks are a result of the way the inputs are generated so that's not really an issue |
| 2026-03-23 11:47:33 | <Freakie> | and the valleys correspond to one full iteration of the algorithm (which is then stored to be combined with the next input) |
| 2026-03-23 11:48:45 | <Freakie> | the priority queues and outputs should be roughly equal in complexity size, and the ridges show how much data is actually carried into the next iteration, as far as I can tell |
| 2026-03-23 11:48:59 | <[exa]> | yeah that looks sensible |
| 2026-03-23 11:49:05 | <[exa]> | which kind of queue is the LPQueue? |
| 2026-03-23 11:49:09 | <Freakie> | Arc1s are fully expected to take a lot of space because they contain unresolved information |
| 2026-03-23 11:49:13 | <Freakie> | in this case it's just a list of lists |
| 2026-03-23 11:50:17 | <Freakie> | but this is a refinement from a previous version that used two priority queues of either Data.Set or a min heap from the pqueue package |
| 2026-03-23 11:51:17 | <[exa]> | ok I assume the queue is not to blame then |
| 2026-03-23 11:51:26 | <[exa]> | weird. |
| 2026-03-23 11:51:30 | <Freakie> | no, but again the queues do not contain `UnreducedArc`s which are the issue |
| 2026-03-23 11:52:01 | <Freakie> | what just fucks me is that manually forcing garbage collections does nothing (except explode the runtime) but measuring *anything* with GHC.DataSize collapses the memory consumption |
| 2026-03-23 11:52:20 | <[exa]> | how do you "run" the outputInternalArcsPQ1? |
| 2026-03-23 11:52:23 | → | Googulator joins (~Googulato@2a01-036d-0106-2888-7906-f38b-8800-979e.pool6.digikabel.hu) |
| 2026-03-23 11:52:35 | <[exa]> | anyway yeah that's the point of space leaks, if you force evaluate the data, the data disappears |
| 2026-03-23 11:52:47 | <Freakie> | fair point I guess |
| 2026-03-23 11:52:49 | <[exa]> | you might just `deepseq` instead of the DataSize, it will fix itself just as well |
| 2026-03-23 11:53:05 | <Freakie> | I playing around with forcing but maybe I used it on the wrong argument |
| 2026-03-23 11:53:20 | <[exa]> | btw a good strategy is to try to bisect the problem with deepseq in random places |
| 2026-03-23 11:53:32 | <[exa]> | "good" as in "if everything else fails this gives hope" |
| 2026-03-23 11:53:48 | <[exa]> | yeah I suspect there's something that slipped |
| 2026-03-23 11:53:51 | <Freakie> | I feel like "good strategy" and "random places" is a dire symptom of what I'm doing here |
| 2026-03-23 11:54:19 | <Freakie> | but I call the function (as well an equivalent one for another priority queue) just before recursion to the next iteration |
| 2026-03-23 11:54:34 | × | rainbyte quits (~rainbyte@181.47.219.3) (Read error: Connection reset by peer) |
| 2026-03-23 11:55:08 | <Freakie> | https://pastebin.com/Dztw40Dg |
| 2026-03-23 11:55:37 | → | merijn joins (~merijn@host-cl.cgnat-g.v4.dfn.nl) |
| 2026-03-23 11:55:47 | <[exa]> | which function do you use from the DataSize? the recursive one? |
| 2026-03-23 11:55:55 | → | rainbyte joins (~rainbyte@181.47.219.3) |
| 2026-03-23 11:56:35 | <Freakie> | I mean I refactored it back to not use IO because I need purity but I tried playing with both recursiveSize and closureSize |
| 2026-03-23 11:57:23 | <[exa]> | throw in this thing https://hackage.haskell.org/package/deepseq-1.4.8.1/docs/Control-DeepSeq.html#v:force |
| 2026-03-23 11:57:27 | <[exa]> | see if it helps |
| 2026-03-23 11:57:45 | <[exa]> | if it helps, search for more thunks :D |
| 2026-03-23 11:57:48 | <Freakie> | think I'll try seeing if can `seq` my way out of it first |
| 2026-03-23 11:58:25 | <[exa]> | btw it might be useful to get a normal profile from ghc (the one that gets generated from profiling build if you add +RTS -p) |
| 2026-03-23 11:58:39 | <Freakie> | I have as well, it's just incompatible with some of the heap profiling |
| 2026-03-23 11:58:54 | <Freakie> | the .prof output generally suggests that both halves of the algorithm are doing equal work in both time and allocations |
| 2026-03-23 11:58:56 | <[exa]> | yeah well, it points to functions that call the allocations, which MIGHT correlate |
| 2026-03-23 11:59:08 | <Freakie> | but the heap profiling is overwhelmingly biased from this one function |
| 2026-03-23 12:00:10 | × | merijn quits (~merijn@host-cl.cgnat-g.v4.dfn.nl) (Ping timeout: 245 seconds) |
| 2026-03-23 12:04:15 | <Freakie> | huh, profiling with -hC shows two functions that make up 80% of the live data, but -p shows that they only account for like 2% of the allocations |
| 2026-03-23 12:04:32 | <Freakie> | I mean that's obviously because the accumulator is the one leaking but still |
All times are in UTC.