Coding: Fleeting Thoughts

A place to discuss the implementation and style of computer programs.

Moderators: phlip, Moderators General, Prelates

User avatar
Xanthir
My HERO!!!
Posts: 5336
Joined: Tue Feb 20, 2007 12:49 am UTC
Location: The Googleplex
Contact:

Re: Coding: Fleeting Thoughts

Postby Xanthir » Thu Feb 15, 2018 1:05 am UTC

Yeah, a dict just connects keys to values. Values can be anything, keys are anything "hashable" (strings, numbers, tuples of those, and some other things).

An array is basically just a dict that only accept integer keys, and only in a certain range - it just connects the numbers 0, 1, 2, etc to values.

You just use different syntax to define/refer to them. Previous posts already went into detail about what sort of dict you'd want to use - probably with keys that are 3-tuples for the coordinates, like `{(0,0,0): 0, (0,0,1): 0, ...}`; I won't repeat the previous posts here. To get/set them you can still use the `a[foo]` syntax, but it'll look like `a[0,0,1] = 2` (set the key (0,0,1) to the value 2), instead of your current drilling down into a multidimensional array like `a[0][0][1] = 2` (look up the value at index 0, then the value at index 0 of the subarray, then the value at index 1 of the sub-subarray, and set it to 2).
(defun fibs (n &optional (a 1) (b 1)) (take n (unfold '+ a b)))

User avatar
Flumble
Yes Man
Posts: 2082
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Thu Feb 15, 2018 2:45 am UTC

Assuming that cells are either alive or dead, I'd go with sets. Sets are simple things that hold a bunch of items. For example, a set in which each item is the coordinate of a live cell. So the question "is the cell at (x,y,z) alive or dead?" becomes "does the coordinate (x,y,z) exist in the set of alive cells or not?". And birthing/killing cells becomes a matter of adding/deleting coordinates to/from the set.

The example below also adds some functional programming to the mix leading to only a few (>5) lines of code:

Code: Select all

# in haskell typology: getNeighbours :: Coordinate -> Set Coordinate
def getNeighbours(coordinate):
    (x,y,z) = coordinate # python 3 doesn't allow deconstructing a tuple in the parameter definition, so it needs its own line
    return frozenset({(x+dx,y+dy,z+dz) for dx in [-1,0,1] for dy in [-1,0,1] for dz in [-1,0,1] if not dx == dy == dz == 0}) # butifel
    # also, if you want a wrapping field or dead boundaries or whatever, this is the place to go

# in haskell typology: evolveState :: Set Coordinate -> Set Coordinate
def evolveState(oldAlive):
    newAlive = set() # starting with a blank slate (practice may show that copying the old state and adding/deleting some elements performs better)
    activeCells = oldAlive.union(*map(getNeighbours, oldAlive)) # such ugly notation to take the union of a couple of sets (also simply assuming that all live cells and their neighbours may change; this may be optimized)

    for cell in activeCells:
        aliveNeighbours = sum(1 for neighbour in getNeighbours(cell) if neighbour in oldAlive)
        if aliveNeighbours == 5 or (cell in oldAlive and 1 < aliveNeighbours < 8): # copied a rule from the blogosphere
            newAlive.add(cell) # I would've gone for `newAlive = newAlive.union({cell})` if it weren't both less readable and significantly slower because python won't realize that it can reuse the set instead of making a modified copy

    return frozenset(newAlive) # just pretend it was a frozen set all along

def __main__():
    alive = someInitialLiveCells # like `frozenset({(-1, 0, 0), (0, -1, 0), (0, 0, 0), (0, 1, 0), (1, -1, 0), (1, 1, 0)})`

    for generation in range(1, 1001):
        alive = evolveState(alive)
        somehowShowThisState(generation, alive) # like... I dunno, I avoid graphics libraries, and 3D data doesn't translate to text output nicely

It becomes more readable if you remove the comments, except for the oldAlive.union(*map(getNeighbours, oldAlive)), which is alive ∪ { N | N ∈ getNeighbours(A), A ∈ alive } or "grab all the alive cells and all their surrounding cells".
I have no idea about the performance of this thing, though I expect something of an O(n²) time per iteration because of the way the active cells are decided.

User avatar
Xenomortis
Not actually a special flower.
Posts: 1426
Joined: Thu Oct 11, 2012 8:47 am UTC

Re: Coding: Fleeting Thoughts

Postby Xenomortis » Tue Mar 27, 2018 12:30 pm UTC

*Spends a few days writing code at work*

Code: Select all

me@work:project$ sloccount src
...
Total Physical Source Lines of Code (SLOC) = 700
Development Effort Estimate, Person-Years (Person-Months) = 0.14 (1.65)
...
Total Estimated Cost to Develop = $18,578

So where's my cut of that $18k?
Image

Tub
Posts: 409
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Tue Mar 27, 2018 1:09 pm UTC

There's extensive documentation about the calculation and interpretation of those values. I bet you haven't produced a single UML diagram, barely did any testing, and there's zero documentation, so your 700 loc aren't finished yet.

Sloccount is also highly inaccurate in small projects. Writing the first 700 lines for a new project is cheap (In java, you can write 700 loc just by clicking "create new empty project"!). On the other hand, try getting 700 lines of code commited to the linux kernel, see if you can do it in less than 1.65 months.


Then again, $18k sounds much better, so run with it. Apparently I've been producing 2 man-years worth of code in half a man-year, so I'm going to ask for a 300% raise tomorrow.

User avatar
Yakk
Poster with most posts but no title.
Posts: 11092
Joined: Sat Jan 27, 2007 7:27 pm UTC
Location: E pur si muove

Re: Coding: Fleeting Thoughts

Postby Yakk » Tue Mar 27, 2018 1:21 pm UTC

Lines of code have negative value. Every line of code added to our code base is a liability that will cost us money every year until the code base goes under.

Negative lines of code, now that is quality stuff.
One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision - BR

Last edited by JHVH on Fri Oct 23, 4004 BCE 6:17 pm, edited 6 times in total.

User avatar
Xenomortis
Not actually a special flower.
Posts: 1426
Joined: Thu Oct 11, 2012 8:47 am UTC

Re: Coding: Fleeting Thoughts

Postby Xenomortis » Tue Mar 27, 2018 1:58 pm UTC

Tub wrote:There's extensive documentation about the calculation and interpretation of those values. I bet you haven't produced a single UML diagram, barely did any testing, and there's zero documentation, so your 700 loc aren't finished yet.

To be fair, it has had zero testing.
And basically no error handling.
Also ANSI C.

Yakk wrote:Lines of code have negative value. Every line of code added to our code base is a liability that will cost us money every year until the code base goes under.

Negative lines of code, now that is quality stuff.

There's probably more than one git repo at work where my "Lines Added minus Lines Deleted" statistic is negative.
Image

User avatar
raudorn
Posts: 366
Joined: Mon Jun 13, 2011 11:59 am UTC

Re: Coding: Fleeting Thoughts

Postby raudorn » Wed Apr 04, 2018 4:00 pm UTC

Ciber wrote:Lately I have been working on force directed graph layouts. Currently trying to port (is that the right word here?) my naive implementation to numpy because with the 400 node, 900 link test graph I'm using it takes several seconds per iteration.

Uhm, I may be a bit late to the party here, but are you still working on this? I'm asking because I just came out of the end-of-semester slog and both force directed graph layouts and general spatial data structures are still fresh in my mind.

Basically, the best approaches I know boil down to desperately avoiding O(N²) runtime and bring it down to O(N*log(N)). Also cutting the number of iterations down can help, but that usually grows slower than the cost of each iteration anyway.

User avatar
You, sir, name?
Posts: 6983
Joined: Sun Apr 22, 2007 10:07 am UTC
Location: Chako Paul City
Contact:

Re: Coding: Fleeting Thoughts

Postby You, sir, name? » Fri Apr 13, 2018 7:23 pm UTC

I'm between stuff to do at work, so I've been reading papers about Haskell and implementing the ideas as haskell:y as possible in Java.

Works... surprisingly well once you find a middle road between Java's clunkiness and Haskell's weirdness. But now I have a expressive powerful monadic parser library cooking based on Hutton & Meijer's paper. Took some head scratching before I figured out how to express chainl1 and chainr1, but in the end the were pretty clean too.
I edit my posts a lot and sometimes the words wrong order words appear in sentences get messed up.

User avatar
Soupspoon
You have done something you shouldn't. Or are about to.
Posts: 3721
Joined: Thu Jan 28, 2016 7:00 pm UTC
Location: 53-1

Re: Coding: Fleeting Thoughts

Postby Soupspoon » Fri Apr 13, 2018 10:35 pm UTC

I was browsing a computing magazine in the supermarket, the other day (wondering whether to buy it for a couple the whatever-you-call-clickbait-when-it's-on-a-magazine-cover items) and one thing that (subsequent to the original interest) caught my eye was a suggestion to try Ada, as a strong-typed overloadable language that people might prefer to Haskell.

I have to say, Ada was the most tortuous of the strongly-typed languages that I've 'learnt'. It looks like the IDE surrounding the suggested Ada compiler does solve a lot of the issues I had with the (probably) emacs-editing I did, back in the day, but I still have bad memories of its awkwardness, compared to my experience around the same time with Forth, LISP, Fortran and even COBOL. Heck, I'd even prefer to write my missile-flight controller modules1 in Redcode, if I could make it sufficiently Imp-proof!

Anyway, just thought I'd open the vent, slightly, on this long internal simmering disquiet. It quite possibly was the reason I was driven to particularly like Perl, and I'll leave it up to you whether this was ultimately a very good consequence (for me and/or the world at large) or a very bad one (ditto). The Ada of 2018 may even be worth a new try, I suppose, although I've probably forgotten more about it than I ever learnt in the first place (as the post-'80s implemention doubtless was revised and contemperaneously restyled several times to address problems such as I might have identified for myself) so I might effectively be starting from scratch again.


1 The alleged historic reasoning behind its belligerent awkwardness to use. Though I personally think someone didn't like female programmers, and so wanted to blacken Countess Lovelace's name, rather than insult Grace Hopper.

gd1
Posts: 210
Joined: Wed Nov 14, 2012 5:42 am UTC

Re: Coding: Fleeting Thoughts

Postby gd1 » Sun Jul 29, 2018 10:37 am UTC

Dungeon siege 1

[t:template,n:spell_sky_turret_lightning]
{
category_name = "magic";
doc = "spell sky turret lightning";
specializes = base_spell_good;
[aspect]
{
gold_value = 206;
}
[attack]
{
damage_max = 0;
damage_min = 0;
}
[common]
{
description = "Expels a split of lightning at the Target.";
screen_name = "Turret Lightning";
}
[gui]
{
active_icon = b_gui_ig_i_ic_sp_038;
inventory_icon = b_gui_ig_i_ic_sp_038_inv;
}
[magic]
{
attack_damage_modifier_max = ((#magic+1)*3.17+5)*(1+((1/(#magic+8))+.041));
attack_damage_modifier_min = ((#magic+1)*3.17+5)*(1-((1/(#magic+8))+.041));
cast_range = 54;
cast_reload_delay = 0.01;
effect_duration = 500000;
mana_cost = 1;
mana_cost_modifier = (#magic*0)+1;
max_level = 27000;
required_level = 90;
requires_line_of_sight = false;
speed_bias = 1;
target_type_flags = tt_conscious_enemy | tt_unconscious_enemy | tt_breakable;
usage_context_flags = uc_offensive;
cast_sub_animation = 0;
}
[spell_turret]
{
initial_delay = 0.01;
lightning_dur = 0.01;
shot_rate = 0.01;
effect_script = gom_turret_lightning;
charge_effect = gom_turret_lightning_charge;
}
}

Not even remotely fun, but fun to think about.
There is no emotion more useless in life than hate.

Tub
Posts: 409
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Tue Jul 31, 2018 11:43 am UTC

Has anyone found a valid use case for a WeakMap in JavaScript?

It's supposed to prevent memory leaks, by allowing you to attach data to objects without leaking the data. The example for a key is often a DOM element, because those tend to disappear every now and then.

Code: Select all

let wm = new WeakMap();
let e = document.getElementById('foo');
wm.set(e, {
  private_data: 1
})

There. The private data can be reclaimed by the GC when the element disappears, so there's no memory leak.

But there wasn't any memory leak in the legacy variant, either:

Code: Select all

const KEY = '_private_data';
let e = document.getElementById('foo');
e[KEY] = {
  private_data: 1
}

No leak, though the additional property may mess up for...in loops or something; it's not properly hidden. But another ES6 feature does fix that:

Code: Select all

const KEY = Symbol('private data');


So.. why would I use a WeakMap to associate data with an object when I can use a Symbol?


Considering that a WeakMap has no size and is not iterable, I don't see any other use than associating data to an object. It's intentionally impossible to use a weakmap to detect when an object was garbage collected.

User avatar
Xanthir
My HERO!!!
Posts: 5336
Joined: Tue Feb 20, 2007 12:49 am UTC
Location: The Googleplex
Contact:

Re: Coding: Fleeting Thoughts

Postby Xanthir » Tue Jul 31, 2018 4:30 pm UTC

Note that you *can* retrieve all the Symbols attached to an object; they're not "hidden" in any way whatsoever, they're just guaranteed to never collide with anything else unless you specifically use the same Symbol object. So if you really do have some private data to associate, you can't attach it with a Symbol.

That said, *most* of the time using a Symbol or a WeakMap is just a choice of how you want to interact with the data; whether it makes more sense to think of it as a property of the object or as data collected into a map.
(defun fibs (n &optional (a 1) (b 1)) (take n (unfold '+ a b)))

Tub
Posts: 409
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Tue Jul 31, 2018 5:39 pm UTC

I'm aware, but being collision-free and non-enumerable is enough to prevent accidental bugs when multiple separate modules mess with the same object.

You can access or change private members in Java with the reflection API, but we still consider them private. You can mangle symbol properties in JavaScript using getOwnPropertySymbols(), but unless someone hands you the symbol as part of an API contract, you have no idea what the associated data means, and you know you shouldn't access it. That's private enough for me.

I suppose WeakMap has its uses if you use Object.freeze() a lot, or if you want to delete associated data from all objects (just delete the WeakMap). It's just that I have seen zero use cases where it would prevent memory leaks as advertised. It certainly won't help me prevent the memory leaks I'm facing right now.

Tub
Posts: 409
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Tue Jul 31, 2018 8:57 pm UTC

I've done some benchmarks.

Code: Select all

const LOOPS = 10000000;
const ITERATIONS = 10;

(function() {
   const PROP = '_private_data';
   let o = {};
   bench('prop-single',
      function() {
         o[PROP] = {
            a: 1,
            b: 1
         };
      },
      function() {
         let p = o[PROP];
         let sum = p.a + p.b;
         p.a = p.b;
         p.b = sum;
      }
   );
   console.log(o[PROP].a);
})();

(function() {
   const PROP_A = '_private_data_a';
   const PROP_B = '_private_data_b';
   let o = {};
   bench('prop-multi',
      function() {
         o[PROP_A] = 1;
         o[PROP_B] = 1;
      },
      function() {
         let sum = o[PROP_A] + o[PROP_B];
         o[PROP_A] = o[PROP_B];
         o[PROP_B] = sum;
      }
   );
   console.log(o[PROP_A]);
})();


(function() {
   const PROP = Symbol();
   let o = {};
   bench('symbol-single',
      function() {
         o[PROP] = {
            a: 1,
            b: 1
         };
      },
      function() {
         let p = o[PROP];
         let sum = p.a + p.b;
         p.a = p.b;
         p.b = sum;
      }
   );
   console.log(o[PROP].a);
})();

(function() {
   const PROP_A = Symbol();
   const PROP_B = Symbol();
   let o = {};
   bench('symbol-multi',
      function() {
         o[PROP_A] = 1;
         o[PROP_B] = 1;
      },
      function() {
         let sum = o[PROP_A] + o[PROP_B];
         o[PROP_A] = o[PROP_B];
         o[PROP_B] = sum;
      }
   );
   console.log(o[PROP_A]);
})();


(function() {
   let wm = new WeakMap();
   let o = {};
   bench('weakmap-single',
      function() {
         wm.set(o, {
            a: 1,
            b: 1
         });
      },
      function() {
         let p = wm.get(o);
         let sum = p.a + p.b;
         p.a = p.b;
         p.b = sum;
      }
   );
   console.log(wm.get(o).a);
})();

(function() {
   let wm_a = new WeakMap();
   let wm_b = new WeakMap();
   let o = {};
   bench('weakmap-multi',
      function() {
         wm_a.set(o, 1);
         wm_b.set(o, 1);
      },
      function() {
         let sum = wm_a.get(o) + wm_b.get(o);
         wm_a.set(o, wm_b.get(o));
         wm_b.set(o, sum);
      }
   );
   console.log(wm_a.get(o));
})();

function bench(name, init, func) {
   let times = [];
   let t = Date.now();
   for (let i = 0; i < ITERATIONS; i++) {
      init();
      for (let l = 0; l < LOOPS; l++) {
         func();
      }
      let t2 = Date.now();
      times.push(t2 - t);
      t = t2;
   }
   let avg = times.reduce((a, e) => a + e) / times.length;
   console.log(name, times, avg);
}


Code: Select all

prop-single [ 115, 114, 101, 100, 101, 100, 101, 101, 101, 100 ] 103.4
prop-multi [ 332, 309, 290, 293, 292, 294, 292, 293, 284, 293 ] 297.2
symbol-single [ 232, 227, 226, 226, 226, 226, 227, 227, 226, 225 ] 226.8
symbol-multi [ 253, 239, 241, 240, 264, 251, 241, 240, 240, 248 ] 245.7
weakmap-single [ 464, 460, 469, 467, 465, 459, 457, 457, 457, 456 ] 461.1
weakmap-multi [ 4180, 4278, 3888, 3791, 3743, 3762, 3700, 3697, 3856, 3701 ] 3859.6


I'm not sure why the first one is faster, but the WeakMap variants are slowest in all engines I've tested (those numbers are nodejs v10.5)

gd1
Posts: 210
Joined: Wed Nov 14, 2012 5:42 am UTC

Re: Coding: Fleeting Thoughts

Postby gd1 » Fri Aug 31, 2018 4:38 am UTC

Best Coding Language (or operating system I guess):
ThanOS
"Perfectly balanced, as all things should be."
There is no emotion more useless in life than hate.

User avatar
Sizik
Posts: 1224
Joined: Wed Aug 27, 2008 3:48 am UTC

Re: Coding: Fleeting Thoughts

Postby Sizik » Fri Aug 31, 2018 2:34 pm UTC

gd1 wrote:Best Coding Language (or operating system I guess):
ThanOS
"Perfectly balanced, as all things should be."

You'll never run out of hard drive space! As the hard drive gets too full, half of the files are deleted to make room.
gmalivuk wrote:
King Author wrote:If space (rather, distance) is an illusion, it'd be possible for one meta-me to experience both body's sensory inputs.
Yes. And if wishes were horses, wishing wells would fill up very quickly with drowned horses.

User avatar
Moo
Oh man! I'm going to be so rebellious! I'm gonna...
Posts: 6417
Joined: Thu Aug 16, 2007 3:15 pm UTC
Location: Beyond the goblin city
Contact:

Re: Coding: Fleeting Thoughts

Postby Moo » Mon Sep 03, 2018 7:19 am UTC

I don't know what has made me happier - the fact that Scott Hanselman liked and responded to my tweet, or that my boss' boss saw and liked it.

In summary:

Image
Proverbs 9:7-8 wrote:Anyone who rebukes a mocker will get an insult in return. Anyone who corrects the wicked will get hurt. So don't bother correcting mockers; they will only hate you.
Hawknc wrote:FFT: I didn't realise Proverbs 9:7-8 was the first recorded instance of "haters gonna hate"

User avatar
Flumble
Yes Man
Posts: 2082
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Sun Sep 09, 2018 4:53 pm UTC

I was trying some real world™ Haskell the other day and stumbled on monads within monads, specifically IO [IO a] due to reading from files inside a directory. I can slam it all together as a one-liner \f dir -> listDirectory dir >>= traverse (\file -> fmap f (readFile file)) (or in a slightly pointless style \f -> listDirectory >=> traverse (readFile >=> purify f)), but is there a way to cleanly write this in do notation?

Also, is there a way to tell IO to actually close the current file before going on to the next so it doesn't crash because of "too many files open"? To me it's clear that each element in the returned list is independent from the others, so, when one element is eaten (in my case printed) by the function that maps over the IO[a], it can't affect the next element of the traversal so the file can be closed.
Would a deepSeq help hint GHC that the data has fully been extracted from a file after finishing an IO action?

In the end I just flattened the whole thing with unsafePerformIOs, but now I feel dirty.

User avatar
Xanthir
My HERO!!!
Posts: 5336
Joined: Tue Feb 20, 2007 12:49 am UTC
Location: The Googleplex
Contact:

Re: Coding: Fleeting Thoughts

Postby Xanthir » Wed Sep 12, 2018 3:47 pm UTC

Using do notation should just be the obvious "three lines, each pulling the value out of a deeper monad" thing:

Code: Select all

do files <- listDirectory dir
    file <- files
    contents <- readFile file
    doStuffWithContents contents


And yeah, to tell IO to close the file you force strictness. Pulling some example code from SO:

Code: Select all

import Control.Parallel.Strategies (rnf)
-- rnf means "reduce to normal form"
main = do inFile <- openFile "foo"
          contents <- hGetContents inFile
          rnf contents `seq` hClose inFile -- force the whole file to be read, then close
          putStr contents
(defun fibs (n &optional (a 1) (b 1)) (take n (unfold '+ a b)))

User avatar
Xeio
Friends, Faidites, Countrymen
Posts: 5099
Joined: Wed Jul 25, 2007 11:12 am UTC
Location: C:\Users\Xeio\
Contact:

Re: Coding: Fleeting Thoughts

Postby Xeio » Sun Sep 16, 2018 5:50 pm UTC

I'm not really sure I understand why this fixes a stack overflow exception I was having. Or why I wasn't having the overflow exception prior to today (was using this on Thursday at least).

https://github.com/Xeio/TwitchPokemonHe ... 47bc01189c

I'm not even entirely sure why I decided to try changing the scope of the variable as a possible solution (ignoring that I should have scoped it to begin with probably). The initialization line shouldn't even run more than once which presumably is the only way the stack overflow could happen (and attaching a debugger seems to prove this to be the case, and yet the variable value is as though the initialization code did run twice).

User avatar
Flumble
Yes Man
Posts: 2082
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Sun Sep 16, 2018 6:35 pm UTC

Xanthir wrote:Using do notation should just be the obvious "three lines, each pulling the value out of a deeper monad" thing:

That's what you'd think at first, yes. But GHC will immediately complain that it can't match [] with IO. And then you realize (>>=) (and related combinators) work with m a -> (a -> m b) -> m b rather than m a -> (a -> othermonad b) -> whichevermonad b.

I just realized you can tactically insert sequence to end up with something like

Code: Select all

test :: FilePath -> (String -> a) -> IO [a]
test dir f = do
  files <- listDirectory dir
  sequence $ do
    file <- files
    pure $ do
      contents <- readFile file
      pure $ f contents

which still isn't pretty with all the nested dos and dollars, but at least there's no join . fmap sequence involved to smash the result together.


Thanks for the file reader. I really hoped there would be a solution without resorting to handles and seqs, but it can hide as a function :: FilePath -> IO String in some library file anyway.

User avatar
Xanthir
My HERO!!!
Posts: 5336
Joined: Tue Feb 20, 2007 12:49 am UTC
Location: The Googleplex
Contact:

Re: Coding: Fleeting Thoughts

Postby Xanthir » Sun Sep 16, 2018 9:30 pm UTC

Flumble wrote:
Xanthir wrote:Using do notation should just be the obvious "three lines, each pulling the value out of a deeper monad" thing:

That's what you'd think at first, yes. But GHC will immediately complain that it can't match [] with IO. And then you realize (>>=) (and related combinators) work with m a -> (a -> m b) -> m b rather than m a -> (a -> othermonad b) -> whichevermonad b.

Oh duh, right, I always forget that do notation uses `bind` and not `fmap` (and there's no way around it...), so you have to care a lot more about your monad stack.

So yeah, it looks like you want to end up with an `IO List a`, when your naive monad stack is an `IO List IO a`. List is Traversable, so throwing a sequence into there will indeed do what you need.

I just realized you can tactically insert sequence to end up with something like

Code: Select all

test :: FilePath -> (String -> a) -> IO [a]
test dir f = do
  files <- listDirectory dir
  sequence $ do
    file <- files
    pure $ do
      contents <- readFile file
      pure $ f contents

A few notes:

1. Since you're working with these as monads, you can just use the standard `return` rather than `pure`. Makes it a touch more readable; we don't have to think about the Applicative context here.

2. Dont' overuse do when you dont' need to! The innermost one is just calling `f` on the contents of the IO returned by readFile, so you can replace that whole block with `f <$> readFile file`. (Any time you find yourself writing a two-liner do-block like that, consider just fmapping instead.) This gives:

Code: Select all

test :: FilePath -> (String -> a) -> IO [a]
test dir f = do
  files <- listDirectory dir
  sequence $ do
    file <- files
    pure (f <$> (readFile file))


3. Then you end up with another trivial two-liner on the new innermost block, where you're once again just fmap'ing over the structure. You could leave it like this, as it's reasonably readable, but I find it better to rearrange a bit, so you do the sequencing up front. That way you retain a single non-nested `do`, where the RHS of every `<-` is an IO:

Code: Select all

test :: FilePath -> (String -> a) -> IO [a]
test dir f = do
  files <- listDirectory dir
  openedFiles <- sequence $ readFile <$> files
  return f <$> openedFiles


I think this'll work? I'm writing from my head right now, as I have no interpreter handy.

Edit:

Or, since map + sequence is just traverse, use that directly, duh:

Code: Select all

test :: FilePath -> (String -> a) -> IO [a]
test dir f = do
  files <- listDirectory dir
  openedFiles <- traverse readFile files
  return f <$> openedFiles


Edit2:

And at this point we're *nearly* back to your original one-liner: \f dir -> listDirectory dir >>= traverse (\file -> fmap f (readFile file)).

The only significant thing to realize is that the lambda you're passing to traverse is too complicated. If you break down the sequence of events into another map, you recover the structure of my final do block:

Code: Select all

\f dir -> listDirectory dir >>= traverse readFile >>= return $ fmap f


`listDirectory` is `FilePath -> IO [File]`, `traverse readFile` is `[File] -> IO [String]`, and `return $ fmap f` is `[String] -> IO [a]`, so all the types match up.

It's a little annoying that you have to do the `>>= return` dance at the end, just because you don't care about the IO monad at all with the last chunk of code. Even more annoying is that there's no convenient flipped <$>, so you can't easily just tack a fmap onto that bind sequence and do away with the extra monadic frippery. But if you create one yourself, you can write it more simply:

Code: Select all

(<&>) = flip fmap
\f dir -> listDirectory dir >>= traverse readFile <&> fmap f


(I chose <&> because Lens uses & as a flip of $, so <&> as a flip of <$> seems to make good sense. Alternatives are something related to |>, the pipeline operator many languages have, with is also a flip of $; maybe |$> ?)

Whether this is simpler for you mentally or not may be a personal choice. ^_^
Last edited by Xanthir on Mon Sep 17, 2018 6:50 pm UTC, edited 3 times in total.
(defun fibs (n &optional (a 1) (b 1)) (take n (unfold '+ a b)))

User avatar
Xeio
Friends, Faidites, Countrymen
Posts: 5099
Joined: Wed Jul 25, 2007 11:12 am UTC
Location: C:\Users\Xeio\
Contact:

Re: Coding: Fleeting Thoughts

Postby Xeio » Sun Sep 16, 2018 9:52 pm UTC

Xeio wrote:
I'm not really sure I understand why this fixes a stack overflow exception I was having. Or why I wasn't having the overflow exception prior to today (was using this on Thursday at least).

https://github.com/Xeio/TwitchPokemonHe ... 47bc01189c

I'm not even entirely sure why I decided to try changing the scope of the variable as a possible solution (ignoring that I should have scoped it to begin with probably). The initialization line shouldn't even run more than once which presumably is the only way the stack overflow could happen (and attaching a debugger seems to prove this to be the case, and yet the variable value is as though the initialization code did run twice).
Phhht, I'm an idiot, I had the extension installed twice, once from the chrome store and once from an unpacked version. Explains some other weird behavior was seeing too...

EDIT: On the other hand, I don't understand why I only saw one toolbar button for the extension so I'm still unsure as to why that would happen.

My guess is it happened because I installed the extension on my laptop to test with, and it probably sync'd to my main computer.

User avatar
Xenomortis
Not actually a special flower.
Posts: 1426
Joined: Thu Oct 11, 2012 8:47 am UTC

Re: Coding: Fleeting Thoughts

Postby Xenomortis » Mon Sep 24, 2018 10:37 am UTC

How terrible is this?

Code: Select all

try:
   # bunch of method calls on objects from modules
   mod_1_object.call_1()
   mod_2_object.other_call()
   mod_1_object.call_2()
   # some stuff
   mod_1_object.call_4()
   mod_2_object.call_3()
except:
   e_t, e_v, e_tb = sys.exc_infO()
   if any(mod_1.__file__ in trace for trace in traceback.format_tb(e_tb)):
      # this will trigger special error handling
      raise SomeMagicException(...)
   else:
      raise

Errors coming from one module typically indicate a particular failure, usually handled by just waiting for a device controller (represented by mod_1_object) to auto-restart and trying again.
However errors coming from anywhere else cannot be handled.
There isn't a single exception type to handle here - you may get one of several exceptions (some are overly generic) and there is significant overlap between the exceptions coming form mod_1 and mod_2.

I could go into mod_1 and wrap every method there with "try-except-raise", but I don't really like wrapping exceptions without good reason (and Python 2 doesn't have exception chaining).
Or I could surround every relevant call with "try-except" - less hacky but far uglier.
Image

Tub
Posts: 409
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Mon Sep 24, 2018 6:00 pm UTC

Errors coming from one module typically indicate a particular failure, usually handled by just waiting for a device controller (represented by mod_1_object) to auto-restart and trying again.

"typically"? Is there an exception that cannot be handled by retrying? Exceptions for invalid arguments, maybe? If so, add +1 terrible to your proposal.

I see three possibilities:
a) accept your hack or a variation thereof. I would advise against it unless there's only a single consumer of mod1. If you start copy/pasting that catch block into multiple places you just know how it's going to end.

b) Give mod1 a proper API to signal temporary vs. permanent errors. Unless you like returning EAGAIN, that means subclassing a new DeviceIsAfkPleaseRetryLaterError.

If you worry about losing information about the swallowed exceptions, you can either log them, or jury-rig your own exception chaining. (I'm not fluent in python; in JS I'd just try overriding toString() and getStackTrace() or something). Note that swallowing the exceptions in mod1 isn't any worse than swallowing them in your example code.

c) make mod1's API asynchronous and let it do the retrying on its own. Depending on your actual use case, that may end up being incredibly clean, or incredibly ugly.

Or I could surround every relevant call with "try-except" - less hacky but far uglier.

It avoids using debugging information to guide control flow, but it's verbose. A compromise might be

Code: Select all

bool can_retry = true
try:
   can_retry = true
   mod_1_object.call_1()
   can_retry = false
   mod_2_object.other_call()
   can_retry = true
   mod_1_object.call_2()
   mod_1_object.call_4()
   can_retry = false
   mod_2_object.call_3()
except ArgumentError:
  raise // can never be handled
except:
   if (can_retry)
      raise SomeMagicException(...)
   else:
      raise

User avatar
Flumble
Yes Man
Posts: 2082
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Thu Oct 11, 2018 3:01 pm UTC

Is there any shell out there that supports selecting and copying text on the line? (with support for sensible keybindings like home to go to the start of a line, ctrl+shift+left to select the word to the left)
Most of the time I'm working from within screen, so I can copy anything with ctrl+a,[ but I'd like to know if there's an alternative for the times I'm working directly from a terminal.

Tub
Posts: 409
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Thu Oct 11, 2018 4:54 pm UTC

A shell can only do very limited selection, because it isn't a terminal. It simply doesn't know (or care) what's on your screen at the moment. A shell can only copy parts of the current command line - and even bash can do that. It'll go into a bash-internal buffer though, so you cannot copy/paste between different sessions.

For complicated selections on your screen content, this will need to be implemented in a terminal. You mentioned screen, which happens to be a terminal emulator, but (unless heavily scripted), its internal paste buffer is per-session, too.

To copy into your system's clipboard, look for features in your actual terminal. There are scripting extensions for both urxvt and konsole that allow mouseless selections. For xterm or PuTTy, you'll need the mouse. Anything else, ask google.

User avatar
Quercus
Posts: 1757
Joined: Thu Sep 19, 2013 12:22 pm UTC
Location: London, UK
Contact:

Re: Coding: Fleeting Thoughts

Postby Quercus » Thu Oct 11, 2018 7:17 pm UTC

Flumble wrote:Is there any shell out there that supports selecting and copying text on the line? (with support for sensible keybindings like home to go to the start of a line, ctrl+shift+left to select the word to the left)
Most of the time I'm working from within screen, so I can copy anything with ctrl+a,[ but I'd like to know if there's an alternative for the times I'm working directly from a terminal.


Bash has had vi mode forever, which I don't know if you consider more sensible than the default EMACS-like bindings (religious wars is thataway), but it's an option if you like vi(m).

User avatar
Flumble
Yes Man
Posts: 2082
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Thu Oct 11, 2018 10:09 pm UTC

Thanks, I never knew virtually all shells support emacs/vi-style editing (I guess the thought has never occurred because windows shells don't have it). I couldn't find anything with a preliminary search for "shell with selection", obvious in hindsight because the jargon is mark, yank, region, kill ring etc. :oops:

User avatar
phlip
Restorer of Worlds
Posts: 7557
Joined: Sat Sep 23, 2006 3:56 am UTC
Location: Australia
Contact:

Re: Coding: Fleeting Thoughts

Postby phlip » Sat Oct 13, 2018 5:03 am UTC

It's less a bash feature and more a readline feature, so a lot of different CLIs from the GNU space have that sort of functionality.

Code: Select all

enum ಠ_ಠ {°□°╰=1, °Д°╰, ಠ益ಠ╰};
void ┻━┻︵​╰(ಠ_ಠ ⚠) {exit((int)⚠);}
[he/him/his]

User avatar
Flumble
Yes Man
Posts: 2082
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Tue Oct 16, 2018 12:10 am UTC

Hmph, I've been messing with mounting an NTFS partition for an hour. Turns out ntfs-3g wasn't installed, while the builtin kernel module reports the mount as "rw" despite being read-only. :x

User avatar
Xeio
Friends, Faidites, Countrymen
Posts: 5099
Joined: Wed Jul 25, 2007 11:12 am UTC
Location: C:\Users\Xeio\
Contact:

Re: Coding: Fleeting Thoughts

Postby Xeio » Tue Oct 23, 2018 8:10 pm UTC

IMB MQ's .Net bindings poop all over type safety.

User avatar
Link
Posts: 1365
Joined: Sat Mar 07, 2009 11:33 am UTC
Location: ᘝᓄᘈᖉᐣ
Contact:

Re: Coding: Fleeting Thoughts

Postby Link » Tue Nov 13, 2018 3:00 pm UTC

Does anyone know if there's an autoconf macro to find the LIBS, LDFLAGS and CXXFLAGS needed to use C++11 threads? From what I've seen just using -pthread isn't the magic bullet that works for all mainstream compilers, e.g. Clang needs -stdlib=libstdc++ as well and some bugged versions of GCC need -Wl,--no-as-needed.

User avatar
Xanthir
My HERO!!!
Posts: 5336
Joined: Tue Feb 20, 2007 12:49 am UTC
Location: The Googleplex
Contact:

Re: Coding: Fleeting Thoughts

Postby Xanthir » Tue Nov 13, 2018 10:20 pm UTC

Link wrote:


I asked my friend on the C++ standards committee, and here's his answer:

IIRC (@wakomeup / @LouisDionne can correct me) libstdc++ needs -pthread -lpthread, and libc++ needs nothing special to use std::thread from C++11. A concrete repro is necessary to really get what they're trying to do.


Feel free to respond to him directly for further clarification.
(defun fibs (n &optional (a 1) (b 1)) (take n (unfold '+ a b)))

User avatar
Link
Posts: 1365
Joined: Sat Mar 07, 2009 11:33 am UTC
Location: ᘝᓄᘈᖉᐣ
Contact:

Re: Coding: Fleeting Thoughts

Postby Link » Wed Nov 14, 2018 7:21 am UTC

So I take it there's no ready-made autoconf macro to figure everything out automatically?

In any case, this helps me along a good way. I can use AC_COMPILE_IFELSE to check for __GLIBCXX__ and add -pthread if found. Then use AC_LINK_IFELSE to see if the threading configuration works. If it doesn't, I guess I'm content to just error out and tell the user to pass whatever flags their (broken) configuration needs. I'll update this once I cobble something together (or fail to do so).

Tub
Posts: 409
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Wed Nov 14, 2018 9:59 am UTC

clang generally does not require -stdlib=libstdc++. That SO link is quite old, and the runtime error suggests that the then-recent version of libc++ simply lacked support.

I think the autotools way would be to attempt different configurations until one compiles and works (starting with no parameters, then -pthread, then the more exotic variations). Relying on compiler macros is tricky, because compilers (or headers) sometimes lie about these things.

User avatar
Link
Posts: 1365
Joined: Sat Mar 07, 2009 11:33 am UTC
Location: ᘝᓄᘈᖉᐣ
Contact:

Re: Coding: Fleeting Thoughts

Postby Link » Wed Nov 14, 2018 1:37 pm UTC

Tub wrote:
I think the autotools way would be to attempt different configurations until one compiles and works (starting with no parameters, then -pthread, then the more exotic variations). Relying on compiler macros is tricky, because compilers (or headers) sometimes lie about these things.

That does seem better. I wrote a small macro to test whether one configuration works:

Code: Select all

# SYNOPSIS
#
#   AX_CXX_THREAD([EXTRA-CXXFLAGS], [ACTION-IF-TRUE], [ACTION-IF-FALSE])
#
# DESCRIPTION
#
#   Test if C++ standard library threads work, optionally given a set of
#   CXXFLAGS.
#
#   This macro attempts to compile and link a piece of stub code using C++11's
#   std::thread, after adding EXTRA-CXXFLAGS to CXXFLAGS. If successful, it
#   sets
#
#     have_cxx_thread=true
#
#   and runs ACTION-IF-TRUE
#
#   If unsuccessful, it sets
#
#     have_cxx_thread=false
#
#   and runs ACTION-IF-FALSE.
#

#serial 1

AC_DEFUN([AX_CXX_THREAD], [
    AC_REQUIRE([AC_PROG_CXX])
    AC_LANG_PUSH([C++])
    oldCXXFLAGS="${CXXFLAGS}"
    CXXFLAGS="${CXXFLAGS} $1"
    AC_LINK_IFELSE([AC_LANG_PROGRAM([[#include <chrono>
#include <thread>]], [[std::thread t([](void){ std::this_thread::sleep_for(
            std::chrono::seconds(5)); });]])],
        [have_cxx_thread=true],
        [have_cxx_thread=false])
    AC_LANG_POP([C++])
    CXXFLAGS="${oldCXXFLAGS}"
    if test "x$have_cxx_thread" = "xtrue" ; then
        m4_ifvaln([$2],[$2],[:])
        m4_ifvaln([$3],[else $3])
    fi
])

Then I just run through some configurations:

Code: Select all

AC_MSG_CHECKING([if C++11 threads work without extra CXXFLAGS])
AX_CXX_THREAD([], [
        AC_MSG_RESULT([yes])
        AC_SUBST([threading_CXXFLAGS],[""])], [
    AC_MSG_RESULT([no])
    AC_MSG_CHECKING([if C++11 threads work with -pthread])
    AX_CXX_THREAD([-pthread], [
        AC_MSG_RESULT([yes])
        AC_SUBST([threading_CXXFLAGS], [-pthread])], [
    AC_MSG_RESULT([no])
    AC_MSG_CHECKING([if C++11 threads work with -pthread -lpthread])
    AX_CXX_THREAD([-pthread -lpthread], [
        AC_MSG_RESULT([yes])
        AC_SUBST([threading_CXXFLAGS], [-pthread -lpthread])], [
    AC_MSG_RESULT([no])
    AC_MSG_CHECKING([if C++11 threads work with -pthread -stdlib=libstdc++])
    AX_CXX_THREAD([-pthread -stdlib=libstdc++], [
        AC_MSG_RESULT([yes])
        AC_SUBST([threading_CXXFLAGS], [-pthread -stdlib=libstdc++])], [
    AC_MSG_RESULT([no])
    AC_MSG_CHECKING([if C++11 threads work with -pthread -Wl,--no-as-needed])
    AX_CXX_THREAD([-pthread -Wl,--no-as-needed], [
        AC_MSG_RESULT([yes])
        AC_SUBST([threading_CXXFLAGS], [-pthread -Wl,--no-as-needed])], [
    AC_MSG_ERROR([Cannot determine how to use C++11 threads.])])])])])])

More can be added as needed, of course (-pthread always works with my preferred configuration, and I'm not going out of my way to test vastly different configurations until someone else hits a snag).

Tub
Posts: 409
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Wed Nov 14, 2018 5:51 pm UTC

Link wrote:More can be added as needed, of course (-pthread always works with my preferred configuration, and I'm not going out of my way to test vastly different configurations until someone else hits a snag).

That seems reasonable.

If you take a closer look at that SO thread you linked, you'll notice that the program compiled just fine, but failed to run. If you wish to support those configurations, you'll actually need to run the program (AC_RUN_IFELSE) instead of just linking, and make the code test for successful thread execution. Something like this:

int exit_code = EXIT_FAILURE;
std::thread t([&exit_code](void) {
exit_code = EXIT_SUCCESS;
});
t.join();
return exit_code;

User avatar
Link
Posts: 1365
Joined: Sat Mar 07, 2009 11:33 am UTC
Location: ᘝᓄᘈᖉᐣ
Contact:

Re: Coding: Fleeting Thoughts

Postby Link » Wed Nov 14, 2018 6:56 pm UTC

Tub wrote:
Link wrote:More can be added as needed, of course (-pthread always works with my preferred configuration, and I'm not going out of my way to test vastly different configurations until someone else hits a snag).

That seems reasonable.

If you take a closer look at that SO thread you linked, you'll notice that the program compiled just fine, but failed to run. If you wish to support those configurations, you'll actually need to run the program (AC_RUN_IFELSE) instead of just linking, and make the code test for successful thread execution. Something like this:

int exit_code = EXIT_FAILURE;
std::thread t([&exit_code](void) {
exit_code = EXIT_SUCCESS;
});
t.join();
return exit_code;

Ha, that's exactly the kind of mistake you make when reading about the bug in the evening and hastily implementing the macro the next day like I did. Will fix tomorrow. ;)

EDIT: the updated macro:

Code: Select all

# SYNOPSIS
#
#   AX_CXX_THREAD([EXTRA-CXXFLAGS], [ACTION-IF-TRUE], [ACTION-IF-FALSE])
#   AX_CXX_THREAD_LINKONLY([EXTRA-CFLAGS], [ACTION-IF-TRUE],
#                          [ACTION-IF-FALSE])
#
# DESCRIPTION
#
#   Test if C++ standard library threads work, optionally given a set of
#   CXXFLAGS.
#
#   AX_CXX_THREAD attempts to compile, link and run a piece of stub code using
#   C++11's std::thread, after adding EXTRA-CXXFLAGS to CXXFLAGS. If
#   successful, it sets
#
#     have_cxx_thread=true
#
#   and runs ACTION-IF-TRUE
#
#   If unsuccessful, it sets
#
#     have_cxx_thread=false
#
#   and runs ACTION-IF-FALSE.
#
#   AX_CXX_THREAD_LINKONLY behaves identically, save for the fact that it does
#   not attempt to run the compiled and linked program. This is useful when
#   cross-compiling.

#serial 2

AC_DEFUN([AX_CXX_THREAD], [
    AC_REQUIRE([AC_PROG_CXX])
    AC_LANG_PUSH([C++])
    oldCXXFLAGS="${CXXFLAGS}"
    CXXFLAGS="${CXXFLAGS} $1"
    AC_RUN_IFELSE([AC_LANG_PROGRAM([[#include <thread>]], [[
    int exit_code = EXIT_FAILURE;
    std::thread t([](int *x){ *x = EXIT_SUCCESS; }, &exit_code);
    t.join();
    return exit_code;]])],
        [have_cxx_thread=true],
        [have_cxx_thread=false])
    AC_LANG_POP([C++])
    CXXFLAGS="${oldCXXFLAGS}"
    if test "x$have_cxx_thread" = "xtrue" ; then
        m4_ifvaln([$2],[$2],[:])
        m4_ifvaln([$3],[else $3])
    fi
])

AC_DEFUN([AX_CXX_THREAD_LINKONLY], [
    AC_REQUIRE([AC_PROG_CXX])
    AC_LANG_PUSH([C++])
    oldCXXFLAGS="${CXXFLAGS}"
    CXXFLAGS="${CXXFLAGS} $1"
    AC_LINK_IFELSE([AC_LANG_PROGRAM([[#include <thread>]], [[
    int exit_code = EXIT_FAILURE;
    std::thread t([](int *x){ *x = EXIT_SUCCESS; }, &exit_code);
    t.join();
    return exit_code;]])],
        [have_cxx_thread=true],
        [have_cxx_thread=false])
    AC_LANG_POP([C++])
    CXXFLAGS="${oldCXXFLAGS}"
    if test "x$have_cxx_thread" = "xtrue" ; then
        m4_ifvaln([$2],[$2],[:])
        m4_ifvaln([$3],[else $3])
    fi
])
I've opted to explicitly pass a pointer to the return value instead of relying on the lambda's capture-by-reference, but otherwise it's the same. I've defined a second macro AX_CXX_THREAD_LINKONLY that uses AC_LINK_IFELSE, which is useful when cross-compiling.

User avatar
Yakk
Poster with most posts but no title.
Posts: 11092
Joined: Sat Jan 27, 2007 7:27 pm UTC
Location: E pur si muove

Re: Coding: Fleeting Thoughts

Postby Yakk » Thu Nov 15, 2018 4:16 pm UTC

So there are two projects I want to work on in my copious free time.

1: "Virtual considered harmful":
Play with C++23 reflection and metaclasses/type functions. Write a system that replaces the standard C++ object model with one that permits plain old data polymorphic types, and with vtable split from instance. That permits polymorphic value types, polymorphic views, and polymorphism stored outside of type instance (RLE polymorphism for text, for example). It would also permit runtime polymorphism (dynamic vtables) and other non-C++ OO tricks.

2: "Algebraic immutable git documents":
Take libgit2 and immvec and the like, and build a logically immutable document format that stores its data either serialized in a git blob or deserialized in memory (to permit lazy loading of a document from a file). Figure out ways to cleanly have complex structures (lists, properties, maps) while easily composing "transform object of type foo" and "find object of type foo in document" into "transform document" algebraicly. Leave things open for disk/memory/remote documents to be worked on.
One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision - BR

Last edited by JHVH on Fri Oct 23, 4004 BCE 6:17 pm, edited 6 times in total.


Return to “Coding”

Who is online

Users browsing this forum: No registered users and 8 guests