Coding: Fleeting Thoughts

[hotaru]

Is there a linuxy way to monitor services and send an email whenever a services dies or comes back up (and preferably every hour while something's down)?
I'm currently using systemd timers to run a small script to log the status of things like the xkcd minecraft server and I could run a sendmail command when the status is negative, but it seems there ought to be a better way.
If there's no linuxy way, I'll look into graphite+grafana or <insert better suggestion>.

you could use OnFailure= in the the unit file with https://github.com/kylemanna/systemd-utils to send an email notification.

[Xeio]

↶
Hey, those Homestar Runner cartoons aren't gonna play themselves.

Actually something more banal, extracting media from them. Granted there is swfexport to do that, though it makes Windows Defender very very angry, and also some of the files I'm exporting from seem to just... sometimes not follow the spec so... fun.

It was kinda interesting though. Why do I do this for fun.

Related, it's really annoying that BitMap.SetPixel is soooo much slower than using an unsafe block with UnlockBits. Like I get being slower due to some bounds checks, but why is it so much slower?

[commodorejohn]

Yeah, swfexport was already spotty back in the days when Flash MX was new, I can't imagine it copes well with newer versions.

This takes me back to the days of extracting random J-pop tracks from animutations because they never credited the original source and I had no idea what to Google...

[Flumble]

you could use OnFailure= in the the unit file with https://github.com/kylemanna/systemd-utils to send an email notification.

I'm not running the services on that that device, just the monitors (which are mostly of the form "try to establish a TCP connection/HTTP session" to find out whether the (black box) program is running properly). I can let the monitor purposefully stop with an error code to trigger OnFailure, but that's as ugly as having a separate script to send emails.


My eye is caught by the influxdb TICK stack now, mostly because it comes with docker images and a simple logging plugin that runs a script periodically and takes care of the output by itself ...and you can configure it to send emails when something goes wrong.

[Tub]

I can let the monitor purposefully stop with an error code to trigger OnFailure, but that's as ugly as having a separate script to send emails.

What's ugly about that? Your watchdog reports failure via exit code, and systemd does the notification as configured. That seems like it's working as intended.

It would be cleaner to have the OnFailure on the server where the minecraft server is running, but apparently that's not an option?

Also, what exactly is causing the minecraft server to fail? Does the server process crash? Does the server process enter a deadlock?. Does the whole server crash and fail to reboot? Network connectivity issues? Something else entirely? Depending on the problem, I'd suggest different fixes.

My eye is caught by the influxdb TICK stack now, mostly because it comes with docker images and a simple logging plugin that runs a script periodically and takes care of the output by itself ...and you can configure it to send emails when something goes wrong.

That seems like overkill. If you seek reliability, then complexity is your enemy. Unless you're willing to ignore issues like updates, security fixes and general maintenance, then dumping a docker image somewhere is usually more trouble than it's worth. It's certainly more brittle than a sendmail script. I may be a pessimist, but I wouldn't be surprised if that monitoring solution has more downtime than the minecraft server you meant to monitor.


On my home network, I'm using collectd. It's still overkill for monitoring a single service, but it has plugins for many things I want to monitor (network bandwidth on my router, temperature of all CPUs, free disk space and SMART errors etc), supports custom plugins (e.g. I have a few raspberry pis with DHT22 temperature/humidity sensors) and it has threshold-based alerts, including things like "only notify if the server couldn't be reached 3 times in a row".
Distributed logging is supported, e.g. every computer runs a collectd instance, which reports the data to a central server where it gets written to some kind of database. Collectd is flexible about its storage backends. IIRC influxdb works, but I'm using rrdtool for its simplicity and built-in graphing features.

Configuration of the whole thing takes a while, but installation is trivial (most distributions have a package), updating is trivial (the configuration format is very stable; only once did I have to manually adjust something) and the overhead is low enough that I can just start an instance on each computer I need to monitor, including the low-powered raspberry pis.

[Xeio]

Calculating the number of years between two dates, particularly when the client wants it to be a decimal... based on # of months out of 12... is apparently a non-trivial problem.

[Tub]

The bad news: dates are hard.
The good news: if you cheat, the client (probably) won't notice. Because the client doesn't understand dates either, they have no idea what they're requesting, and thus won't notice even if your results are a few millenia off.

Then again, even though your problem is underspecified, I don't think it'd be more than 2-5 lines with a standard date API?

[Xeio]

Well, I probably over-engineered given you're right they probably have no clue what they're asking for, but something like this seemed to work:

Code: Select all

if(larger.Year > smaller.Year)
{
    years = larger.Year - smaller.Year
    if(smaller.AddYears(years) > larger) years--;
    smaller = smaller.AddYears(years);
}

if(smaller.Month < larger.Month)
{
    months = larger.Month - smaller.Month;
    if(smaller.AddMonths(months) > larger) months--;
}
else if(smaller.Month > larger.Month)
{
    months = 12 - smaller.Month + larger.Month;
    if(smaller.AddMonths(months) > larger) months--;
}

total = years + months / 12;

.Net doesn't provide something ilke this out of box, primarily because a TimeSpan doesn't make sense to have a .Months or .Years property. A timespan is measured in ticks, and months/years have different amounts of ticks depending on when the span would start...

EDIT: I think I fixed it more, though it's even more verbose now.

[Thesh]

Basically, you wish .NET had a DateDiff function like Visual Basic and SQL Server.

[Xeio]

Hmmm, well there is interop so I probably could include the VB DLL... On the other hand that's pretty terrible so I'd rather not.

[Tub]

Just connect to an SQL server and SELECT DATEDIFF($date1, $date2)

With a date library based on unix time, it's rather simple.

Code: Select all

function date_diff(date1, date2) {
  var diff_in_ms = Math.abs(date1.getTime() - date2.getTime());
  return diff_in_ms / 1000 / 60 / 60 / 24 / 365.25; // close enough.
}

But microsoft does not like unix time, and if you want the difference between the beginning of the months, try this:

Code: Select all

function date_diff_months(date1, date2) {
  var date1_in_months = date1.getYear() * 12 + date1.getMonth();
  var date2_in_months = date2.getYear() * 12 + date2.getMonth();
  return Math.abs(date1 - date2) / 12;
}

This works, no matter if your language returns getYear() as 2 or 4 digits or 3 digits since 1900, and no matter if getMonth() returns 0- or 1-based values.

[You, sir, name?]

Working with time is hard. Working with dates is even harder. There's leap days, leap seconds, sometimes the timezone changes based on the date, indexing is comically inconsistent on a lot of big APIs.

My favorite is stuff is financial rules like "find the third wednesday of the month that date X is in, unless it's a red day, then the following business day". That's... kinda hard. On a level where I sort of want to just create a static table of wednesdays per month from 1970 through 2040 or whatever and then use that instead of looking it up. You know I'm going to use it for my unit test anyway.

[Tub]

My favorite is stuff is financial rules like "find the third wednesday of the month that date X is in, unless it's a red day, then the following business day". That's... kinda hard.

Is it? You're only working with dates, thus no timezones, leap seconds or anything actually troublesome. And weeks are surprisingly regular - there has not been an instance where, say, a wednesday was skipped or where two wednesdays followed each other. Calculating whether a specific date is a wednesday is something that any crappy date class should be able to do for you.

The "for (day = 1; day < 30; day++)"-solution isn't beautiful, but works (and may thus be a good reference implementation for your test cases, so your actual implementation can be [18,17,16,15,21,20,19][new Date(d.getFullYear(), d.getMonth(), 1).getDay()] or something).

The hardest part is the function that returns all holidays, because that needs constant updates, and maybe you got to deal with the cases where the calculated date changes because a holiday appeared.

[Xanthir]

↶
And weeks are surprisingly regular - there has not been an instance where, say, a wednesday was skipped or where two wednesdays followed each other.

Russia in 1918 would like to have a word with you. (If I'm reading the calendar right, they had two thursdays in a row.)

(But I'm being a pedant.)

[Tub]

Sorry, I don't believe in dates before 1970.

Seriously, do you have a source for that? Google is surprisingly resistant against the search terms I'm using..

[Xenomortis]

https://en.wikipedia.org/wiki/Adoption_ ... ern_Europe
The USSR switched from the Julian to the Gregorian calendar in 1918 - the 30th Jan 1918 (Wednesday) was followed by 14th Feb (Thursday).
Not quite "two sequential Thursdays", but a date-time complication nonetheless.

[phlip]

Possible confounding factor here being that 30 Jan 1918 on the Gregorian calendar was a Thursday... but, 30 Jan 1918 on the Julian calendar was a Wednesday, so Russia had a Wednesday followed by a Thursday, nothing unusual there, just the actual date did weird things.

There are also cases where a place jumped the dateline... switching timezone from UTC-12 to UTC+12 (or vice-versa, I can't remember). They skipped a day, but on both the "days of the week" and "days of the month" sense, so the idea that a particular calendar date is a particular day of the week, and these are an entirely regular pattern, still holds up. As long as you're still working with the Gregorian calendar, your Wednesdays are going to have dates that are 7 days apart.

[ucim]

In any case, the issue is the same as travelers who cross a time zone. They have (say) 3:13 followed by 2:14. But it's 3:13 central time, followed by 2:14 mountain time, which is not incongruous at all. If you don't include the timezone (or the calendar system), you are not specifying time completely, and the ambiguity is simply stored in the unstated parts.

Tomorrow, "the thirty-first" is going to be followed by "the first", and few people will be confused by the omission of the month name.

Of course, for those following the One True Calendar, it will be 25 Voyage, 4AT, followed by 26 Voyage, 4AT. It's just a question of standards.

Jose

[Xanthir]

↶
Possible confounding factor here being that 30 Jan 1918 on the Gregorian calendar was a Thursday... but, 30 Jan 1918 on the Julian calendar was a Wednesday, so Russia had a Wednesday followed by a Thursday, nothing unusual there, just the actual date did weird things.

Ah, indeed. I just did a cal lookup and counted backwards, and didn't realize the Julian day was different.

[Tub]

Today's random thought: "I wish I had a javascript-to-php transpiler."

As horrible as that sounds, it would solve a real problem I have.

[Xenomortis]

No, I can see that one would need a translator if they wanted to convert one lunatic's mad ramblings into the dialect of a depressed drunk.
What I can't envisage is wanting to communicate with either of these people whilst sane and/or sober.

[Xanthir]

↶
Today's random thought: "I wish I had a javascript-to-php transpiler."

As horrible as that sounds, it would solve a real problem I have.

AND I HAVE A SOLUTION FOR YOU

My friend's project, Cassis, is a subset of JS and PHP that works the same across both languages (plus a supporting library to get a bunch of functions working in both). You can write code in Cassis and deploy it both in browsers and servers. ^_^

It purposely takes on a more PHP-centric, functions-not-methods approach, as that maximizes the surface area of identical syntax.

[Tub]

My friend's project, Cassis, is a subset of JS and PHP that works the same across both languages (plus a supporting library to get a bunch of functions working in both). You can write code in Cassis and deploy it both in browsers and servers. ^_^

That's great, now all I need is a JavaScript to Cassis transpiler.

Honestly, that's not even a transpiler. Actual transpilers exist, and they can transpile a lot more code - just not the code I have. Solutions like Cassis are a dime a dozen, though most of them never bother going for a public release - you always end up implementing only the subset of the APIs that your project needs, which means that reusability is low. I've had similar code somewhere, but it's useless for my current problem - existing code that was actually written as JavaScript.

The basic premise is interesting though. Remembering that $ is a valid character in javascript variable names, you can observe that php and javascript appear lexically compatible:

Code: Select all

function isPrime($n) {
    for ($i=2;$i<$n;$i++) {
        if ($n % $i == 0)
            return false;
    }
    return true;
}

This is both valid JavaScript and PHP! We can use variables (though with restricted naming), loops, if/else, functions and simple arithmetic operations.

Recent PHP versions (>5.4) also offer the [] array literal syntax, which allows compatible syntax for arrays!

Code: Select all

function sumPrimes($array) {
    $sum = 0;
    for ($i=0;$i<count($array);$i++) {
        $e = $array[$i];
        if (isPrime($e))
            $sum += $e;
    }
    return $sum;
}

$sum = sumPrimes([2,4,6,9,5,8,13]);

Well, it's not entirely compatible. count($array) is not a javascript function. That's where Cassis enters the scene, re-implementing count() in javascript. Also, let's implement print().

Code: Select all

function print(n) {
    console.log(n);
}

function count(arr) {
    return arr.length;
}

Minor differences apply (carriage returns after print, count() behavior on non-arrays), but those could be fixed if one felt like it.
All together, this is our first useful example program!

Code: Select all

function isPrime($n) {
    for ($i=2;$i<$n;$i++) {
        if ($n % $i == 0)
            return false;
    }
    return true;
}

function sumPrimes($array) {
    $sum = 0;
    for ($i=0;$i<count($array);$i++) {
        $e = $array[$i];
        if (isPrime($e))
            $sum += $e;
    }
    return $sum;
}

print(sumPrimes([2,4,6,9,5,8,13]));

Code: Select all

> cat lib.js code.phpjs | node ; echo '-------' ; cat lib.php code.phpjs | php
20
-------
20

This wasn't a lot of code, right? Very manageable, very readable, even tested and correct. Did you spot the problem?

[gd1]

So basically in C,
Array = duct tape
Malloc = wd40
?

[Tub]

This wasn't a lot of code, right? Very manageable, very readable, even tested and correct. Did you spot the problem?

Aww, nobody wanted to bite.

Here's the thing: php does not have variable declarations. You just initialize them, and then you have your variable.

Code: Select all

$i = 0;

Functions generate a new scope, but blocks don't. To access a variable from a parent scope, you must use the global keyword.

Code: Select all

$a = 0;$b = 1;
function a() {
   global $a;
   echo $a; // 0
   echo $b; // PHP Notice:  Undefined variable: b in..
}

JavaScript on the other hand does have variable declarations, which creates those variables in a local scope (function level with var, block level with let or const).
When using a variable that was not declared in the current scope, JavaScript bubbles up the scope hierarchy until a variable with the name was found. If no variable was found, reads return undefined, but writes will create the variable in the top-level scope.

Code: Select all

var $i = 0;
function a() {
   console.log($i); // 0
   var $a = 1;
   $b = 2; // $b is created in the top-level scope
   console.log($a, $b); // 1, 2
}
console.log($a, $b); // undefined, 2
}

tl;dr: in my code above, when interpreted as JavaScript, $i is a global variable, and both loops keep influencing each other. The only reason it appears to return the correct value is by careful choice of the array. Xanthir, you should get better friends.

[Xanthir]

↶
Xanthir, you should get better friends.

And you should be less randomly insulting?

[Tub]

I meant to end the post on a flippant twist, but.. yeah, sarcarsm and the internet. Apologies.

I do encourage your friend to extend the documentation though. He's clearly trying to get others to use the code, with a name and no less than two domains and everything, but then he refuses to even mention the giant, non-obvious and hard to debug footgun hidden within. That seems kinda impolite.

[Xanthir]

Hm, indeed, and I'm not sure off the top of my head how to fix it; PHP technically has a "var" keyword, but it's used in class declarations, not normal code.

ETA: Ah, he has some comment hacks to put var declarations for all the function's variables at the function's head, only visible to JS: https://github.com/tantek/cassis/blob/master/cassis.js#L530 I've asked him to document it.

[Tub]

Yeah, I already checked, the only way to solve it in a syntax compatible way is to require all variables to be declared const

[chridd]

Yeah, I already checked, the only way to solve it in a syntax compatible way is to require all variables to be declared const :roll:

...or you could use default arguments:

Code: Select all

function f($n, $i = 0, $sum = 0) {
   for($i = 0; $i <= $n; $i++)
      $sum += $i;
   return $sum;
}
f(10)

(gives 55 in both JS and PHP)

[Xanthir]

↶
Yeah, I already checked, the only way to solve it in a syntax compatible way is to require all variables to be declared const :roll:

Why would they need to be declared const? That would prevent the variables from being bound to new values, but PHP imposes no such constraints.

↶
...or you could use default arguments:

Ah, that's extremely clever! Means you're accidentally exposing a weird API, but still, very clever. ^_^

[Tub]

Why would they need to be declared const? That would prevent the variables from being bound to new values, but PHP imposes no such constraints.

Code: Select all

$a = 3; // valid js (but it's a global), valid php
var $a = 3; // valid js, invalid php
let $a = 3; // valid js, invalid php
const a = 3; // valid js, valid php

It's useless for implementing loops or anything that depends on user input, but at least the syntax is compatible. It's also good training if you ever plan to program directly in a SSA IR.

Ah, that's extremely clever! Means you're accidentally exposing a weird API, but still, very clever. ^_^

Clever indeed, unfortunately not supported by all browsers. Default arguments are a more recent addition to JS, and IE won't support it.

[gd1]

So, when programming in C apparently we deal with streams... is it okay to cross the streams?

<stdio.h>, <stdlib.h>, etc. Why so many std's in C?

linux command line stuff: cat, tail, scratch, :3 :>

[commodorejohn]

The best thing about std's, there are so many to choose from

[ucim]

Why so many std's in C?

There was sport at its making?

Jose

[hotaru]

Why so many std's in C?

at least C doesn't have "using namespace std"...

[Xanthir]

↶

Why would they need to be declared const? That would prevent the variables from being bound to new values, but PHP imposes no such constraints.

Code: Select all

$a = 3; // valid js (but it's a global), valid php
var $a = 3; // valid js, invalid php
let $a = 3; // valid js, invalid php
const a = 3; // valid js, valid php

It's useless for implementing loops or anything that depends on user input, but at least the syntax is compatible. It's also good training if you ever plan to program directly in a SSA IR.

Ah, I see. It still wouldn't be a good idea; PHP's const has some significant constraints on its usage, due to the fact that it defines a compile-time constant, and thus can only appear in the outermost scope. (Thus, it's not a block-scoped variable like JS's const is; it's not even a function-scoped variable. It also can only contain a very constrained set of literal expressions, that can be evaluated at compile-time.) I guess you can use a bare `const` in both for defining global constants to simple literal values, but that's about it.

Ah, that's extremely clever! Means you're accidentally exposing a weird API, but still, very clever. ^_^

Clever indeed, unfortunately not supported by all browsers. Default arguments are a more recent addition to JS, and IE won't support it.

Meh.

[Yakk]

I have been messing with pipes.

My primitive plumbing types are:

Sink of T is: void(T)
Source of T is: void(Sink of T)
Pipe from T to U is: void(Source of T, Sink of U)

This creates something interesting, in that when you invoke a Source you never know how much you'll be Sinking. At the same time, the Source can produce them in any number and doesn't have to pass an object that represents that enumeration to you, it can do that internally.

Maybe this is just useful because we don't have good generators/coroutines in C++ yet; a source of T being a generator of T might be simpler?

It still gets you some interesting operations. (PipeA|PipeB)(src, sink) is PipeB( src|PipeA, sink), which is just so tidy.

I sort of want a Sink of T to be a void(Source of T), but that leads to an infinite definition recursion.

Converting U(T) to a Pipe(T->U) is easy. Converting Pipe(T->range of U) to Pipe(T->U) is easy (Pipe(range U->U) is trivial to write). Sadly, Pipe(U->range U) basically requires terminator signals (to know when to flush the buffer). Which makes me think I have a design problem.

Turning a T into a Source of T is a bit awkward, in that we either copy T needlessly or need to know if the Source will be invoked more than once.

I've experimented a bit with multiple argument sinks and sources. But it gets awkward, because notation on how to "line them up" is tricky. And moving them "together" is also awkward. A source of (A,B,C) isn't the same as a source of (A) a source of (B) and a source of (C), because the (A,B,C) source guarantees they come in 3s. On the other hand, a sink of (A,B,C) can be generated by stiching together a sink(A), sink(B) and sink(C). You lose some information, but it works.

Is there something analogous to covariant/contravariant going on here? If feels similar in some sense.

This is continuation passing style, in that source's don't return T's they instead take a continuation that consumes Ts, and similar for pipes. So I'd guess that functional programming has a rich heritage of design knowledge to plumb here that I'm ignorant of.

Is there a relationship with React Streaming?

[Xanthir]

You might be interested in looking into Streams, like RxJS and the like. The concepts are very similar, and they've thought about many of these problems already.

[Xeio]

Todays Google doodle is nice.