This looks excellent. The ability to generate the Python code for the pandas dataframe transformations looks to be more useful than OpenRefine, TBH.

How much work would it be to use Dask (and Dask-ML) as a backend?

I see the OneHotEncoder button. Have you considered integration with Yellowbrick? They've probably already implemented a few of your near-future and someday roadmap items involving hyperparameter selection and model selection and visualization? https://www.scikit-yb.org/en/latest/

This video shows more of the advanced bamboolib features: https://youtu.be/I0a58h1OCcg

The live histogram rebinning looks useful. Recently I read about a 'shadowgram' / ~KDE approach with very many possible bin widths translucently overlaid in one chart. https://stats.stackexchange.com/questions/68999/how-to-smear...

Yellowbrick also has a bin width optimization visualization in yellowbrick.target.binning.BalancedBinningReference: https://www.scikit-yb.org/en/latest/api/target/binning.html

Great work.

There's a sphinx extension (sphinxcontrib-sdedit) [1] for sdedit ("Quick Sequence Diagram Editor") which does require Java [2].

[1] https://pypi.org/project/sphinxcontrib-sdedit/

[2] http://sdedit.sourceforge.net/

Taskwarrior is also built upon the todo.txt format. [1]

Taskw supports various task dates – { due: scheduled: wait: until: recur: } [2]

Taskw supports various named dates like soq/eocq, som/eom (start/end of [current] quarter, start/end of month), tomorrow, later [3]

Taskw recurring tasks (recur:) use the duration syntax: weekly/wk/w, monthly/mo, quarterly/qtr, yearly/yr, … [4]

Pandas has a "date offset" "frequency string" microsyntax that supports business days, quarters, and years; e.g. BQuarterEnd, BQuarterBegin [5]

IDK how usable by other tools these date string parsers are.

W/ just a text editor, having `todo.txt`, `daily.todo.txt`, and `weekly.todo.txt` (and `cleanhome.todo.txt` and `hygiene.todo.txt` with "## heading" tasks that get lost @where +sorting) works okay.

I have physical 43 folders, too: A 12 month and a 31 day expanding file. [6]

[1] http://todotxt.org/

[2] https://taskwarrior.org/docs/using_dates.html

[3] https://taskwarrior.org/docs/named_dates.html

[4] https://taskwarrior.org/docs/durations.html

[5] https://pandas.pydata.org/pandas-docs/stable/user_guide/time...

[6] http://www.43folders.com/

pytest-docker-pexpect: https://github.com/nvbn/pytest-docker-pexpect

Pexpect: https://pexpect.readthedocs.io/en/stable/

pytest with subprocess.popen (or Sarge) may be sufficient for checking return codes and checking stdout and stderr output streams. Pytest has tmp_path and tmpdir fixtures that provide less test isolation than Docker containers: http://doc.pytest.org/en/latest/tmpdir.html

sarge.Capture.expect() takes a regex and returns None if there's no match: https://sarge.readthedocs.io/en/latest/tutorial.html#looking...

We really could get more out of this data through international collaboration and through linked data (e.g. URIs for columns). See: "Open, and Linked, FDA data" https://github.com/FDA/openfda/issues/5#issuecomment-5392966... and "ENH: Adverse Event Count / 'Use' Count Heatmap" https://github.com/FDA/openfda/issues/49

With sales/usage counts, we'd have a denominator with which we could calculate relative hazard.

Shiv does not solve for what containers and Docker/Podman/Buildah/Containerd solve for: re-launching processes at boot and failure, launching processes in chroots or cgroups (with least privileges), limiting access to network ports, limiting access to the host filesystem, building chroots / images, [...]

You can run build tools like shiv with a RUN instruction in a Dockerfile and get some caching.

You can build a zipapp with shiv (in a build container) and run the zipapp in a container.

Should the zipapp contain the test suite(s) and test_requires so that the tests can be run in an environment most similar to production?

It's much easier to develop with code on the filesystem (instead of in a zipapp).

It's definitely faster to read the whole zipapp into RAM than to stat and read each imported module from the filesystem once at startup.

There may be a better post title than the current "Python Alternative to Docker"? Shiv is a packaging utility for building Python zipapps. Shiv is not an alternative to process isolation with containers (or VMs)

"UNICEF launches Cryptocurrency Fund: UN Children’s agency becomes first UN Organization to hold and make transactions in cryptocurrency" https://www.unicef.org/press-releases/unicef-launches-crypto...

From https://www.unicefusa.org/ :

> UNICEF USA helps save and protect the world's most vulnerable children. UNICEF USA is rated one of the best charities to donate to: 89% of every dollar spent goes directly to help children.

Here are the Python 3 docs for sorting [1], in-place list.sort() [2], and sorted() [3] (which makes a sorted copy of the references). And the Timsort Wikipedia page [4].

[1] https://docs.python.org/3/howto/sorting.html#sort-stability-...

[2] https://docs.python.org/3/library/stdtypes.html#list.sort

[3] https://docs.python.org/3/library/functions.html#sorted

[4] https://en.wikipedia.org/wiki/Timsort

"a11y": Accessibility

https://a11yproject.com/ has patterns, a checklist for checking web accessibility, resources, and events.

awesome-a11y has a list of a number of great resources for developing accessible applications: https://github.com/brunopulis/awesome-a11y

In terms of W3C specifications [1], you've got: WAI-ARIA (Web Accessibility Initiative: Accessibile Rich Internet Applications) [2], and WCAG: Web Content Accessibility Guidelines [3]. The new W3C Payment Request API [4] makes it easy for browsers to offer a standard (and probably(?) already accessible) interface for the payment data entry screen, at least.

There are a number of automated accessibility testing platforms. "[W3C WAI] Web Accessibility Evaluation Tools List" [5] lists quite a few. Can someone recommend a good accessibility testing tools? Is Google Lighthouse (now included with Chrome Devtools and as a standalone script) a good tool for accessibility reviews?

[1] https://github.com/brunopulis/awesome-a11y/blob/master/topic...

[2] https://www.w3.org/TR/using-aria/

[3] https://www.w3.org/WAI/standards-guidelines/wcag/

[4] https://www.w3.org/TR/payment-request/

[5] https://www.w3.org/WAI/ER/tools/

Cool!

requests_cache caches HTML requests into one SQLite database. [1] pandas-datareader can cache external data requests with requests-cache. [2]

dask.cache can do opportunistic caching (of 2GB of data). [3]

How does streamlit compare to jupyter voila dashboards (with widgets and callbacks)? They just launched a new separate github org for the project. [4] There's a gallery of voila dashboard examples. [5]

> Voila serves live Jupyter notebooks including Jupyter interactive widgets.

> Unlike the usual HTML-converted notebooks, each user connecting to the Voila tornado application gets a dedicated Jupyter kernel which can execute the callbacks to changes in Jupyter interactive widgets.

> - By default, voila disallows execute requests from the front-end, preventing execution of arbitrary code.

[1] https://github.com/reclosedev/requests-cache

[2] https://pandas-datareader.readthedocs.io/en/latest/cache.htm...

[3] https://docs.dask.org/en/latest/caching.html

[4] https://github.com/voila-dashboards/voila

[5] https://blog.jupyter.org/a-gallery-of-voil%C3%A0-examples-a2...

Acess control and resource exhaustion are challenges with building any {Flask, framework_x,} app [from Jupyter notebooks]. First it's "HTTP Digest authentication should be enough for now"; then it's "let's use SSO and LDAP" (and review every release); then it's "why is it so sloww?". JupyterHub has authentication backends, spawners, and per-user-container/vm resource limits.

> Each user on your JupyterHub gets a slice of memory and CPU to use. There are two ways to specify how much users get to use: resource guarantees and resource limits. [6]

[6] https://zero-to-jupyterhub.readthedocs.io/en/latest/user-res...

Some notes re: voila and JupyterHub:

> The reason for having a single instance running voila only is to allow non JupyterHub users to have access to the dashboards. So without going through the Hub auth flow.

> What are the requirements in your case? Voila can be installed in the single user Docker image, so that each user can also use it on their own server (as a server extension for example). [7]

[7] https://github.com/voila-dashboards/voila/issues/112

Who even asked these questions?

I question this. All of this.

For $100, you can buy a Pinebook with an 11" or 14" screen, a multitouch trackpad, gigabytes of storage, WiFi, a keyboard without a numpad, and an ARM processor.

On this machine, you can create reproducible analyses with JupyterLab; do arithmetic with Python; work with multidimensional arrays with NumPy, SciPy, Pandas, xarray, Dask; do machine learning with Statsmodels, Scikit-learn, Dask-ML, TPOT; create books of these notebooks (containing code and notes (in Markdown, which easily transformed to HTML) and LaTeX equations) with jupyter-book, nbsphinx, git + BinderHub; store the revision history of your discoveries; publish what you've discovered and learned to public or private git repositories; and complete graded exercises with nbgrader.

But the task is to prepare for a world of mental arithmetic, no validation, no tests, no reference materials, and no search engines; and CAS (Computer Algebra Systems) systems like SymPy and Sage are not allowed.

On this machine, you can run write code, write papers, build spreadsheets and/or Jupyter notebooks, run physical simulations, explore the stars, and play games and watch videos. Videos like: Khan Academy videos and exercises that you can watch and do, with validation, until you've achieved mastery and move on to the next task on your todo.txt list.

But the task is to preserve your creativity and natural curiosity despite the compulsory education system's demands for quality control and allocative efficiency; in an environment where drama and popularity are the solutions to relatedness and acceptance needs.

I have three of these $100 calculators in my toolbox. It's been so long since I've powered them on that I'm concerned that the rechargeable AAA batteries are leaking battery acid.

For $100, you can buy an ARM notebook and install conda and conda-forge packages and build sweet visualizations to collaborate with colleagues on (with Seaborn (matplotlib), HoloViews, Altair, Plotly)

"You must buy a $100 calculator that only runs BASIC and ASM, and only use it for arithmetic so that we can measure you."

Hand tools are fun, but please don't waste any more of my compulsory time.

It'd be interesting to see how much this same workload would cost with e.g. OpenFaaS on k8s with autoscaling to zero; but there also you'd need to include maintenance costs like OS and FaaS stack upgrades. https://docs.openfaas.com/architecture/autoscaling/

Maximum entropy: https://en.wikipedia.org/wiki/Maximum_entropy

Here's a quote of a tweet about a (my own): comment on a schema:BlogPost: https://twitter.com/westurner/status/1048125281146421249:

> “When Bayes, Ockham, and Shannon come together to define machine learning” https://towardsdatascience.com/when-bayes-ockham-and-shannon...

> Comment: "How does this relate to the Principle of Maximum Entropy? How does Minimum Description Length relate to Kolmogorov Complexity?"

Graph query languages are nice and all, but what about Linked Data here? Queries of schemaless graphs miss lots of data because without a schema this graph calls it "color" and that graph calls it "colour" and that graph calls it "色" or "カラー". (Of course this is also an issue even when there is a defined schema; but it's hardly possible to just happen to have comprehensible inter or even intra-organizational cohesion without e.g. RDFS and/or OWL and/or SHACL for describing (and changing) the shape of the data)

So, the task is then to compile schema-aware SPARQL to GQL or GraphQL or SQL or interminable recursive SQL queries or whatever it is.

For GraphQL, there's GraphQL-LD (which somewhat unfortunately contains a hashtag-indeterminate dash). I cite this in full here because it's very relevant to the GQL task at hand:

"GraphQL-LD: Linked Data Querying with GraphQL" (2018) https://comunica.github.io/Article-ISWC2018-Demo-GraphQlLD/

> GraphQL is a query language that has proven to be a popular among developers. In 2015, the GraphQL framework [3] was introduced by Facebook as an alternative way of querying data through interfaces. Since then, GraphQL has been gaining increasing attention among developers, partly due to its simplicity in usage, and its large collection of supporting tools. One major disadvantage of GraphQL compared to SPARQL is the fact that it has no notion of semantics, i.e., it requires an interface-specific schema. This therefore makes it difficult to combine GraphQL data that originates from different sources. This is then further complicated by the fact that GraphQL has no notion of global identifiers, which is possible in RDF through the use of URIs. Furthermore, GraphQL is however not as expressive as SPARQL, as GraphQL queries represent trees [4], and not full graphs as in SPARQL.

> In this work, we introduce GraphQL-LD, an approach for extending GraphQL queries with a JSON-LD context [5], so that they can be used to evaluate queries over RDF data. This results in a query language that is less expressive than SPARQL, but can still achieve many of the typical data retrieval tasks in applications. Our approach consists of an algorithm that translates GraphQL-LD queries to SPARQL algebra [6]. This allows such queries to be used as an alternative input to SPARQL engines, and thereby opens up the world of RDF data to the large amount of people that already know GraphQL. Furthermore, results can be translated into the GraphQL-prescribed shapes. The only additional requirement is their queries would now also need a JSON-LD context, which could be provided by external domain experts.

> In related work, HyperGraphQL [7] was introduced as a way to expose access to RDF sources through GraphQL queries and emit results as JSON-LD. The difference with our approach is that HyperGraphQL requires a service to be set up that acts as a intermediary between the GraphQL client and the RDF sources. Instead, our approach enables agents to directly query RDF sources by translating GraphQL queries client-side.

All of these RDFS vocabularies and OWL ontologies provide structure that minimizes the costs of merging and/or querying multiple datasets: https://lov.linkeddata.es/dataset/lov/

All of these schema.org/Dataset s in the "Linked Open Data Cloud" are easier to query than a schemaless graph: https://lod-cloud.net/ . Though one can query schemaless graphs with SPARQL, as well.

For reference, RDFLib has a bunch of RDF graph implementations over various key/value and SQL store backends. RDFLib-sqlachemy does query parametrization correctly in order to minimize the risk of query injection. FOR THE RECORD, SQL Injection is the CWE Top 25 #1 most prevalent security weakness; which is something that any new spec and implementation should really consider before launching anything other than an e.g. overly-verbose JSON-based query language that people end up bolting a micro-DSL onto. https://github.com/RDFLib/rdflib-sqlalchemy

Most practically, I frequently want to read a graph of objects into RAM; update, extend, and interlink; and then transactionally save the delta back to the store. This requires a few things: (1) an efficient binary serialization protocol like Apache Arrow (SIMD), Parquet, or any of the BSON binary JSONs; (2) a transactional local store that can be manually synchronized with the remote store until it's consistent.

SPARQL Update was somewhat of an out-of-scope afterthought. Here's SPARQL 1.1 Update: https://www.w3.org/TR/sparql11-update/

Here's SOLID, which could be implemented with SPARQL on GQL, too; though all the re-serialization really shouldn't be necessary for EAV triples with a named graph URI identifier: https://solidproject.org/

5 star data: PDF -> XLS -> CSV -> RDF (GQL, AFAIU (but with no URIs(!?))) -> LOD https://5stardata.info/en/

What an excellent 500 lines introduction to the byterun bytecode interpreter / virtual machine: https://github.com/nedbat/byterun

Also, proceeds from optional purchases of the AOSA books go to Amnesty International. https://aosabook.org/

What a good idea.

Here's the hyperlinkified link to the {systemd-homed.service, systemd-userdbd.service, homectl, userdbctl} sources from the PDF: https://github.com/poettering/systemd/tree/homed

Hadn't heard of varlink: https://varlink.org/

Is there a FIPS-like subset of the most-widely-available LUKS configs? Otherwise home directories won't work on systems that have a limited set of LUKS modules.

There's also RustPython, a Rust implementation of CPython 3.5+: https://news.ycombinator.com/item?id=20686580

> https://github.com/RustPython/RustPython

> To be successful over the course of a career requires the application and accumulation of expertise. This assumes that for any given undertaking you either provide expertise or you are just a bystander. It’s the experts that are the drivers — an expertise that is gained from a curiosity, and a mindset of treating one’s craft very seriously.

The stratifications on this are troubling.

> But there are wide differences in assessments by partisanship. Nine in 10 Democrats think weather disasters are more extreme, compared with about half of Republicans.

It's not a partisan issue: we all pay these costs.

> Majorities of adults across demographic groups think weather disasters are getting more severe, according to the poll. College-educated Americans are slightly more likely than those without a degree to say so, 79 percent versus 69 percent.

Weather disasters are getting more severe. It is objectively, quantitatively true that weather disasters are getting more frequent and more severe.

I, for one, really appreciate the raytracing in these visualizations. I wish for more box surfing examples.

`Ctrl + 4` to center view on page!

Pressure sensitive pencil for the PowerStroke Live Path Effect (LPE) "if a pressure sensitive device is available"!

The 1914 $10 Dollar Bill was printed on hemp paper. Today, they're worth like $49.99. IDK how steady that price is over time; relative to the prices of other CPI All goods.

Volatility works out for people who save (who park capital in liquid assets that aren't doing work in order to have wheat for the eventual famine). These guys. They save, short like heck when the market is falling, and swoop in to save the day. What a great time to be selling 0% loans.

Personal Savings Rate (PSR) stratified by greatest generation and not greatest generation is also relevant. Are relatively fixed living expenses higher now? Yes. Is my generation just blowing what they could invest into interest-bearing investments on unnecessary stuff from Amazon? Yes. And expensive meals and drinks.

How have corporate profits and wages changed?

In their day, you put you gosh-danged money aside. For later. So that you have money later.

And that is why you should buy my book, entitled: "Invest in things with long term returns: don't buy shtuff you don't f need, save for tomorrow; and other financial advice"

Which brings me to: the cost of college textbooks and a college education in terms of average hourly wages.

By the way, over the longer term, index funds are likely to outperform funds. Gold may be likely to outperform the stock market. And, over the recent term -- this is for all you suckers out there -- cryptocurrencies have outperformed all stock and commodities markets. How much total wealth is being created on an annual basis here?

Payday loans have something like 300% APY.

How does 2% inflation affect trade when other central banking cabals haven't chosen the same target? "Devaluation"! "Treachery"!

> At the same time, all the current implementations of rootless containers rely on user namespaces at their core. Not to be confused with what is referred to as non-root containers in this article, rootless containers are containers that can be run and managed by unprivileged users on the host. While Docker and other runtimes require a daemon running as root, rootless containers can be run by any user without additional capabilities.

non-root / rootless

FWIW, here's awesome-static-analysis > Programming Languages > C/C++: https://github.com/mre/awesome-static-analysis/blob/master/R...

These tools have lists of functions not to use. Most of them — at least the security-focused ones — likely also include: strcpy, strcat, strncpy, strncat, sprints, and vsprintf just like banned.h

"OWASP Top 10 Most Critical Web Application Security Risks" https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Proje...

> A1:2017-Injection, A2:2017-Broken Authentication, A3:2017-Sensitive Data Exposure, A4:2017-XML External Entities (XXE), A5:2017-Broken Access Control, A6:2017-Security Misconfiguration, A7:2017-Cross-Site Scripting (XSS), A8:2017-Insecure Deserialization, A9:2017-Using Components with Known Vulnerabilities, A10:2017-Insufficient Logging&Monitoring

"OWASP Top 10 compared to SANS CWE 25" https://www.templarbit.com/blog/2018/02/08/owasp-top-10-vs-s...

> This seven-metre-tall machine can squeeze carbon into diamonds

OT but, is this a thing now? Diamonds can be entangled.

I map <up> to history-search-backward in my .inputrc; so I can type 'sudo ' and press <up> to cycle through everything starting with sudo:

    #  <up>      -- history search backward (match current input)
    "\e[A": history-search-backward
    #  <down>    -- history search forward (match current input)
    "\e[B": history-search-forward

This looks useful.

FWIW, you can build a curses-style terminal GUI with Urwid (in Python) and use that through the web. AFAIU, it requires Apache; but it's built on Tornado (which is now built on Asyncio) so something more lightweight than Apache on a Pi should definitely be doable. Termpage with like a Go or Rust REST API may still be more lightweight, but more work.

I have a shell script I named 'e' (for edit) that does basically this. If VIRTUAL_ENV_NAME is set (by virtualenvwrapper), e opens a new tab in that gui vim remote if gvim or macvim are on PATH, or just in a console vim if not. https://github.com/westurner/dotfiles/blob/develop/scripts/e

'editwrd'/'ewrd'/'ew' does tab-completion relative to whatever $_WRD (working directory) is set to (e.g. by venv) and calls 'e' with that full path: https://github.com/westurner/dotfiles/blob/develop/scripts/_...

It's unfortunately not platform portable like vimer, though.

What a cool use of gravitational potential energy. It would be interesting to learn how much more energy is produced by the regenerative breaking system on the downhill and whether they use the excess to load the truck?

Technically, a checkbox may qualify as a digital signature; however, identification / authentication and storage integrity are fairly challengeable (just as a written signature on a piece of paper with a date written on it is challengeable)

Given that notarization is not required for parental consent forms, I'm not sure what sort of server security expense is justified or feasible.

How much does processing all of the paper forms cost each school? Per-student?

In terms of storing digital record of authorization, a private set of per-student OpenBadges with each OpenBadge issued by the school would be easy enough. W3C Verified Claims (and Linked Data Signatures) are the latest standards for this sort of thing.

We could evaluate our current standards for chain of custody in regards to the level of trust we place in commercial e-signature platforms.

The school could send home a sheet with a QR code and a shorturl, but that would be more expensive than running hundreds of copies of the same sheet of paper.

The school could require a parent or guardian's email address for each student in the SIS Student Information System and email unique links to prefilled forms requesting authorization(s).

Just as with e-Voting, assuring that the person who checks a checkbox or tries to scribble their signature with a mouse or touchscreen is the authorized individual may be more difficult than verifying that a given written signature is that of the parent or guardian authorized to authorize.

AFAIU, Google Forms for School can include the logged-in user's username; but parents don't have school domain accounts with Google Apps for Education or Google Classroom.

How would the solution integrate with schools' existing SIS (Student Information Systems)? Upload a CSV of (student, {student info}, {guardian email (s)})? This is private information that deserves security, which costs money.

Which users can log-in for the school and/or district to check the state of the permission / authorization requests and PII personally-identifiable information.

While cryptographic signatures may be overkill as a substitute for permission slips, FWIW, a timestamp within a cryptographically-signed document only indicates what the local clock was set to at the time. Blockchains have relatively indisputable timestamps ("certainly no later than the time that the tx made it into a block"), but blockchains don't solve for proving the key-person relation at a given point in time.

And also, my parent or guardian said you can take me on field trips if you want. https://backpack.openbadges.org/

There are lots of apps that do database introspection. Some also generate forms on the fly, but eventually it's necessary to: specify a forms widget for a particular field because SQL schema only describes the data and not the UI; and specify security authorization restrictions on who can create, read, update, or delete data.

And then you want to write arbitrary queries to filter on columns that aren't indexed; but it's really dangerous to allow clients to run arbitrary SQL queries because there basically are no row/object-level database permissions (the application must enforce row-level permissions).

Datasette is a great tool for read-only database introspection and queries of SQLite databases. https://github.com/simonw/datasette

Sandman2 generates a REST API for an arbitrary database. https://github.com/jeffknupp/sandman2

You can generate Django models and then write admin.py files for each model/table that you want to expose in the django.contrib.admin interface.

There are a number of apps for providing a GraphQL API given introspection of a database that occurs at every startup or at runtime; but that doesn't solve for row-level permissions (or web forms)

If you have an OpenAPI spec for the REST API that runs atop The database, you can generate forms ("scaffolding") from the OpenAPI spec and then customize those with form widgets; optionally with something like json-schema.

It's not safe to allow introspected CRUD like e.g. phpMyAdmin for anything but development. If there are no e.g. foreign-key constraints specified in the SQL schema,a blindly-introspected UI very easily results in database corruption due to invalid foreign key references (because the SQL schema doesn't specify what table.column a foreign key references).

Django models, for example, unify SQL schema and forms UI in models.py; admin.py is optional but really useful for scaffolding (such as when you're doing manual testing because you haven't yet written automated tests) https://docs.djangoproject.com/en/2.2/ref/contrib/admin/#mod...

> The press release from the company said, “heavy-duty vehicles produce more particulate matter than all of the state’s power plants combined”.

> […] for instance why only “10 school buses”?

IARC has recognized diesel exhaust as carcinogenic (lung cancer) since 2012.

Are there other electric school bus programs in the US?

(edit)

https://www.trucks.com/2019/03/22/can-electric-school-buses-...

> Most school systems don’t have sufficient capital to finance the high initial costs of electric bus purchases and charging infrastructure development, he said.

> In the U.S., the school bus market is about 33,000 to 35,000 vehicles per year – about six times more than transit buses.

Ansible is worth the extra few minutes, IMHO.

+ (minimal) Bootstrap System playbook

+ Complete System playbook (that references group_vars and host_vars)

+ Per-machine playbooks stored alongside the ansible inventory, group_vars, and host_vars in a separate repo (for machine-specific kernel modules and e.g. touchpad config)

+ User playbook that calls my bootstrap dotfiles shell script

+ Bootstrap dotfiles shell script, which creates symlinks and optionally installs virtualenv+virtualenvwrapper, gitflow and hubflow, and some things with pipsi. https://github.com/westurner/dotfiles/blob/develop/scripts/b...

+ setup_miniconda.sh that creates a CONDA_ROOT and CONDA_ENVS_PATH for each version of CPython (currently py27-py37)

Over the years, I've worked with Bash, Fabric, Puppet, SaltStack, and now Ansible + Bash

I log shell commands with a script called usrlog.sh that creates a $USER and per-virtualenv tab-delimited logfiles with unique per-terminal-session identifiers and ISO8601 timestamps; so it's really easy to just grep for the apt/yum/dnf commands that I ran ad-hoc when I should've just taken a second to create an Ansible role with `ansible-galaxy init ansible-role-name ` and referenced that in a consolidated system playbook with a `when` clause. https://westurner.github.io/dotfiles/usrlog.html#usrlog

A couple weeks ago I added an old i386 netbook to my master Ansible inventory and system playbook and VScode wouldn't install because VScode Linux is x86-64 only and the machine doesn't have enough RAM; so I created when clauses to exclude VScode and extensions on that box (with host_vars). Gvim with my dotvim works great there too though. Someday I'll merge my dotvim with SpaceVim and give SpaceMacs a try; `git clone; make install` works great, but vim-enhanced/vim-full needs to be installed with the system package manager first so that the vimscript plugin installer works and so that the vim binary gets updated when I update all.

I've tested plenty of Ansible server configs with molecule (in docker containers), but haven't yet taken the time to do a full workstation build with e.g. KVM or VirtualBox or write tests with testinfra. It should be easy enough to just run Ansible as a provisioner in a Vagrantfile or a Packer JSON config. VirtualBox supports multi-monitor VMs and makes USB passthrough easy, but lately Docker is enough for everything but Windows (with a PowerShell script that installs NuGet packages with chocolatey) and MacOS (with a few setup scripts that download and install .dmg's and brew) VMs. Someday I'll write or adapt Ansible roles for Windows and Mac, too.

I still configure browser profiles by hand; but it's pretty easy because I just saved all the links in my tools doc: https://westurner.github.io/tools/#browser-extensions

Someday, I'll do bookmarks sync correctly with e.g. Chromium and Firefox; which'll require extending westurner/pbm to support Firefox SQLite or a rewrite in JS with the WebExtension bookmarks API.

A few times, I've decided to write docs for my dotfiles and configuration management policies like someone else is actually going to use them; it seemed like a good exercise at the time, but invariably I have to figure out what the ultimate command sequence was and put that in a shell script (or a Makefile, which adds a dependency on GNU make that's often worth it)

Clonezilla is great and free, but things get out of date fast in a golden master image. It's actually possible to PXE boot clonezilla with Cobbler, but, AFAICT, there's no good way to secure e.g. per-machine disk or other config with PXE. Apt-cacher-ng can proxy-cache-mirror yum repos, too. Pulp requires a bit of RAM but looks like a solid package caching system. I haven't yet tested how well Squid works as a package cache when all of the machines are simultaneously downloading the exact same packages before a canary system (e.g. in a VM) has populated the package cache.

I'm still learning to do as much as possible with Docker containers and Dockerfiles or REES (Reproducible Execution Environment Specifications) -compatible dependency configs that work with e.g. repo2docker and https://mybinder.org/ (BinderHub)

This system will need to interface with other domestic and international settlement and payments networks.

There is thus an opportunity for standards, a need for federation, and a need to make it easy for big players to offer liquidity.

As far as I understand, e.g. Ripple and Stellar solve basically exactly the 24x7x365 RTGS problem that FedNow intends to solve; and, they allow all sorts of assets to be plugged into the network. Could FedNow just use a different UNL (Unique Node List) with participating banks operating trusted validators and/or offering liquidity ("liquidity provisioning")?

Notably, Ripple is specifically positioned to do international interbank real time gross settlement (RTGS) and remittances. Ripple could integrate with FedNow directly. Most efficiently, if it complies with KYC/AML requirements, FedNow could operate an XRP Ledger. Or, each bank could operate XRP Ledgers. https://xrpl.org/become-an-xrp-ledger-gateway.html

Getting thousands of banks to comply with an evolving API / EDI spec is no small task. Blockchain solutions require API compliance, have solutions for governance where there are a number of stakeholders seeking to reach consensus, and lack single points of failure.

Here's to hoping that we've learned something about decentralizing distributed systems for resiliency.

>> In contrast, the XRP Ledger requires 80 percent of validators on the entire network, over a two-week period, to continuously support a change before it is applied. Of the approximately 150 validators today, Ripple runs only 10. Unlike Bitcoin and Ethereum — where one miner could have 51 percent of the hashing power — each Ripple validator only has one vote in support of an exchange or ordering a transaction. https://news.ycombinator.com/item?id=19195050

So, you want to get banks onboard with only one s'coin USD stablecoin; but you don't want to deal with exchanges or FOREX or anything because that's a different thing? And, this is not just yet another ACH with lower clearance time?

> Interledger Architecture

https://interledger.org/rfcs/0001-interledger-architecture/

> Interledger provides for secure payments across multiple assets on different ledgers. The architecture consists of a conceptual model for interledger payments, a mechanism for securing payments, and a suite of protocols that implement this design.

> The Interledger Protocol (ILP) is the core of the Interledger protocol suite. Colloquially, the whole Interledger stack is sometimes referred to as "ILP". Technically, however, the Interledger Protocol is only one layer in the stack.

> Interledger is not a blockchain, a token, nor a central service. Interledger is a standard way of bridging financial systems. The Interledger architecture is heavily inspired by the Internet architecture described in RFC 1122, RFC 1123 and RFC 1009.

[...]

> You can envision the Interledger as a graph where the points are individual nodes and the edges are accounts between two parties. Parties with only one account can send or receive through the party on the other side of that account. Parties with two or more accounts are connectors, who can facilitate payments to or from anyone they're connected to.

> Connectors provide a service of forwarding packets and relaying money, and they take on some risk when they do so. In exchange, connectors can charge fees and derive a profit from these services. In the open network of the Interledger, connectors are expected to compete among one another to offer the best balance of speed, reliability, coverage, and cost.

Why should we prefer an immutable, cryptographically-signed blockchain solution over SQL/BigTable/MQ for FedNow?

Blockchain and payments standards: https://news.ycombinator.com/item?id=19813340

... Here's the notice and request for comment PDF: "Docket No. OP – 1670: Federal Reserve Actions to Support Interbank Settlement of Faster Payments" https://www.federalreserve.gov/newsevents/pressreleases/file...

"Federal Reserve announces plan to develop a new round-the-clock real-time payment and settlement service to support faster payments" https://www.federalreserve.gov/newsevents/pressreleases/othe...

> this example shows that real wealth doesn’t actually come from golden hoards. It comes from the productive activities of human beings creating things that other human beings desire.

Value, Price, and Wealth

HDD – Hypothesis-Driven Development – Research, Plan, Prototype, Develop, Launch, Review.

The article lists (and links to!) "Lean UX" [1] and Google Ventures' Design Sprint Methodology as inspirations.

[1] "Lean UX: Applying Lean Principles to Improve User Experience" http://shop.oreilly.com/product/0636920021827.do

[2] https://www.gv.com/sprint/

"How To Write A Technical Paper" [3][4] has: (Related Work, System Model, Problem Statement), (Your Solution), (Analysis), (Simulation, Experimentation), (Conclusion)

[3] https://news.ycombinator.com/item?id=18226543

[4] https://westurner.github.io/hnlog/#story-18225197

> The unusual wind-powered car hit a top speed 2.86 times faster than the wind during one recent run,

I can't even.

> Working with other scientists, Lin developed a model to represent how flowing air carries these substances. The new model divided the atmosphere into cells or boxes and used computer code based on the laws of physics to simulate how air and chemical substances move through each cell and around the globe.

> The model paid close attention to conserving energy, mass and momentum in the atmosphere in each box. This precision resulted in dramatic improvements in the accuracy and realism of the atmospheric chemistry.

Global Forecast System > Future https://en.wikipedia.org/wiki/Global_Forecast_System#Future

It's more work to verify all formulas that reference unnamed variables in a spreadsheet than to review the code inputs and outputs in a notebook.

"Teaching Pandas and Jupyter to Northwestern journalism students" [in DC] https://www.californiacivicdata.org/2017/06/07/dc-python-not...

> http://www.firstpythonnotebook.org/

You can also develop d3.js visualizations — just like NYT — with jupyter notebooks and whichever language(s).

"Data-Driven Journalism" ("ddj") https://en.wikipedia.org/wiki/Data-driven_journalism

http://datadrivenjournalism.net/

"The Data Journalism Handbook 1" https://datajournalism.com/read/handbook/one

"The Data Journalism Handbook 2" https://datajournalism.com/read/handbook/two

While there are a number of ScholarlyArticle journals that can publish notebooks, I'm not aware of any newspapers that are prepared to publish notebooks as NewsArticles. It's pretty easy to `jupyter convert --to html` and `--to markdown` or just 'Save as'

Regarding expressing facts as verifiable claims with structured data in HTML and/or blockchains: "Fact Checks" https://news.ycombinator.com/item?id=15529140

Does this course recommend linking to every source dataset and/or including full citations (with DOI) in the article? Does this course recommend getting a free DOI for the published revision of an e.g. GitHub project repository (containing data, and notebooks and/or the article text) with Zenodo?

Group decision-making > Formal systems: https://en.wikipedia.org/wiki/Group_decision-making#Formal_s...

> Consensus decision-making, Voting-based methods, Delphi method, Dotmocracy

Consensus decision-making: https://en.wikipedia.org/wiki/Consensus_decision-making

There's a field that some people are all calling "Collaboration Engineering". I learned about this from a university course in Collaboration.

6 Patterns of Collaboration [GRCOEB] — Generate, Reduce, Clarify, Organize, Evaluate, Build Consensus

7 Layers of Collaboration [GPrAPTeToS] — Goals, Products, Activities, Patterns of Collaboration, Techniques, Tools, Scripts

The group decision making processes described in the article may already be defined with the thinkLets design pattern language.

A person could argue against humming for various unspecified reasons.

I'll just CC this here from my notes, which everyone can read here [1]:

“Collaboration Engineering: Foundations and Opportunities” de Vreede (2009) http://aisel.aisnet.org/jais/vol10/iss3/7/

“A Seven-Layer Model of Collaboration: Separation of Concerns for Designers of Collaboration Systems” Briggs (2009) http://aisel.aisnet.org/icis2009/26/

Six Patterns of Collaboration “Defining Key Concepts for Collaboration Engineering” Briggs (2006) http://aisel.aisnet.org/amcis2006/17/

“ThinkLets: Achieving Predictable, Repeatable Patterns of Group Interaction with Group Support Systems (GSS)” http://www.academia.edu/259943/ThinkLets_Achieving_Predictab...

https://scholar.google.com/scholar?q=thinklets

[1] https://wrdrd.github.io/docs/consulting/team-building#collab...

It at least makes sense to pay for doctors and nurses to go to school, right? If you want to care for others and you do the work to earn satisfactory grades, I think that investing in your education would have positive ROI.

We had plans here in the US to pay for two years of community college for whoever ("America's College Promise"). IDK what happened to that? We should have called it #ObamaCollege so that everyone could attack corporate welfare and bad investments with no ROI.

New York has the Excelsior scholarship for CUNY and SUNY. Tennessee pays for college with lottery proceeds. Are there other state-level efforts to fund higher education in the US such that students can finish school debt-free or close to it?

There are MOOCs (online courses) which are worth credit hours for the percentage of people that commit to finishing the course. https://www.classcentral.com/

Khan Academy has free SAT, MCAT, NCLEX-RN, GMAT, and LSAT test prep and primary and supplementary learning resources. https://www.khanacademy.org/test-prep

Free education: https://en.wikipedia.org/wiki/Free_education

> I'm a software engineer with a diverse background in backend, frontend development.

> How do I find jobs related to tackling global warming and climate change in Europe for an English speaker?

While not directly answering the question, here are some ideas for purchasing, donating, creating new positions, and hiring people that care:

Write more efficient code. Write more efficient compilers. Optimize interpretation and compilation so that the code written by people with domain knowledge who aren't that great at programming who are trying to solve other important problems is more efficient.

Push for PPAs (Power Purchase Agreements) that offset energy use. Push for directly sourcing clean energy.

Use services that at least have 100% PPAs for the energy they use: services that run on clean energy sources.

Choose green datacenters.

- [ ] Add the capability for cloud resource schedulers like Kubernetes and Terraform to prefer or require clean energy datacenters.

Choose to work with companies that voluntarily choose to do sustainability reporting.

Work to help develop (and popularize) blockchain solutions that are more energy efficient and that have equal or better security assurances as less efficient chains.

Advocate for clean energy. Donate to NGOs working for our environment and for clean energy.

Invest in clean energy. There are a number of clean energy ETFs, for example. Better energy storage is a good investment.

Push for certified green buildings and datacenters.

- [ ] We should create some sort of a badge and structured data (JSONLD, RDFa, Microdata) for site headers and/or footers that lets consumers know that we're working toward '200% green' so that we can vote with our money.

Do not vote for people who are rolling back regulations that protect our environment. Pay an organization that pays lobbyists to work the system: that's the game.

Help explain why it's both environment-rational and cost-rational to align with national and international environmental sustainability and clean energy objectives.

Argue that we should make external costs internal in order that markets will optimize for what we actually want.

There's money to be earned in solving for the #GlobalGoals Goals, Targets, and Indicators:

The Global Goals

1. No Poverty

2. Zero Hunger

3. Good Health & Well-Being

4. Quality Education

5. Gender Equality

6. Clean Water & Sanitation

7. Affordable & Clean Energy

8. Decent Work & Economic Growth

9. Industry, Innovation & Infrastructure

10. Reduced Inequalities

11. Sustainable Cities and Communities

12. Responsible Consumption & Production

13. Climate Action

14. Life Below Water

15. Life on Land

16. Peace and Justice & Strong Institutions

17. Partnerships for the Goals

https://en.wikipedia.org/wiki/Sustainable_Development_Goals

> The U.S. Digital Service isn’t perfect, but it is clearly working. The team estimates that for every $1 million invested in USDS that the government will avoid spending $5 million and save thousands of labor hours. Over a five-year period, the team’s efforts will save $1.1 billion, redirect almost 2,000 labor years towards higher value work, and generate over 400 percent return on investment. Most importantly, USDS will continue to deliver better government services for the American people, including Veterans who deserve better.

> In the private sector, these kinds of numbers would not lead to a 50 percent cut in budget. Instead, you’d clearly invest further with that kind of return. Considering the ambitious goals set out in the President’s Management Agenda, the Trump Administration should double down on better support for the public, our troops, and our veterans. The best way to do that is clearly through investments like USDS.

Why would you halve the budget of a team that's yielding a more than 400% ROI (in terms of cost savings)?

https://en.wikipedia.org/wiki/United_States_Digital_Service

Apparently the deaths are justified because energy.

Are the subsidies and taxes (incentives and penalties) rational in light of the relative harms of each form of energy?

"Study: U.S. Fossil Fuel Subsidies Exceed Pentagon Spending" https://www.rollingstone.com/politics/politics-news/fossil-f...

> The IMF found that direct and indirect subsidies for coal, oil and gas in the U.S. reached $649 billion in 2015. Pentagon spending that same year was $599 billion.

> The study defines “subsidy” very broadly, as many economists do. It accounts for the “differences between actual consumer fuel prices and how much consumers would pay if prices fully reflected supply costs plus the taxes needed to reflect environmental costs” and other damage, including premature deaths from air pollution.

IDK whether they've included the costs of responding to requests for help with natural disasters that are more probable due to climate change caused by these "externalties" / "external costs" of fossil fuels.

A thing may be identified by a URI (/person/123) for which there are zero or more URL routes (/person/123, /v1/person/123). Each additional route complicates caching; redirects are cheap for the server but slower for clients.

JSONLD does define a standard way to indicate that a value is a link: @id (which can be specified in a/an @context) https://www.w3.org/TR/json-ld11/

One additional downside to storing URIs instead of bare references is that it's more complicated to validate a URI template than a simple regex like \d+ or [abcdef\=\d+]+

/usr/bin/python can point to either /usr/bin/python3 or (as PEP 394 currently recommends) /usr/bin/python2

  $ alternatives --config python