Contents^
Items^
Ask HN: Resources to encourage teen on becoming computer engineer?
Howdy HN
A teenager I am close with would like to become a computer engineer. Whet resources, books, podcasts, camps, or experiences do you recommend to support this teen's endeavor?
CadQuery: A Python parametric CAD scripting framework based on OCCT
I wish more people would try this over openscad. I like openscad but the UI and with cadquery you use python! and now I noticed "CadQuery supports Jupyter notebook out of the box"
The jupyter-cadquery extension renders models with three.js via pythreejs in a sidebar with jupyterlab-sidecar: https://github.com/bernhard-42/jupyter-cadquery#b-using-a-do...
https://github.com/bernhard-42/jupyter-cadquery/blob/master/...
Array Programming with NumPy
Looks like there's a new citation for NumPy in town.
"Citing packages in the SciPy ecosystem" lists the existing citations for SciPy, NumPy, scikits, and other -Py things: https://www.scipy.org/citing.html ( source: https://github.com/scipy/scipy.org/blob/master/www/citing.rs... )
A better way to cite requisite software might involve referencing a https://schema.org/SoftwareApplication record in JSON-LD, RDFa, or Microdata; for example: https://news.ycombinator.com/item?id=24489651
But there's as of yet no way to publish JSON-LD, RDFa, or Microdata Linked Data from LaTeX with Computer Modern.
For some reason this struck me as inappropriate for the outlet. It's a nice piece as an introduction to array programming with numpy, but seemed out of place to me.
If, going forward, 5% of all papers that use NumPy to get their results actually cite this paper, it will be one of Nature's most cited papers every year.
There's an interesting trend of what content gets published in peer-reviewed journals vs. blogs/github/etc. I suspect there is an audience segment that strongly values peer reviewed pieces that are equivalent content wise to introductory material in a variety of formats.
I wonder if github should add a "Review" feature to provide a similar content authoring experience.
It would be nice if citing repositories were easier-- either for generating a reference for my own code or acknowledging when I've used someone else's code in my research.
There's tons of math and physics blogs that contain useful results that the author wanted to make available but didn't manage to incorporate into a paper. I wonder if there'd be any interest in a sort of GitHub for proofs? It could even use git, since (assuming consistency) isn't math just a DAG anyways (and therefore isomorphic to a neural net, as are all things).
You can get a free DOI for and archive a tag of a Git repo with FigShare or Zenodo.
If you have repo2docker REES dependency scripts (requirements.txt, environment.yml, postInstall,) in your repo, a BinderHub like https://mybinder.org can build and cache a container image and launch a (free) instance in a k8s cloud.
Journals haven't yet integrated with BinderHub.
Putting the suggested citation and DOI URI/URL in your README and cataloging citations in an e.g. wiki page may increase the crucial frequency of citation.
A Linked Data format for presenting well-formed arguments with #StructuredPremises would help to realize the potential of the web as a graph of resources which may satisfy formal inclusion criteria for #LinkedMetaAnalyses.
The issue is that none of the citation count engines (Google scholar, scopus, Web of Science...) count citations on those DOIs. So for a researcher who needs to somehow demonstrate impact through citation counts, it does not really help unfortunately.
We could reason about sites that index https://schema.org/ScholarlyArticle according to our own and others' observations. Google Scholar, Semantic Scholar, and Meta all index Scholarly Articles: they copy the bibliographic metadata and the abstract for archival and schoarly purposes.
AFAIU, e.g. Zotero and Mendeley do not crawl and index articles or attempt to parse bibliographic citations from the astounding plethora of citation styles [citationstyles, citationstyles_stylerepo] into a citation graph suitable for representative metrics [zenodo_newmetrics].
bitcoin.org/bitcoin.pdf does not have a DOI, does not have an ORCID [orcid], and is not published in any journal but is indexed by e.g. Google Scholar; though there are apparently multiple records referring to a ScholarlyArticle with the same name and author. Something like "Hell's Angels" (1930)? No DOI, no ORCID, no parseable PDF structure: not indexed.
AFAIU, Google Scholar does not yet index ScholarlyArticle (or SoftwareApplication < CreativeWork) bibliographic metadata. GScholar indexes an older set of bibliographic metadata from HTML <meta> tags and also attempts to parse PDFs. [gscholar_inclusion]
Google Scholar is also not (yet?) integrated with Google Dataset Search (which indexes https://schema.org/Dataset metadata).
FigShare DOIs and Zenodo DOIs are DataCite DOIs [figshare_howtocite, zenodo_principles]; which apparently aren't (yet?) all indexed by Google Scholar [rescience_gscholar].
IIUC, all papers uploaded to https://arxiv.org are indexed by Google Scholar. In order for arxiv-vanity.org [arxiv_vanity] to render a mobile-ready, font-resizeable HTML5 version of a paper uploaded to ArXiV, the PostScript source must be uploaded. Arxiv hosts certain categories of ScholarlyArticles.
JOSS (Journal of Open Source Software) has managed to get articles indexed by Google Scholar [rescience_gscholar]. They publish their costs [joss_costs]: $275 Crossref membership, DOIs: $1/paper:
> Assuming a publication rate of 200 papers per year this works out at ~$4.75 per paper
[citationstyles]: https://citationstyles.org
[citationstyles_stylerepo]: https://github.com/citation-style-language/styles
[gscholar_inclusion]: https://scholar.google.com/intl/en/scholar/inclusion.html#in...
[figshare_howtocite]: https://knowledge.figshare.com/articles/item/how-to-share-ci...
[zenodo_principles]: https://about.zenodo.org/principles/
[zenodo_newmetrics]: https://www.frontiersin.org/articles/10.3389/frma.2017.00013...
[rescience_gscholar]: https://github.com/ReScience/ReScience/issues/38
[arxiv_vanity]: https://www.arxiv-vanity.com/
[joss_costs]: https://joss.theoj.org/about#costs
[orcid]: https://en.wikipedia.org/wiki/ORCID
Do you like the browser bookmark manager?
How do you think it compares to services like webcull.com, raindrop.io, or getpocket.com? Have they advanced the field to the point that it's worth switching?
Things I'd add to browser bookmark managers someday:
- Support for (persisting) bookmarks tags. From the post re: the re-launch of del.icio.us: https://news.ycombinator.com/item?id=23985623
> "Allow reading and writing bookmark tags" https://bugzilla.mozilla.org/show_bug.cgi?id=1225916
> Notes re: how this could be standardized with JSON-LD: https://bugzilla.mozilla.org/show_bug.cgi?id=1225916#c116
> The existing Web Experiment for persisting bookmark tags: https://github.com/azappella/webextension-experiment-tags/bl...
- Standard search features like operators: ((term) AND (term2)) OR term3
- Regex search
- (Chrome) show the createdDate and allow (non-destructive) sort by date
- Native sync API for syncing to zero or more bookmarks / personal data storage providers
- Support for integration with extensions that support actual resource metadata like Zotero
- Linked Data support: extract and store bibliographic metadata like Zotero and OpenLink Structured Data Sniffer
What are the current limitations of the WebExtensions Bookmarks API (now supported by Firefox, Chrome, Edge, and hopefully eventually Safari)?: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/Web...
NIST Samate – Source Code Security Analyzers
Additional lists of static analysis, dynamic analysis, SAST, DAST, and other source code analysis tools:
OWAP > Source Code Analysis Tools: https://owasp.org/www-community/Source_Code_Analysis_Tools
https://analysis-tools.dev/ (supports upvotes and downvotes)
analysis-tools-dev/static-analysis: https://github.com/analysis-tools-dev/static-analysis
analysis-tools-dev/dynamic-analysis: https://github.com/analysis-tools-dev/dynamic-analysis
devsecops/awesome-devsecops: https://github.com/devsecops/awesome-devsecops , https://github.com/TaptuIT/awesome-devsecops
kai5263499/awesome-container-security: https://github.com/kai5263499/awesome-container-security
https://en.wikipedia.org/wiki/DevOps#DevSecOps,_Shifting_Sec... :
> DevSecOps is an augmentation of DevOps to allow for security practices to be integrated into the DevOps approach. The traditional centralised security team model must adopt a federated model allowing each delivery team the ability to factor in the correct security controls into their DevOps practices.
awesome-safety-critical: https://awesome-safety-critical.readthedocs.io/en/latest/
A Handwritten Math Parser in 100 lines of Python
Slightly more amusingly, but non-equivalently:
import sys
from functools import reduce
from operator import add, sub, mul, truediv as div
def calc(s, ops=[(add, '+'), (sub, '-'), (mul, '*'), (div, '/')]):
if not ops:
return float(s)
(func, token), *remaining_ops = ops
return reduce(func, (calc(part, remaining_ops) for part in s.split(token)))
if __name__ == '__main__':
print(calc(''.join(sys.argv[1:])))
This is pretty clever. There's a video by Jonathan Blow, creator of Braid and currently making a programming language called Jai, that talks about parsing expressions with order of operations. He gives an simple approach that apparently was discovered recently (though he didn't remember the paper name) that does this in a single pass and hardly any more complicated than what you wrote. You can see it here [1]. He also trashes on using yacc/bison before that and on most academic parsing theory.
Reverse Polish notation (RPN) > Converting from infix notation https://en.wikipedia.org/wiki/Reverse_Polish_notation#Conver... > Shunting-yard algorithm https://en.wikipedia.org/wiki/Shunting-yard_algorithm
Infix notation supports parentheses.
Infix notation: 3 + 4 × (2 − 1)
RPN: 3 4 2 1 − × +
PEP – An open source PDF editor for Mac
I have a dream... that one day people will name their projects with names that don't exist on google yet.
If you search for PEP now you'll find python enhancement proposals, and the "Philippine Entertainment Portal" and the stock code for PepsiCo.
I wish that once people do name their project, they would assign it a 128-bit random number in lower case hex, and include that number on any web page that they would like people searching for their project to find.
That way once I know that say PEP the PDF editor exists and find its 128-bit number (let's say that is 379dd864b16eaca3ce94c15a6bdfcc73), at least I can subsequently toss a +379dd864b16eaca3ce94c15a6bdfcc73 on my searches to effectively let the search engine know I want PEP the PDF editor results rather than PEP the python enhancement results or PEP the entertainment portal results or PEP that refreshing beverage company stock symbol.
"xxd -l 16 -p /dev/urandom" is a handy way to get a 128-bit random hex number. A UUID generator works, too, although they usually include some punctuation you will need to delete and you might have to lower case their output.
> I wish that once people do name their project, they would assign it a 128-bit random number in lower case hex
We already have something similar: URLs.
tzs is proposing URNs rather than textual program names. A URL is unnecessarily specific (though I suppose you could anycast URL resolution)
> RFC 4122 defines a Uniform Resource Name (URN) namespace for UUIDs. A UUID presented as a URN appears as follows:[1]
> > urn:uuid:123e4567-e89b-12d3-a456-426655440000
https://en.wikipedia.org/wiki/Universally_unique_identifier#...
Version 4 UUIDs have 122 random bits (out of 128 bits total).
In Python:
>>> import uuid
>>> _id = uuid.uuid4()
>>> _id.urn
'urn:uuid:4c466878-a81b-4f22-a112-c704655fa4ee'
>>> _id.hex
'4c466878a81b4f22a112c704655fa4ee'
{"@context": "https://schema.org",
"@type": "WebPage",
"about": {
"@type": "SoftwareApplication",
"identifier": "urn:uuid:4c466878-a81b-4f22-a112-c704655fa4ee",
"url": ["", ""],
"name": [
"a schema.org/SoftwareApplication < CreativeWork < Thing",
{"@value": "a rose by any other name",
"@language": "en"}]}}
<body vocab="https://schema.org/" typeof="WebPage">
<div property="about" typeof="SoftwareApplication">
<meta property="identifier" content="urn:uuid:4c466878-a81b-4f22-a112-c704655fa4ee"/>
<span property="name">a schema.org/SoftwareApplication < CreativeWork < Thing</span>
<span property="name" lang="en">a rose by any other name</span>
</div>
</body>
<div itemtype="https://schema.org/WebPage" itemscope>
<link itemprop="http://www.w3.org/ns/rdfa#usesVocabulary" href="https://schema.org/" />
<div itemprop="about" itemtype="https://schema.org/SoftwareApplication" itemscope>
<meta itemprop="identifier" content="urn:uuid:4c466878-a81b-4f22-a112-c704655fa4ee" />
<meta itemprop="name" content="a schema.org/SoftwareApplication < CreativeWork < Thing"/>
<meta itemprop="name" content="a rose by any other name" lang="en"/>
</div>
</div>The Unix timestamp will begin with 16 this Sunday
Redox: Unix-Like Operating System in Rust
I'm Jeremy Soller, the creator of Redox OS. Let me know if you have questions!
Are there tools to support static analysis and formal methods in Rust yet?
From https://news.ycombinator.com/item?id=21839514 re: awesome-safety-critical https://awesome-safety-critical.readthedocs.io/en/latest/ :
> > Does Rust have a chance in mission-critical software? (currently Ada and proven C niches) https://www.reddit.com/r/rust/comments/5iv5j7/does_rust_have...
FWIU, Sealed Rust is in progress.
And there's also RustPython for the userspace.
Ask HN: How are online communities established?
HN, Reddit, Stack Overflow, etc. are all established communities with users. How do you start a community when you don't have any users?
You should read the book People Powered by Jono Bacon. Has some really good insights and is a 101 course on exactly this.
Seconded. "People Powered: How Communities Can Supercharge Your Business, Brand, and Teams" (2019) https://g.co/kgs/CF5TEk
"The Art of Community: Building the New Age of Participation" (2012) https://g.co/kgs/P2V1kn
"Tribes: We need you to lead us" (2011) https://g.co/kgs/T8jaFS
The 1% 'rule' https://en.wikipedia.org/wiki/1%25_rule_(Internet_culture) :
> In Internet culture, the 1% rule is a rule of thumb pertaining to participation in an internet community, stating that only 1% of the users of a website add content, while the other 99% of the participants only lurk. Variants include the 1–9–90 rule (sometimes 90–9–1 principle or the 89:10:1 ratio),[1] which states that in a collaborative website such as a wiki, 90% of the participants of a community only consume content, 9% of the participants change or update content, and 1% of the participants add content.
... Relevant metrics:
- Marginal cost of service https://en.wikipedia.org/wiki/Marginal_cost
- Customer acquisition cost: https://en.wikipedia.org/wiki/Customer_acquisition_cost
- [Quantifiable and non-quantifiable] Customer Lifetime Value: https://en.wikipedia.org/wiki/Customer_lifetime_value
Last words of the almost-cliche community organizer surrounded by dormant accounts: "Network effects will result in sufficient (grant) funding"
Business model examples that may be useful for building and supporting sustainable communities with clear Missions, Objectives, and Criteria for Success: https://gist.github.com/ndarville/4295324
Python Documentation Using Sphinx
I usually generate new Python projects with a cookiecutter; such as cookiecutter-pypackage. I like the way that cookiecutter-pypackage includes a Makefile which has a `docs` task so that I can call `make docs` to build the sphinx docs in the docs/ directory which include:
- a /docs/readme.rst that includes the /README.rst as the first document in the toctree
- a sensible set of default documents: readme (.. include:: /README.rst), installation, usage, modules (sphinx-autodoc output), contributing, authors, history (.. include:: /HISTORY.rst)
- a sphinx conf.py that sets the docs' version and release attributes to pkgname.__version__; so that the version number only needs to be changed in one place (as long as setup.py or setup.cfg also read the version string from pkgname.__version__)
- a default set of extensions: ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] that generates API docs and includes '[source]' hyperlinks from the generated API docs to the transcluded syntax-highlighted source code and links back to the API docs from the source code
https://github.com/audreyfeldroy/cookiecutter-pypackage/tree...
There are a few styles of docstrings that Sphinx can parse and include in docs with e.g. sphinx-autodoc:
`:param, :type, :returns, :rtype` docstrings (which OP uses; and which pycontracts can read runtime parameter and return type contracts from https://andreacensi.github.io/contracts/ (though Python 3 annotations are now the preferred style for compile or editing-time typechecks))
Numpydoc docstrings: https://numpydoc.readthedocs.io/en/latest/format.html
Googledoc docstrings: https://sphinxcontrib-napoleon.readthedocs.io/en/latest/
You can use Markdown with Sphinx in at least three ways:
MyST Markdown supports Sphinx and Docutils roles and directives. Jupyter Book builds upon MyST Markdown. With Jupyter Book, you can include Jupyter notebooks (which can include MyST Markdown) in your Sphinx docs. Executable notebooks are a much easier way to include up-to-date code outputs in docs. https://myst-parser.readthedocs.io/en/latest/
Sphinx (& ReadTheDocs) w/ recommonmark: https://docs.readthedocs.io/en/stable/intro/getting-started-...
Nbsphinx predates Jupyter Book and doesn't yet support MyST Markdown, but does support Markdown cells in Jupyter notebooks. Nbsphinx includes a parser for including .ipynb Jupyter notebooks in Sphinx docs. nbsphinx supports raw RST (ReST) cells in Jupyter notebooks and has great docs: https://nbsphinx.readthedocs.io/en/latest/
Nbdev is another approach; though it's not Sphinx:
> nbdev is a library that allows you to fully develop a library in Jupyter Notebooks, putting all your code, tests and documentation in one place.
> [...] Add %nbdev_export flags to the cells that define the functions you want to include in your python modules
https://github.com/fastai/nbdev
A few additional sources of docs for Sphinx and ReStructuredText:
Read The Docs docs > Getting Started with Sphinx > External Resources https://docs.readthedocs.io/en/stable/intro/getting-started-...
CPython Devguide > "Documenting Python" https://devguide.python.org/documenting/
"How to write [Linux] kernel documentation" https://www.kernel.org/doc/html/latest/doc-guide/index.html
awesome-sphinxdoc: https://github.com/yoloseem/awesome-sphinxdoc
... "Ask HN: Recommendations for Books on Writing [for engineers]?" https://news.ycombinator.com/item?id=23945580
Traits of good remote leaders
And the evidence for these considerations is what?
They reference a study: https://link.springer.com/article/10.1007%2Fs10869-020-09698..., titled: "Who Emerges into Virtual Team Leadership Roles? The Role of Achievement and Ascription Antecedents for Leadership Emergence Across the Virtuality Spectrum".
Fortunately the references are free to view.
"Table 4 – Correlation of Development Phases, Coping Stages and Comfort Zone transitions and the Performance Model" in "From Comfort Zone to Performance Management" White (2008) tabularly correlates the Tuckman group development phases (Forming, Storming, Norming, Performing, Adjourning) with the Carnall coping cycle (Denial, Defense, Discarding, Adaptation, Internalization) and Comfort Zone Theory (First Performance Level, Transition Zone, Second Performance Level), and the White-Fairhurst TPR model (Transforming, Performing, Reforming). The ScholarlyArticle also suggests management styles for each stage (Commanding, Cooperative, Motivational, Directive, Collaborative); and suggests that team performance is described by chained power curves of re-progression through these stages.
https://scholar.google.com/scholar?hl=en&as_sdt=0%2C43&q=%E2...
IDK what's different about online teams in regards to performance management?
Show HN: Eiten – open-source tool for portfolio optimization
Is it possible to factor (e.g. GRI) sustainability criteria into the portfolio fitness function? https://news.ycombinator.com/item?id=21922558
My concern is that - like any other portfolio optimization algorithm - blindly optimizing on fundamentals and short term returns will lead to investing in firms who just dump external costs onto people in the present and future; so, screening with sustainability criteria is important to me.
From https://news.ycombinator.com/item?id=19111911 :
> awesome-quant lists a bunch of other tools for algos and superalgos: https://github.com/wilsonfreitas/awesome-quant
This is perfect. Thank you for sharing this.
I might start implementing some of these but would love for someone else to add a few PRs as well. The code is pretty modular especially if we want to add new strategies.
(Sustainable) Index ETFs in the stocks.txt universe would likely be less sensitive to single performers' effects in unbalanced portfolios.
> pyfolio.tears.create_interesting_times_tear_sheet measures algorithmic trading algorithm performance during "stress events" https://github.com/quantopian/pyfolio/blob/03568e0f328783a6a...
Ask HN: Any well funded tech companies tackling big, meaningful problems?
Are there any well funded tech startups / companies tackling major societal problems? Any of these fair game: https://en.wikipedia.org/wiki/List_of_global_issues
----
I don't see or hear of any and want to know if this is just my bias or if there really is a shortage of resources in tech being allocated to solving the worlds most important problems. I'm sure I'm not the only engineer that's looking out for companies like this.
Ran into this previous Ask HN (https://news.ycombinator.com/item?id=24168902) that asked a similar question. However, here I wanna focus on the better funded efforts (not side projects, philanthropy etc).
One example I've heard so far is Tesla. Any others?
You can make an impact by solving important local and global problems by investing your time, career, and savings; by listing and comparing solutions.
As a labor market participant, you can choose to work for places that have an organizational mission that strategically aligns with local, domestic, and international objectives.
https://en.wikipedia.org/wiki/Strategic_alignment ... "Schema.org: Mission, Project, Goal, Objective, Task" https://news.ycombinator.com/item?id=12525141
As an investor, you can choose to invest in organizations that are making the sort of impact you're looking for: you can impact invest.
https://en.wikipedia.org/wiki/Impact_investing
You mentioned "List of global issues"; which didn't yet have a link to the UN Sustainable Development Goals (the #GlobalGoals). I just added this to the linked article:
> As part of the 2030 Agenda for Sustainable Development, the UN Millenium Development Goals (2000-2015) were superseded by the UN Sustainable Development Goals (2016-2030), which are also known as The Global Goals. There are associated Targets and Indicators for each Global Goal.
There are 17 Global Goals.
Sustainability reporting standards can align with the Sustainable Development Goals. For example, the GRI standards are now aligned with the UN Sustainable Development Goals.
https://en.wikipedia.org/wiki/Sustainable_Development_Goals
Investors, fund managers, and potential employees can identify companies which are making an impact by reviewing corporate sustainability and ESG reports.
From https://www.undp.org/content/undp/en/home/sustainable-develo... :
> SDG Target 12.6: "Encourage companies, especially large and transnational companies, to adopt sustainable practices and to integrate sustainability information into their reporting cycle"
From https://news.ycombinator.com/item?id=21302926 :
> > What are some of the corporate sustainability reporting standards?
> > From https://en.wikipedia.org/wiki/Sustainability_reporting#Initi... :
> >> Organizations can improve their sustainability performance by measuring (EthicalQuote (CEQ)), monitoring and reporting on it, helping them have a positive impact on society, the economy, and a sustainable future. The key drivers for the quality of sustainability reports are the guidelines of the Global Reporting Initiative (GRI),[3] (ACCA) award schemes or rankings. The GRI Sustainability Reporting Guidelines enable all organizations worldwide to assess their sustainability performance and disclose the results in a similar way to financial reporting.[4] The largest database of corporate sustainability reports can be found on the website of the United Nations Global Compact initiative.
> >The GRI (Global Reporting Initiative) Standards are now aligned with the UN Sustainable Development Goals (#GlobalGoals). https://en.wikipedia.org/wiki/Global_Reporting_Initiative
> >> In 2017, 63 percent of the largest 100 companies (N100), and 75 percent of the Global Fortune 250 (G250) reported applying the GRI reporting framework.[3]
What are some good ways to search for companies who (1) do sustainability reports, (2) engage in strategic alignment in corporate planning sessions, (3) make sustainability a front-and-center issue in their company's internal and external communications?
What are some examples of companies who have a focus on sustainability and/or who have developed a nonprofit organization for philanthropic missions which are sometimes best accounted for as a distinct organization or a business unit (which can accept and offer receipts for donations as a non-profit)?
How can an employee drive change in a small or a large company? Identify opportunities to deliver value and goodwill. Read through the Global Goals, Targets, and Indicators; and get into the habit of writing down problems and solutions.
3 pillars of [Corporate] Sustainability: (Environment (Society (Economy))). https://en.wikipedia.org/wiki/Sustainability#Three_dimension...
"Launch HN: Charityvest (YC S20) – Employee charitable funds and gift matching" https://news.ycombinator.com/item?id=23907902 :
> We created a modern, simple, and affordable way for companies to include charitable giving in their suite of employee benefits.
> We give employees their own tax-deductible charitable giving fund, like an “HSA for Charity.” They can make contributions into their fund and, from their fund, support any of the 1.4M charities in the US, all on one tax receipt.
> Using the funds, we enable companies to operate gift matching programs that run on autopilot. Each donation to a charity from an employee is matched automatically by the company in our system.
> A company can set up a matching gift program and launch giving funds to employees in about 10 minutes of work.
"Salesforce Sustainability Cloud Becomes Generally Available" https://news.ycombinator.com/item?id=22068522 :
> Are there similar services for Sustainability Reporting and accountability?
Column Names as Contracts
I really like the idea of being more thoughtful about naming columns and being more explicit about the “type” of data contained in them.
Is this idea already known among data modelers or data engineers?
I’d love to read any other references, if available.
I am a data architect in my day job. Within the realm of data management, I'd say "metadata management" [1] is the general category this fits within.
I would say, yes this idea is known/very common, as data architecture is as much about the descriptive language we use as anything. I mean, "business glossaries", taxonomy, even just naming conventions [2] in coding, these are all related.
If you build enough databases/tables or even code yourself, you inevitably come across the "how to name things" problem [3]. If all you have to sort on for the known meaning of a thing (column, table, file, etc.) is a single string value, then encoding meaning into it is quite common. This way, a sort creates a kind of "grouping". Many database vendors follow standard naming conventions - such as Oracle, for example [4]. It is considered a best practice when designing/building the metadata for a large system, to establish a naming convention. Among other things, it makes finding things easier, as well as all the potential for automation.
You get all kinds of variations on this, such as, should the "ID_" come as a prefix or a suffix (i.e. "_ID"). One's initial thought is to use it as a prefix so all the related types group together, but then that becomes much more difficult if you want to sort items by their functional area (e.g. DRIVER_ID, DRIVER_IND, etc.).
One other place you see something similar is in "smart numbers" which is an eternal argument - should I use a "dumb identifier" (GUID, integer) or a "smart one" (one encoding additional meaning) [5].
I mean, basically, any time you can encode information in the meta-data of data, I think you can then operate on it by following "convention over configuration" (as mentioned elsewhere in the discussion comments).
The only problem I see is that such conventions can, at times be limiting - depending on the length of your metadata columns, and the variability you are trying to capture - which is why I believe, generally, metadata is often better separated and linked to the data it describes - this decoupling allows for much more descriptive metadata than one could encode in simple a single string value. Certainly, you can get a long way with an approach like this, but I suspect you would run into 80/20 rule limitations.
Using naming in this way is a form of tight coupling, which could be seen as an anti-pattern in terms of meta-data flexibility, in some cases.
[1] https://en.wikipedia.org/wiki/Metadata_management
[2] https://en.wikipedia.org/wiki/Naming_convention_(programming...
[3] https://martinfowler.com/bliki/TwoHardThings.html
[4] https://oracle-base.com/articles/misc/naming-conventions
In terms of database normalization, delimiting multiple fields within a column name field violates the "atomic columns" requirement of the first though sixth normal forms (1NF - 6NF)
https://en.wikipedia.org/wiki/Database_normalization
Are there standards for storing columnar metadata (that is, metadata about the columns; or column-level metadata)?
In terms of columns, SQL has (implicit ordinal, name, type) and then primary key, index, and [foreign key] constraints.
RDFS (RDF Schema) is an open W3C linked data standard. An rdf:Property may have a rdfs:domain and a rdfs:range; where the possible datatypes are listed as instances of rdfs:range. Primitive datatypes are often drawn from XSD (XML Schema Definition), or https://schema.org/ . An rdfs:Class instance may be within the rdfs:domain and/or the rdfs:range of an rdf:Property.
RDFS is generally not sufficient for data validation; there are a number of standards which build upon RDFS: W3C SHACL (Shapes and Constraint Language), W3C CSVW (CSV on the Web).
There is some existing work on merging JSON Schema and SHACL.
CSVW builds upon the W3C "Model for Tabular Data and Metadata on the Web"; which supports arbitrary "annotations" on columns. CSVW can be represented as any RDF representation: Turtle/Trig/M3, RDF/XML, JSON-LD.
https://www.w3.org/TR/tabular-data-primer/
https://www.w3.org/TR/tabular-data-model/ :
> an annotated tabular data model: a model for tables that are annotated with metadata. Annotations provide information about the cells, rows, columns, tables, and groups of tables […]
...
From https://twitter.com/westurner/status/901992073846456321 :
> "7 metadata header rows (column label, property URI path, DataType, unit, accuracy, precision, significant figures)" https://wrdrd.github.io/docs/consulting/linkedreproducibilit...
...
From https://twitter.com/westurner/status/1295774405923147778 :
> Relevant: https://discuss.ossdata.org/ topics: "Linked Data formats, tools, challenges, opportunities; CSVW, https://schema.org/Dataset , https://schema.org/ScholarlyArticle " https://discuss.ossdata.org/t/linked-data-formats-tools-chal...
> "A dataframe protocol for the PyData ecosystem" https://discuss.ossdata.org/t/a-dataframe-protocol-for-the-p...
> A .meta protocol should implement the W3C Tabular Data Model: [...]
...
The various methods of doing CSV2RDF and R2RML (SQL / RDB to RDF Mapping) each have a way to specify additional metadata annotations. None stuff data into a column name (which I'm also guilty of doing with e.g. "columnspecs" in a small line-parsing utility called pyline that can cast columns to Python types and output JSON lines).
...
Even JSON5 is insufficient when it comes to representing e.g. complex fractions: there must be a tbox (schema) in order to read the data out of the abox (assertions; e.g. JSON). JSON-LD is sufficient for representation; and there are also specs like RDFS, SHACL, and CSVW.
Abox: https://en.wikipedia.org/wiki/Abox
I see the line of thinking you're going down. There are ISO standards for data types, in a sense I could see why one would seek a standard language for defining the metadata/specification of a type as data. Have to really think about that some more.. in a way a regex could be seen as a compact form of expressing the capability of a column in terms of value ranges or domains, but to define the meaning of the data, not so much.
Your interpretation of the atomic columns requirement is a little different than my understanding. That requirement of normalization only applies to the "cells" of columnar data, it says nothing about encoding meaning into column names, which are themselves simply descriptive metadata.
I mean, for sure you wouldn't want to encode many values/meanings into a column name (some systems have length restrictions that would make that impossible, I'm not sure it makes sense anyway), but just pointing out that technically the spec does not make that illegal. Certainly, adding minor annotations within the name of a column separated by a supported delimiter does not, in my opinion, violate normalization rules at all. I mean things like "ID_" or similar.
Have you looked at INFORMATION_SCHEMA in SQL databases? [1] You mentioned SQL metadata and constraints, that is as close to a standard feature for querying that information there is, some databases do it using similar but non-standard ways (Oracle for example).
Also, not standard but, many relational databases support extended properties or Metadata for objects (tables, views, columns, etc.) - you can often come up with your own scheme although rarely do I see people utilize these features. [2] [3]
At some point it feels like we are more talking about type definitions and annotations, applied to data columns.
Maybe like, BNF [4] for purely data table columns (which are essentially types)?
[1] https://en.wikipedia.org/wiki/Information_schema
[2] http://www.postgresql.org/docs/current/static/sql-comment.ht...
[3] https://docs.microsoft.com/en-us/sql/relational-databases/sy...
Graph Representations for Higher-Order Logic and Theorem Proving (2019)
ONNX (and maybe RIF) are worth mentioning.
ONNX: https://onnx.ai/ :
> ONNX is an open format built to represent machine learning models. ONNX defines a common set of operators - the building blocks of machine learning and deep learning models - and a common file format to enable AI developers to use models with a variety of frameworks, tools, runtimes, and compilers
RIF (~FOL): https://en.wikipedia.org/wiki/Rule_Interchange_Format
Datalog (not Turing-complete): https://en.wikipedia.org/wiki/Datalog
HOList Benchmark: https://sites.google.com/view/holist/home
"HOList: An Environment for Machine Learning of Higher-Order Theorem Proving" (2019) https://arxiv.org/abs/1904.03241
> Abstract: We present an environment, benchmark, and deep learning driven automated theorem prover for higher-order logic. Higher-order interactive theorem provers enable the formalization of arbitrary mathematical theories and thereby present an interesting, open-ended challenge for deep learning. We provide an open-source framework based on the HOL Light theorem prover that can be used as a reinforcement learning environment. HOL Light comes with a broad coverage of basic mathematical theorems on calculus and the formal proof of the Kepler conjecture, from which we derive a challenging benchmark for automated reasoning. We also present a deep reinforcement learning driven automated theorem prover, DeepHOL, with strong initial results on this benchmark.
I wonder why they don't mention any work based on transformer architectures? The recent work on solving differential equations based on expressions in reverse polish notation seemed like a reasonable idea to apply to theorem proving as well.
Really cool work though!
A transformer is unable to really represent logic, let alone higher order logic and theorem proving.
A transformer is a universal function approximator. The question is whether it can do so reasonably efficiently. Trained on natural language linear sequences, I’m with you. Trained on abstract logical graph representations? I don’t think that question’s answered yet, unless I’m missing something.
How do transformers handle with truth tables, logical connectives, and propositional logic / rules of inference, and first-order logic?
Truth table: https://en.wikipedia.org/wiki/Truth_table
Logical connective: https://en.wikipedia.org/wiki/Logical_connective
Propositional logic: https://en.wikipedia.org/wiki/Propositional_calculus
Rules of inference: https://en.wikipedia.org/wiki/Rule_of_inference
DL: Description logic: https://en.wikipedia.org/wiki/Description_logic (... The OWL 2 profiles (EL, QR, RL; DL, Full) have established decideability and complexity: https://www.w3.org/TR/owl2-profiles/ )
FOL: First-order logic: https://en.wikipedia.org/wiki/First-order_logic
HOL: Higher-order logic: https://en.wikipedia.org/wiki/Higher-order_logic
In terms of regurgitating without critical reasoning?
Critical reasoning: https://en.wikipedia.org/wiki/Critical_thinking
Think about a program that can write code but not execute it. It’s not hard to get a transformer to learn to write python code like merge sort or simple arithmetic code, even though a transformer can’t reasonably learn to sort, nor can it learn simple arithmetic. It’s an important disambiguation. In one view it appears it can’t (learn to sort) and in another it demonstrably can (learn to code up a sort function). It can learn to usefully manipulate the language without needing the capacity to execute the language. They probably can’t learn to “execute” what you’re talking about (execute being a loose analogy), but I’d say the jury’s out on whether they can learn to usefully manipulate it.
Transformer is a function from seq of symbols to seq of symbols. For example, truth table is exactly similar kind of function from variables to [True, False] alphabet.
Transformer can represent some very complex logical operations, and per this article is turing complete: https://arxiv.org/abs/1901.03429, meaning any computable function, including theorem prover can be represented as transformer.
Another question is if it is feasible/viable/rational to build transformer for this? My intuition says: no.
Show HN: Linux sysadmin course, eight years on
Almost eight years ago I launched an online “Linux sysadmin course for newbies” here at HN.
It was a side-project that went well, but never generated enough money to allow me to fully commit to leaving the Day Job. After surviving the Big C, and getting made redundant I thought I might improve and relaunch it commercially – but my doctors are a pessimistic bunch, so it looked like I didn’t have the time.
Instead, I rejigged/relaunched it via a Reddit forum this February as free and open - and have now gathered a team of helpers to ensure that it keeps going each month even after I can’t be involved any longer.
It’s a month-long course which restarts each month, so “Day 1” of September is this coming Monday.
It would be great if you could pass the word on to anyone you know who may be the target market of those who: “...aspire to get Linux-related jobs in industry - junior Linux sysadmin, devops-related work and similar”.
[0] http://www.linuxupskillchallenge.org/
[1] https://www.reddit.com/r/linuxupskillchallenge/
[2] http://snori74.blogspot.com/2020/04/health-status.html
There are a number of resources that may be useful for your curriculum for this project listed in "Is there a program like codeacademy but for learning sysadmin?" https://news.ycombinator.com/item?id=19469266 :
> [ http://www.opsschool.org/ , https://github.com/kahun/awesome-sysadmin/blob/master/README... , https://github.com/stack72/ops-books , https://landing.google.com/sre/books/ , https://response.pagerduty.com/ (Incident Response training)]
To that I'd add that K3D (based on K3S, which is now a CNCF project) runs Kubernetes (k8s) in Docker containers. https://github.com/rancher/k3d
For zero-downtime (HA: High availability) deployments, "Zero-Downtime Deployments To a Docker Swarm Cluster" describes Rolling Updates and Blue-Green Deployments; with illustrations: https://github.com/vfarcic/vfarcic.github.io/blob/master/doc...
For git-push style deployment with more of a least privileges approach (which also has more moving parts) you could take a look at: https://github.com/dokku/dokku-scheduler-kubernetes#function...
And also reference ansible molecule and testinfra for writing sysadmin tests and the molecule vagrant driver for testing docker configurations. https://www.jeffgeerling.com/blog/2018/testing-your-ansible-...
https://molecule.readthedocs.io/en/latest/
https://testinfra.readthedocs.io/en/latest/ :
> With Testinfra you can write unit tests in Python to test actual state of your servers configured by management tools like Salt, Ansible, Puppet, Chef and so on.
> Testinfra aims to be a Serverspec equivalent in python and is written as a plugin to the powerful Pytest test engine.
I wasn't able to find a syllabus or a list of all of the daily posts? Are you focusing on DevOps and/or DevSecOps skills?
EDIT: The lessons are Markdown files in a Git repo: https://github.com/snori74/linuxupskillchallenge
Links to each lesson, the title and/or subjects of the lesson, and the associated reddit posts might be useful in a Table of Contents in the README.md.
Thanks, but most of that would be way over the top for my "newbies".
However, You must be the third or fourth person today to suggest that I add a TOC - so that is something I think I'll need to look at!
Maybe most useful as resources for further study.
Looks like Day 20 covers shell scripting. A few things worth mentioning:
You can write tests for shell scripts and write TAP (Test Anything Protocol) -formatted output: https://testanything.org/producers.html#shell
Quoting in shell scripts is something to be really careful about:
> This and this do different things:
# prints a newline
echo $(echo "-e a\nb")
# prints "-e a\nb"
echo "$(echo "-e a\nb")"
LearnXinYminutes has a good bash reference: https://learnxinyminutes.com/docs/bash/
And an okay Ansible reference, which (like Ops School) we should contribute to: https://learnxinyminutes.com/docs/ansible/
Why do so many pros avoid maintaining shell scripts and writing one-off commands that they'll never remember to run again later?
...
It may be helpful to format these as Jupyter notebooks with input and output cells.
- Ctrl-Shift-Minus splits a cell at the cursor
- M and Y toggle a cell between Markdown and code
If you don't want to prefix every code cell line with a '!' so that the ipykernel Jupyter python kernel (the default kernel) executes the line with $SHELL, you can instead install and select bash_kernel; though users attempting to run the notebooks interactively would then need to also have bash_kernel installed: https://github.com/takluyver/bash_kernel
You can save a notebook .ipynb to any of a number of Markdown and non-Markdown formats https://jupytext.readthedocs.io/en/latest/formats.html#markd... ; unfortunately jupytext only auto-saves to md without output cell content for now: https://github.com/mwouts/jupytext/issues/220
You can make reveal.js slides (that do include outputs) from a notebook: https://gist.github.com/mwouts/04a6dfa571bda5cc59fa1429d1309...
With nbconvert, you can manually save an .ipynb Jupyter notebook as Markdown which includes the cell outputs w/ File > "Download as / Export Notebook as" > "Export notebook to Markdown" or with the CLI: https://nbconvert.readthedocs.io/en/latest/usage.html#conver...
jupyter convert --to markdown
jupyter convert --help
jupyter-book build mybook/
From https://westurner.github.io/tools/#bash :
type bash
bash --help
help help
help type
apropos bash
info bash
man bash
man man
info info
info bash -n "Bash Startup Files"
...
Re: dotfiles, losing commands that should've been logged to HISTFILE when running multiple bash sessions and why I wrote usrlog.sh: https://westurner.github.io/hnlog/#comment-20671184 (Ctrl-F for: "dotfiles", "usrlog.sh", "inputrc")
https://github.com/webpro/awesome-dotfiles
...
awesome-sysadmin > resources: https://github.com/kahun/awesome-sysadmin#resources
Software supply chain security
Estimates of prevalence do assume detection. How would we detect that a dependency that was installed a few deployments and reboots ago was compromised?
How does the classic infosec triad (Confidentiality, Integrity, Availability) apply to software supply chain security?
Confidentiality: Presumably we're talking about open source projects; which aren't confidential. Projects may request responsible disclosure in an e.g. security.txt; and vuln reports may be confidential for at least a little while.
Integrity: Secure transport protocols, checksums, and cryptographic code signing are ways to mitigate data integrity risks. GitHub supports SSH, 2FA, and GPG keys. Can all keys in the package signature keyring be used to sign any package? Can we verify a public key over a different channel? When we specify exact versions of software dependencies, can we also record package hashes which the package installer(s) will verify?
Availability: What are the internal and external data, network, and service dependencies for the development and deployment DevSecOps workflows? Can we deploy from local package mirrors? Who is responsible for securing and updating local package mirrors? Are these service dependencies all HA? Does everything in this system also depend upon the load balancer? Does our container registry support e.g. Docker Notary (TUF)? How should we mirror TUF package repos?
See also: "Guidance for [[transparent] proxy cache] partial mirrors?" https://github.com/theupdateframework/specification/issues/1...
A toolset that answers some of your questions is grafeas- a metadata store at https://github.com/grafeas/grafeas- and kritis, a policy engine at https://github.com/grafeas/kritis.
Cheers.
Thanks for the links. Do you know how this toolset helps to mitigate/prevent what is called in the GitHub blogpost "Supply chain compromises". Quickly checked around and couldn't find anything that applies to the dependencies of applications/binaries before they land into the target runtime (i.e k8s).
Have you seen these preso slides
https://www.slideshare.net/mobile/aysylu/q-con-sp-software-s....
They walk through one of the workflows (end state is deploying to k8s).
Grafeas is a metadata store, Kritis is a policy engine that plugs into k8s as an admission controller- blessing the "admission" (running) of an image in a namespace.
There are existing tools for each language/runtime that produce known vuln lists for individual artifacts in the language ecosystem. These you feed into Grafeas. And you have your CI pipeline providing manifests for each of your built images that contain all upstream dependencies (these produced from each app's build tool). Then at deploy time, Kritis checks the manifest on the image, and for each artifact in the image, checks for vulns and determines whether the vuln should keep the image from being deployed.
Hope that helps. There are many other workflows but that one is the most direct.
Cheers.
OUTSTANDING comment; excellent questions. Bookmarked. Thanks for this concise high-level infosec punchlist.
This Sir is senior.
Mind Emulation Foundation
"While we are far from understanding how the mind works, most philosophers and scientists agree that your mind is an emergent property of your body. In particular, your body’s connectome. Your connectome is the comprehensive network of neural connections in your brain and nervous system. Today your connectome is biological. "
This is a pretty speculative thesis. It's not at all clear that everything relevant to the mind is found in the connections rather than the particular biochemical processes of the brain. It's a very reductionist view that drastically underestimates the biological complexity of even individual cells. There's a good book, Wetware: A Computer in Every Living Cell, by Dennis Bray going into detail on how much functionality and physical processes are at work even in the most simplest cells that is routinely ignored by these analogies of the brain to a digital computer.
There is this extreme, and I would argue unscientific bias towards treating the mind as something that's recreatable in a digital system probably because it enables this science-fiction speculation and dreams of immortality of people living in the cloud.
Indeed. We humans largely create devices that function either through calculation or through physical reaction, relying on the underlying rules of the universe to "do the math" of, say, launching a cannonball and having it follow a consistent arc. The brain combines both at almost every level. It may be fundamentally impossible to emulate a human personality equal to a real one without a physics simulation of a human brain and its chemistry.
A dragonfly brain takes the input from thirty thousand visual receptor cells and uses it to track prey movement using only sixteen neurons. Could we do the same using an equal volume of transistors?
No one is saying a neuron is a one to one equivalent with a transistor. That behavior does seem like it's possible to emulate with many transistors, however.
Was just talking about quantum cognition and memristors (in context to GIT) a few days ago: https://news.ycombinator.com/item?id=24317768
Quantum cognition: https://en.wikipedia.org/wiki/Quantum_cognition
Memristor: https://en.wikipedia.org/wiki/Memristor
It may yet be possible to sufficiently functionally emulate the mind with (orders of magnitude more) transistors. Though, is it necessary to emulate e.g. autonomic functions? Do we consider the immune system to be part of the mind (and gut)?
Perhaps there's something like an amplituhedron - or some happenstance correspondence - that will enable more efficient simulation of quantum systems on classical silicon pending orders of magnitude increases in coherence and also error rate in whichever computation medium.
For abstract formalisms (which do incorporate transistors as a computation medium sufficient for certain tasks), is there a more comprehensive set than Constructor Theory?
Constructor theory: https://en.wikipedia.org/wiki/Constructor_theory
Amplituhedron: https://en.wikipedia.org/wiki/Amplituhedron
What is the universe using our brains to compute? Is abstract reasoning even necessary for this job?
Something worth emulating: Critical reasoning. https://en.wikipedia.org/wiki/Critical_reasoning
13 Beautiful Tools to Enhance Online Teaching and Learning Skills
"Options for giving math talks and lectures online" https://news.ycombinator.com/item?id=22541754 also lists a number of resources for teaching math online.
How close are computers to automating mathematical reasoning?
It's irresponsible to not mention that a series of results from the 1930s through the 1950s proved that generalized automated proof search is impossible, in the sense that we cannot just ask a computer to find a proof or refutation of Goldbach's Conjecture or other "Goldbach-type" statement and have it do any better than a person. In that sense, no, computers will never fully automate mathematical reasoning, because mathematical reasoning cannot be fully automated.
What we can automate, and have successfully automated many times over, is proof checking. In that sense, yes, computers have already fully automated mathematical reasoning, because checking proofs is fully formalized and mechanized.
I think that what people really ought to be asking for is the ease with which we find new results and communicate them to others. In that sense, computer-aided proofs can be hard to read and so there is much work to be done in making them easier to communicate to humans. Similarly, there is interesting work being done on how to make computer-generated proofs which use extremely-high-level axiom schemata to generate human-like handwaving abstractions.
I'm still not sure how I feel about the ATP/ITP terminology used here. ATPs are either of the weak sort that crunch through SAT, graphs, and other complete-but-hard problems, or the strong sort which are impossible. Meanwhile, folks have drifted from "interactive" to "assistant", and talk of "proof assistants" as tools which, like a human, can write down and look at sections of a proof in isolation, but cannot summon complete arbitrary proofs from its own mind.
Edit: One edit will be quicker than two replies and I tend to be "posting too fast". The main point is captured well by [0], but they immediately link to [1], the central result, which links to [2], an important corollary. At this point, I'm just going to quote WP:
> The first [Gödel] incompleteness theorem states that no consistent system of axioms whose theorems can be listed by an effective procedure (i.e., an algorithm) is capable of proving all truths about the arithmetic of natural numbers. For any such consistent formal system, there will always be statements about natural numbers that are true, but that are unprovable within the system.
> The second [Gödel] incompleteness theorem, an extension of the first, shows that the system cannot demonstrate its own consistency.
> Informally, [Tarski's Undefinability] theorem states that arithmetical truth cannot be defined in arithmetic.
I recognize that these statements may seem surprising, but they are provable and you should convince yourself of them. I recently reviewed [3] and found it to be a very precise and complete introduction to all of the relevant ideas; there's also GEB if you want something more fun.
[0] https://en.wikipedia.org/wiki/Automated_theorem_proving#Deci...
[1] https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...
[2] https://en.wikipedia.org/wiki/Tarski%27s_undefinability_theo...
[3] https://www.logicmatters.net/resources/pdfs/godelbook/GodelB...
As someone unfamiliar with the results you describe, why can’t a machine do better than a human? A smart human is better at writing proofs than a dumb one. If we had a highly advanced general artificial intelligence, why couldn’t it generate better results than the smart humans?
Or is automated proof search impossible for humans as well?
Arguably, humans require more energy per operation. So, presumably such an argument hinges upon what types of operations are performed in conducting automated proof search?
What would "automated proof search" even mean in the context of being performed by a human being?
The context here is that certain problems cannot be solved by an algorithm, which doesn't necessarily translate into "cannot be performed by a machine".
It only means that there cannot be any Turing machine capable of solving them, no matter what "operations" it represents.
The task (in terms of constructor theory) is: Find the functions that sufficiently approximate the observations and record their reproducible derivations.
Either the (unreferenced) study was actually arguing that "automated proof search" can't be done at all, or that human neural computation is categorically non-algorothmic.
Grid search of all combinations of bits that correspond to [symbolic] classical or quantum models.
Or better: evolutionary algorithms and/or neural nets.
Roger Penrose argues that human neural computation is indeed non-algorithmic in nature (see his book "The Emperor's New Mind"; 1989) and speculates that quantum processes are involved.
I don't quite understand the role of neural nets in that context, though. Those just classical computations in the end and should be bound by the same limits that every other algorithms are, shouldn't they?
That human cognition is quantum in nature - that e.g. entanglement is necessary - may be unfalsifiable.
Neuromorphic engineering has expanded since the 1980s. https://en.wikipedia.org/wiki/Neuromorphic_engineering
Quantum computing is the best known method for simulating chemical reactions and thereby possibly also neurochemical reactions. But, Is quantum computing necessary to functionally emulate human cognition?
It may be that a different computation medium can accomplish the same tasks without emulating all of the complexity of the brain.
If the brain is only classical and some people are using their brains to perform quantum computations, there may be something there.
Quantum cognition: https://en.wikipedia.org/wiki/Quantum_cognition
Quantum memristors are still elusive.
From "Quantum Memristors in Frequency-Entangled Optical Fields" (2020) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7079656/ :
> Apart from the advantages of using these devices for computation [12] (such as energy efficiency [13], compared to transistor-based computers), memristors can be also used in machine learning schemes [14,15]. The relevance of the memristor lies in its ubiquitous presence in models which describe natural processes, especially those involving biological systems. For example, memristors inherently describe voltage-dependent ion-channel conductances in the axon membrane in neurons, present in the Hodgkin–Huxley model [16,17].
> Due to the inherent linearity of quantum mechanics, it is not straightforward to describe a dissipative non-linear memory element, such as the memristor, in the quantum realm, since nonlinearities usually lead to the violation of fundamental quantum principles, such as no-cloning theorem. Nonetheless, the challenge was already constructively addressed in Ref. [18]. This consists of a harmonic oscillator coupled to a dissipative environment, where the coupling is changed based on the results of a weak measurement scheme with classical feedback. As a result of the development of quantum platforms in recent years, and their improvement in controllability and scalability, different constructions of a quantum memristor in such platforms have been presented. There is a proposal for implementing it in superconducting circuits [7], exploiting memory effects that naturally arise in Josephson junctions. The second proposal is based on integrated photonics [19]: a Mach–Zehnder interferometer can behave as a beam splitter with a tunable reflectivity by introducing a phase in one of the beams, which can be manipulated to study the system as a quantum memristor subject to different quantum state inputs.
Quantum harmonic oscillators have also found application in modeling financial markets. Quantum harmonic oscillator: https://en.wikipedia.org/wiki/Quantum_harmonic_oscillator
New framework for natural capital approach to transform policy decisions
Natural capital: https://en.wikipedia.org/wiki/Natural_capital
> Natural capital is the world's stock of natural resources, which includes geology, soils, air, water and all living organisms. Some natural capital assets provide people with free goods and services, often called ecosystem services. Two of these (clean water and fertile soil) underpin our economy and society, and thus make human life possible.
Natural capital accounting: https://en.wikipedia.org/wiki/Natural_capital_accounting
> Natural capital accounting is the process of calculating the total stocks and flows of natural resources and services in a given ecosystem or region.[1] Accounting for such goods may occur in physical or monetary terms. This process can subsequently inform government, corporate and consumer decision making as each relates to the use or consumption of natural resources and land, and sustainable behaviour.
Opportunity cost: https://en.wikipedia.org/wiki/Opportunity_cost
> When an option is chosen from alternatives, the opportunity cost is the "cost" incurred by not enjoying the benefit associated with the best alternative choice.[1] The New Oxford American Dictionary defines it as "the loss of potential gain from other alternatives when one alternative is chosen."[2] In simple terms, opportunity cost is the benefit not received as a result of not selecting the next best option. Opportunity cost is a key concept in economics, and has been described as expressing "the basic relationship between scarcity and choice". [3] The notion of opportunity cost plays a crucial part in attempts to ensure that scarce resources are used efficiently.[4] Opportunity costs are not restricted to monetary or financial costs: the real cost of output forgone, lost time, pleasure or any other benefit that provides utility should also be considered an opportunity cost. The opportunity cost of a product or service is the revenue that could be earned by its alternative use.
How do we value essential dependencies in terms of future opportunity costs?
In terms of just mental health?
"National parks a boost to mental health worth trillions: study" https://phys.org/news/2019-11-national-boost-mental-health-w...
> Visits to national parks around the world may result in improved mental health valued at about $US6 trillion (5.4 trillion euros), according to a team of ecologists, psychologists and economists
> Professor Bateman's decision-making framework focuses on the links between the environment and economy and has three components: efficiency, assessing which option generates the greatest benefit; sustainability, the effects of each option on natural capital stocks; and equity, regarding who receives the benefits of a decision and when.
Ian J. Bateman et al. "The natural capital framework for sustainably efficient and equitable decision making", Nature Sustainability (2020). DOI: 10.1038/s41893-020-0552-3 https://www.nature.com/articles/s41893-020-0552-3
Challenge to scientists: does your ten-year-old code still run?
"Ten Simple Rules for Reproducible Computational Research" http://www.ploscompbiol.org/article/info%3Adoi%2F10.1371%2Fj... :
> Rule 1: For Every Result, Keep Track of How It Was Produced
> Rule 2: Avoid Manual Data Manipulation Steps
> Rule 3: Archive the Exact Versions of All External Programs Used
> Rule 4: Version Control All Custom Scripts
> Rule 5: Record All Intermediate Results, When Possible in Standardized Formats
> Rule 6: For Analyses That Include Randomness, Note Underlying Random Seeds
> Rule 7: Always Store Raw Data behind Plots
> Rule 8: Generate Hierarchical Analysis Output, Allowing Layers of Increasing Detail to Be Inspected
> Rule 9: Connect Textual Statements to Underlying Results
> Rule 10: Provide Public Access to Scripts, Runs, and Results
... You can get a free DOI for and archive a tag of a Git repo with FigShare or Zenodo.
... re: [Conda and] Docker container images https://news.ycombinator.com/item?id=24226604 :
> - repo2docker (and thus BinderHub) can build an up-to-date container from requirements.txt, environment.yml, install.R, postBuild and any of the other dependency specification formats supported by REES: Reproducible Execution Environment Standard; which may be helpful as Docker Hub images will soon be deleted if they're not retrieved at least once every 6 months (possibly with a GitHub Actions cron task)
BinderHub builds a container with the specified versions of software and installs a current version of Jupyter Notebook with repo2docker, and then launches an instance of that container in a cloud.
“Ten Simple Rules for Creating a Good Data Management Plan” http://journals.plos.org/ploscompbiol/article?id=10.1371/jou... :
> Rule 6: Present a Sound Data Storage and Preservation Strategy
> Rule 8: Describe How the Data Will Be Disseminated
... DVC: https://github.com/iterative/dvc
> Data Version Control or DVC is an open-source tool for data science and machine learning projects. Key features:
> - Simple command line Git-like experience. Does not require installing and maintaining any databases. Does not depend on any proprietary online services. Management and versioning of datasets and machine learning models. Data is saved in S3, Google cloud, Azure, Alibaba cloud, SSH server, HDFS, or even local HDD RAID.
> - Makes projects reproducible and shareable; helping to answer questions about how a model was built.
There are a number of great solutions for storing and sharing datasets.
... "#LinkedReproducibility"
Open textual formats for data and open source application and system software (more precisely, FLOSS), are just as important.
Imagine that x86 - and with it, the PC platform - gets replaced by ARM within a decade. For binary software, this would be a kind of geological extinction event.
The likelihood of there being a [security] bug discovered in a given software project over any significant period of time is near 100%.
It's definitely a good idea to archive source and binaries and later confirm that the output hasn't changed with and without upgrading the kernel, build userspace, execution userspace, and PUT/SUT Package/Software Under Test.
- Specify which versions of which constituent software libraries are utilized. (And hope that a package repository continues to serve those versions of those packages indefinitely). Examples: Software dependency specification formats like requirements.txt, environment.yml, install.R
- Mirror and archive all dependencies and sign the collection. Examples: {z3c.pypimirror, eggbasket, bandersnatch, devpi as a transparent proxy cache}, apt-cacher-ng, pulp, squid as a transparent proxy cache
- Produce a signed archive which includes all requisite software. (And host that download on a server such that data integrity can be verified with cryptographic checksums and/or signatures.) Examples: Docker image, statically-linked binaries, GPG-signed tarball of a virtualenv (which can be made into a proper package with e.g. fpm), ZIP + GPG signature of a directory which includes all dependencies
- Archive (1) the data, (2) the source code of all libraries, and (3) the compiled binary packages, and (4) the compiler and build userspace, and (5) the execution userspace, and (6) the kernel. Examples: Docker can solve for 1-5, but not 6. A VM (virtual machine) can solve for 1-5. OVF (Open Virtualization Format) is an open spec for virtual machine images, which can be built with a tool like Vagrant or Packer (optionally in conjunction with a configuration management tool like Puppet, Salt, Ansible).
When the application requires (7) a multi-node distributed system configuration, something like docker-compose/vagrant/terraform and/or a configuration management tool are pretty much necessary to ensure that it will be possible to reproducibly confirm the experiment output at a different point in spacetime.
A deep dive into the official Docker image for Python
One thing I've learned is that rather than use a docker-entrypoint.sh, most Linux software can be ran just using the `--user 1000:1000` or whatever UID/GID you want to use, as long as you map a volume that can use those permissions. It is a lot cleaner this way.
> Why Tini?
> Using Tini has several benefits:
> - It protects you from software that accidentally creates zombie processes, which can (over time!) starve your entire system for PIDs (and make it unusable).
> - It ensures that the default signal handlers work for the software you run in your Docker image. For example, with Tini, SIGTERM properly terminates your process even if you didn't explicitly install a signal handler for it.
> - It does so completely transparently! Docker images that work without Tini will work with Tini without any changes.
[...]
> NOTE: If you are using Docker 1.13 or greater, Tini is included in Docker itself. This includes all versions of Docker CE. To enable Tini, just pass the `--init` flag to docker run.
https://github.com/krallin/tini#why-tini
I didn't know it was included by default. I'll check it out, thanks!
There are Alpine [1] and Debian [2] miniconda images (within which you can `conda install python==3.8` and 2.7 and 3.4 in different conda envs)
[1] https://github.com/ContinuumIO/docker-images/blob/master/min...
[2] https://github.com/ContinuumIO/docker-images/blob/master/min...
If you build manylinux wheels with auditwheel [3], they should install without needing compilation for {CentOS, Debian, Ubuntu, and Alpine}; though standard Alpine images have MUSL instead of glibc by default, this [4] may work:
echo "manylinux1_compatible = True" > $PYTHON_PATH/_manylinux.py
[4] https://github.com/docker-library/docs/issues/904#issuecomme...
The miniforge docker images aren't yet [5][6] multi-arch, which means it's not as easy to take advantage of all of the ARM64 / aarch64 packages that conda-forge builds now.
[5] https://github.com/conda-forge/docker-images/issues/102#issu...
[6] https://github.com/conda-forge/miniforge/issues/20
There are i686 and x86-64 docker containers for building manylinux wheels that work with many distros: https://github.com/pypa/manylinux/tree/master/docker
A multi-stage Dockerfile build can produce a wheel in the first stage and install that wheel (with `COPY --from=0`) in a later stage; leaving build dependencies out of the production environment for security and performance: https://docs.docker.com/develop/develop-images/multistage-bu...
Interesting! I use miniconda extensively for local development to manage virtual environments for different python versions and love it. I hardly ever actually use the conda packages though.
I assume the main benefit of using these images would be if you are installing from conda repos instead of pip? Otherwise just using the official python images would be as good if not better
Edit: I guess if you needed multiple python versions in a single container this would be a good solution for that as well
Use cases for conda or conda+pip:
- Already-compiled packages (where there may not be binary wheels) instead of requiring reinstallation and subsequent removal of e.g. build-essentials for every install
- Support for R, Julia, NodeJS, Qt, ROS, CUDA, MKL, etc.
- Here's what the Kaggle docker-python Dockerfile installs with conda and with pip: https://github.com/Kaggle/docker-python/blob/master/Dockerfi...
- Build matrix in one container with conda envs
Disadvantages of the official python images as compared with conda+pip:
- Necessary to (re)install build dependencies and a compiler for every build (if there's not a bdist or a wheel for the given architecture) and then uninstall all unnecessary transitive dependencies. This is where a [multi-stage] build of a manylinux wheel may be the best approach.
- No LSM (AppArmor, SELinux, ) for one or more processes in the container (which may have read access to /etc or environment variables and/or --privileged)
- Necessary to build basically everything on non x86[-64] architectures for every container build
Disadvantages of conda / conda+pip:
- Different package repo infrastructure to mirror
- Users complaining that they don't need conda who then proceed to re-download and re-build wheels locally multiple times a day
Additional attributes for comparison:
- The new pip solver (which is slower than the traditional iterative non-solver), conda, and mamba
- repo2docker (and thus BinderHub) can build an up-to-date container from requirements.txt, environment.yml, install.R, postBuild and any of the other dependency specification formats supported by REES: Reproducible Environment Execution Standard; which may be helpful as Docker Hub images will soon be deleted if they're not retrieved at least once every 6 months (possibly with a GitHub Actions cron task)
Quite a few conda packages have patches added by the conda team to help fix problems in packages relying on native code or binaries. Particularly on Windows. If something is available on the primary conda repos it will almost assuredly work with few of any problems cross-platform, whereas pip is hit or miss.
If you’re always on Linux you may never appreciate it but some pip packages are a nightmare to get working properly on Windows.
If you look through the source of the conda repos, you’ll see all kinds of small patches to fix weird and breaking edge cases, particularly in libs with significant C back ends.
Here's the meta.yml for the conda-forge/python-feedstock: https://github.com/conda-forge/python-feedstock/blob/master/...
It includes patches just like distro packages often do.
The Consortium for Python Data API Standards
This actually strikes me as a huuuge waste of effort. I work with every one of the different technologies they mention every day.
The differences and idiosyncrasies are truly, truly not a big deal and really what you want is to allow different library maintainers to do it all differently and just build your own adapters over top of them.
This allows library developers to worry about narrow use cases and have their own separate processes to introduce breaking changes and features that deal with extreme specifics like GPU or TPU interop, heterogeneous distributed backed arrays, jagged arrays, etc. etc.
Let a thousand flowers blossom and a thousand different Python array and record APIs contend.
End users can write their own adapters to mollify irregularities between them, possibly writing different adapters on a case by case basis.
If any “standard” adapters gain popularity as open source projects, great - but don’t try to bake that in from an RFC point of view into the array & data structure libraries themselves. Let them be free / whatever they want to be. That diversity of API approach is super valuable and easily mollified by your own custom adapter logic.
Personally, I like how the Julia ecosystem coalesced around a Tables.jl package that gives some consistency to tabular data packages that choose to implement it. Row-oriented or column-oriented, doesn't matter! Using Tables.jl they can be interchangeable.
There’s a huge difference between “I personally like it when some libraries choose to adhere to a convention” vs “let’s bake this into every tool with a wide, bureaucratic shared RFC process.”
But I don't think people are forced into following right? There's still choice. But if I was developing something in Python, I would rather join than not join.
I’m saying as a consumer of these libraries, I don’t want joiners, I want lots of different APIs that make different trade-offs for each use case, and I will write (or use) adapters if I need to (that’s my responsibility as a consumer).
Just because I personally either do or do not get some value out of a shared API standard doesn’t mean you do or do not, so personal feelings of what one likes need to be set aside, so the service boundary is at the level of consumers choosing their own adapters and making their own trade-offs.
If lots of libraries sign up, we all lose. Much slower compliance-constrained development cycles, less freedom for libraries to break the shared API contract in ways many users would be super happy to abide for the sake of a faster / better feature. Instead of making these trade-offs library by library, case by case, it’s made for you with standardization compliance as the highest constraint, regardless of the user base for which that does / doesn’t work well.
This strikes me as a bit pessimistic. I think you are mostly against _bad_ standards than standards per se.
If the consortium does its job well it will produce a good standard that all the major libraries like: from a UX as well as a performance viewpoint.
If it produces a bad standard, all that will happen is that nobody will sign up. No libraries are going to be _forced_ to do anything.
That’s fair except I’d say even good standards can cause a big slowdown in feature releases, and not everyone values the adherence to the standard as much as the features. It still seems better to just spin out the standard into a set of optional adapters and let library maintainers and users pick their own trade-offs in terms of when to adopt breaking changes, how / whether to smooth over idiosyncrasies across multiple libraries.
Even a good standard has costs and this particular case does not seem like it has good arguments in favor of a wide standard, but many arguments against it.
No, it's easy for library maintainers to offer a compat API in addition to however else they feel they need to differentiate and optimize the interfaces for array operations. People can contribute such APIs directly to libraries once instead of creating many conditionals in every library-utilizing project or requiring yet another dependency on an adapter / facade package that's not kept in sync with the libraries it abstracts.
If a library chooses to implement a spec compatability API, they do that once (optimally, as compared with somebody's hackish adapter facade which has very little comprehension of each library's internals) and everyone else's code doesn't need to have conditionals.
Each of L libraries implements a compat API: O(L)
Each of U library utilizers implements conditionals for every N places arrays are utilized: O(U x N_)
Each of U library utilizers uses the common denominator compat API: O(U)
L < U < (L + U) < (U x N_)
Tech giants let the Web's metadata schemas and infrastructure languish
It's "langushing" and they should do it for us? It's flourishing and they're doing it for us and they have lots of open issues and I want more for free without any work.
Wow! Nobody else does anything to collaboratively, inclusively develop schema and the problem is that search engines aren't just doing it for us?
1) Search engines do not owe us anything. They are not obligated to dominate us or the schema that we may voluntarily decide to include on our pages.
We've paid them nothing. They have no contract for service or agreement with us which compels them to please us or contribute greater resources to an open standard that hundreds of people are contributing to.
2) You people don't know anything about linked data and structured data.
Here's a list of schema: https://lov.linkeddata.es/dataset/lov/ .
Here's the Linked Open Data Cloud: https://lod-cloud.net/
Does your or this publisher's domain include any linked data?
Does this article include any linked data?
Do data quality issues pervade promising, comparatively-expensive, redundant approaches to natural-language comprehension, reasoning, and summarization?
Here, in contributing this example PR adding RDFa to the codeforantarctica web page, I probably made a mistake. https://github.com/CodeForAntarctica/codeforantarctica.githu... . Can you spot the mistake?
There should have been review.
https://schema.org/ClaimReview, W3C Verifiable Claims / Credentials, ld-signatures, and lds-merkleproof2017.
Which brings us to reification, truth values, property graphs, and the new RDF* and SPARQL* and JSON-LD* (which don't yet have repos with ongoing issues to tend to).
3) Get to work. This article does nothing to teach people how to contribute to slow, collaborative schema standards work.
Here's the link to the GitHub Issues so that you can contribute to schema.org: https://github.com/schemaorg/schemaorg
...
"Standards should be better and they should pay for it"
Who are the major contributors to the (W3C) open standard in question?
Is telling them to put up more money or step down going to result in getting what we want? Why or why not?
Who would merge PRs and close issues?
Have you misunderstood the scope of the project? What do the editors of the schema feel in regards to more specific domain vocabularies? Is it feasible or even advisable to attempt to out-schema domain experts who know how to develop and revise an ontology or even just a vocabulary with Protegé?
To give you a sense of how much work goes into creating a few classes and properties defined with RDFS in RDFa in HTML: here's the https://schema.org/Course , https://schema.org/CourseInstance , and https://schema.org/EducationEvent issue: https://github.com/schemaorg/schemaorg/issues/195
Can you find the link to the Use Cases wiki (which was the real work)? What strategy did you use to find it?
...
"Well, Google just does what's good for Google."
Are you arguing that Google.org should make charitable contributions to this project? Is that an advisable or effective way to influence a W3C open standard (where conflicts of interest by people just donating time are disclosed)?
Anyone can use something like extruct or OSDS to extract RDFa, Microdata, and/or JSON-LD from a page.
Everyone can include structured data and linked data in their pages.
There are surveys quantifying how many people have included which types in their pages. Some of that data is included on schema.org types pages.
...
Some written interview questions:
> Which issues have you contributed to? Which issues have you seen all the way to closed? Have you contributed a pull request to the project? Have you published linked data? What is the URL to the docs which explain how to contribute resources? How would you improve them?
https://twitter.com/westurner/status/1291903926007209984
...
After all that's happened here, I think Dan (who built FOAF, which all profitable companies could use instead of https://schema.org/Person ) deserves a week off to add more linked data to the internet now please.
I think that might be fair, but when she makes org came out it pitched itself as trust us we will take care of things, so yeah they don’t owe us anything but track record matters for trust in future ventures by these search engine orgs
schemaorg/schemaorg/CONTRIBUTING.md https://github.com/schemaorg/schemaorg/blob/main/CONTRIBUTIN... explains how you and your organization can contribute resources to the Schema.org W3C project.
If you or your organization can justify contributing one or more people at full or part time due to ROI or goodwill, by all means start sending Pull Requests and/or commenting on Issues.
"Give us more for free or step down". Wow. What PRs have you contributed to justify such demands?
https://schema.org/docs/documents.html links to the releases.
Time-reversal of an unknown quantum state
T-symmetry https://en.wikipedia.org/wiki/T-symmetry > See also links to "reversible computing" but not the "time reversal" disambiguation page?
The "teeter-totter" thought experiment on that page is interesting to me in that it seems to illustrate how the future has more possibilities than the past. But it also occurs to me that the future possibilities are fractal with the past - ie. the toy could fall onto one of many other lower pedestals, teetering as it was before. In this way the "many arbitrary" possibilities could be perceived as anything but.
Electric cooker an easy, efficient way to sanitize N95 masks, study finds
An autoclave is what the electrical cooker is being used to emulate. A pressure cooker is a better option if you'll be on the road, camping, or otherwise without electricity. Researchers in Canada already proved several months ago that N95 masks could be sanitized this way:
Unfortunately the referenced NewsArticle does not link to the ScholarlyArticle https://schema.org/ScholarlyArticle :
"N95 Mask Decontamination using Standard Hospital Sterilization Technologies" (2020-04) https://www.medrxiv.org/content/10.1101/2020.04.05.20049346v... :
> We sought to test the ability of 4 different decontamination methods including autoclave treatment, ethylene oxide gassing, ionized hydrogen peroxide fogging and vaporized hydrogen peroxide exposure to decontaminate 4 different N95 masks of experimental contamination with SARS-CoV-2 or vesicular stomatitis virus as a surrogate. In addition, we sought to determine whether masks would tolerate repeated cycles of decontamination while maintaining structural and functional integrity. We found that one cycle of treatment with all modalities was effective in decontamination and was associated with no structural or functional deterioration. Vaporized hydrogen peroxide treatment was tolerated to at least 5 cycles by masks. Most notably, standard autoclave treatment was associated with no loss of structural or functional integrity to a minimum of 10 cycles for the 3 pleated mask models. The molded N95 mask however tolerated only 1 cycle. This last finding may be of particular use to institutions globally due to the virtually universal accessibility of autoclaves in health care settings.
The ScholarlyArticle referenced by and linked to by the OP NewsArticle is "Dry Heat as a Decontamination Method for N95 Respirator Reuse" (2020-07) https://pubs.acs.org/doi/full/10.1021/acs.estlett.0c00534 . Said article does not reference "N95 Mask Decontamination using Standard Hospital Sterilization Technologies" DOI: 10.1101/2020.04.05.20049346v2 . We would do well to record that (article A, seemsToConfirm, Article B) as third-party linked data (only if both articles do specifically test the efficacy of the given sterilization method with the COVID-19 coronavirus)
None of this is necessary for non-healthcare workers, since the virus will die off on a fabric mask in a few hours.
1) Not a single study has demonstrated that viable Sars-Cov-2 virus survives on porous materials in the real world
2) Even before Covid, it was known for decades (common medical knowledge) that human coronaviruses and flu viruses do not remain viable on porous materials for more than several hours.
That this common knowledge is not so common knowledge among the public is a failure of public health communication. The one or two alarmist studies showing that the virus "survives" X number of days don't reflect the real world because 1) the researchers literally directly douse or soak the surface with a huge viral load, and 2) the researchers usually only look for viral genetic material, not whether the virus can infect cells. Viability != detecting viral genetic material (same story for people - we shed viral genetic material long after we stop being contagious).
There is a reason the CDC has always said surface transmission is rare: https://www.nytimes.com/2020/05/22/health/cdc-coronavirus-to... there have been no documented cases of surface transmission: https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-si... Fortunately some in the media are catching on to the hygiene (security) theater: https://www.theatlantic.com/ideas/archive/2020/07/scourge-hy...
Citations for 1) and 2):
Most studies use unrealistic starting doses: https://www.thelancet.com/pdfs/journals/laninf/PIIS1473-3099...
Other human coronaviruses don't survive long on porous surfaces (gone by 6 hours): https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7134510/. A hospital randomly collected patient and real surface swabs (including non-porous surfaces) for original SARS, a minority were PCR positive, none of the swabs were infectious (viral culture): https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7134510/. Same story for flu (virus can be detected for days, but inactive and not viable after a few hours): https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3222642/.
I could only find this study on Sars-Cov-2 that cultured the virus (still used a huge viral dose in a lab setting, not the real world). Even though they were able to culture the virus, only 1% of virus remains after 6 hours on a surgical mask, several orders of magnitude less for cotton clothing: https://www.medrxiv.org/content/10.1101/2020.05.07.20094805v...
This long post was originally meant to be a reply to someone asking for a citation; it's yet another example of how much effort is required to combat misinformation.
"Interim Recommendations for U.S. Households with Suspected or Confirmed Coronavirus Disease 2019 (COVID-19)" https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-si... :
> On the other hand, transmission of novel coronavirus to persons from surfaces contaminated with the virus has not been documented. Recent studies indicate that people who are infected but do not have symptoms likely also play a role in the spread of COVID-19. Transmission of coronavirus occurs much more commonly through respiratory droplets than through objects and surfaces, like doorknobs, countertops, keyboards, toys, etc. Current evidence suggests that SARS-CoV-2 may remain viable for hours to days on surfaces made from a variety of materials. Cleaning of visibly dirty surfaces followed by disinfection is a best practice measure for prevention of COVID-19 and other viral respiratory illnesses in households and community settings
Fed announces details of new interbank service to support instant payments
If anyone from the FED is reading this I have this advice: don't use ISO20022! (I've been working on the brazilian instant payments project, known as PIX). It's an overly-complicated-try-to-solve-everything standard that promises interoperability but only delivers pain and misery. Since each country has its own standards for person and account identification, in dealing with these local realities, the ISO20022 adds quite a lot of complexity. But all this cost does not result in interoperability. In fact, each payment scheme end up with a localized version of the standard, incompatible with others. What could have been a clean API, ends up a mess.
What alternative do you suggest? Keeping in mind all the parties that will be involved, besides your project.
Create a fit to purpose, simpler, protocol using a good interface specification language (OpenAPI or protocol buffers). Maybe borrow concepts and patterns from the ISO standard, but not adopt it. It will not deliver its promise of interoperability. It might actually make it harder to interoperate with other payment schemes.
Interledger Protocol (ILP, ILPv4).
Interledger Architecture:
https://interledger.org/rfcs/0001-interledger-architecture/#... :
> For purposes of Interledger, we call all settlement systems ledgers. These can include banks, blockchains, peer-to-peer payment schemes, automated clearing house (ACH), mobile money institutions, central-bank operated real-time gross settlement (RTGS) systems, and even more.
[...]
> Interledger provides for secure payments across multiple assets on different ledgers. The architecture consists of a conceptual model for interledger payments, a mechanism for securing payments, and a suite of protocols that implement this design.
> The Interledger Protocol (ILP) is the core of the Interledger protocol suite. Colloquially, the whole Interledger stack is sometimes referred to as "ILP". Technically, however, the Interledger Protocol is only one layer in the stack.
> Interledger is not a blockchain, a token, nor a central service. Interledger is a standard way of bridging financial systems. The Interledger architecture is heavily inspired by the Internet architecture described in RFC 1122, RFC 1123 and RFC 1009.
[...]
> You can envision the Interledger as a graph where the points are individual nodes and the edges are accounts between two parties. Parties with only one account can send or receive through the party on the other side of that account. Parties with two or more accounts are connectors, who can facilitate payments to or from anyone they're connected to.
> Connectors [AKA routers] provide a service of forwarding packets and relaying money, and they take on some risk when they do so. In exchange, connectors can charge fees and derive a profit from these services. In the open network of the Interledger, connectors are expected to compete among one another to offer the best balance of speed, reliability, coverage, and cost.
ILP > Peering, Clearing and Settling: https://interledger.org/rfcs/0032-peering-clearing-settlemen...
ILP > Simple Payment Setup Protocol (SPSP): https://interledger.org/rfcs/0009-simple-payment-setup-proto...
> This document describes the Simple Payment Setup Protocol (SPSP), a basic protocol for exchanging payment information between payee and payer to facilitate payment over Interledger. SPSP uses the STREAM transport protocol for condition generation and data encoding.
> (Introduction > Motivation) STREAM does not specify how payment details, such as the ILP address or shared secret, should be exchanged between the counterparties. SPSP is a minimal protocol that uses HTTPS for communicating these details.
[...]
GET /.well-known/pay HTTP/1.1
Host: example.com
Accept: application/spsp4+json, application/spsp+jsonShrinking deep learning’s carbon footprint
"Unlearning" is one algorithmic approach that may yield substantial energy consumption gains.
With many deep learning models, it's not possible to determine when or from what source something was learned: it's not possible to "back out" a change to the network and so the whole model has to be re-trained from scratch; which is O(n) instead of O(1.x).
The article covers software approaches (more energy-efficient algorithms) and mentions GPUs but not TPUs or ASICs.
Specialized chips (built with dynamic fabrication capacities) are far more energy efficient for specific types of workloads. We see this with mining ASICs, SSL accelerators, and also with Tensor Processing Units (for deep learning).
The externalities of energy production are the ultimate concern. If you're using cheap, clean energy with minimized external costs ("sustainable energy"), the energy-efficiency of the algorithm and the chips is of much less concern.
Could we recognize products, services, and data centers that were produced with and/or run on directly sourced clean energy as "200% Green"; with a logo on the box and/or the footer? 100% offset by PPAs is certainly progress.
Show HN: Starboard – Fully in-browser literate notebooks like Jupyter Notebook
Hi HN, I developed Starboard over the past months.
Cell-by-cell notebooks like Jupyter are great for prototyping, explaining and exploration, but their dependence on a Python server (with often undocumented dependencies) limits their ability to be shared and remixed. Now that browsers support dynamic imports, it has become possible to create a similar workflow entirely in the browser.
That motivated me to build Starboard Notebook, a tool I wished existed. It's:
* Run entirely in the browser, there is no server or setup, it's all static files.
* Web-native, so no widget system is necessary. There is nearly no magic, it's all web tech (HTML, CSS, JS).
* Stores as a plaintext file, it will play nicely with version control systems.
* Hackable: the sandbox that your code gets run in contains the editor itself, so you can metaprogram the editor itself (e.g. adding support for other languages such as Python through WASM).
* Open source (https://github.com/gzuidhof/starboard-notebook).
You can import any code that targets the browser directly (e.g. puts stuff on the window object), or that has exports in ES module format.
I'm happy to answer any questions!
Neat! There's a project called Jyve that compiles Jupyter Lab to WASM (using iodide). https://github.com/deathbeds/jyve There are kernels for JS, CoffeeScript, Brython, TypeScript, and P5. FWIU, the kernels are marked as unsafe because, unfortunately, there seems to be no good way to sandbox user-supplied notebook code from the application instance. The README describes some of the vulnerabilities that this entails.
The jyve project issues discuss various ideas for repacking Python packages beyond the set already included with Pyodide and supporting loading modules from remote sources.
https://developer.mozilla.org/en-US/docs/Web/Security/Subres... : "Subresource Integrity (SRI) is a security feature that enables browsers to verify that resources they fetch (for example, from a CDN) are delivered without unexpected manipulation. It works by allowing you to provide a cryptographic hash that a fetched resource must match."
There's a new Native Filesystem API: "The new Native File System API allows web apps to read or save changes directly to files and folders on the user's device." https://web.dev/native-file-system/
We'll need a way to grant specific URLs specific, limited amounts of storage.
https://github.com/iodide-project/pyodide :
> The Python scientific stack, compiled to WebAssembly
> [...] Pyodide brings the Python 3.8 runtime to the browser via WebAssembly, along with the Python scientific stack including NumPy, Pandas, Matplotlib, parts of SciPy, and NetworkX. The packages directory lists over 35 packages which are currently available.
> Pyodide provides transparent conversion of objects between Javascript and Python. When used inside a browser, Python has full access to the Web APIs.
https://github.com/deathbeds/jyve/issues/46 :
> Would miniforge and conda-forge build a WASM architecture target?
> Emscripten or WASI?
Ask HN: Learning about distributed systems?
I used to love Operating Systems during my undergrads, Modern Operating Systems by Tanenbaum is till date the only academic book I've read entirely. I recently read an article about how Amazon built Aurora by Werner Vogels and I was captivated by it. I want to start reading about Distributed Systems. What would be a good start/Road Map?
The book “Designing Data-Intensive Applications” by Martin Kleppman is a fantastic read with such a concise train of thought. It builds up from basics, adds another thing, and another thing.
I kept asking myself, what would happen if I were to extend on the feature currently presented in the chapter I was reading, only to find out my answers in the next chapter.
Brilliant book
> "Designing Data-Intensive Applications” by Martin Kleppman: https://dataintensive.net/ https://g.co/kgs/xJ73FS
From a previous question re: "Ask HN: CS papers for software architecture and design?" (https://news.ycombinator.com/item?id=15778396 and distributed systems we eventually realize were needed in the first place:
> Bulk Synchronous Parallel: https://en.wikipedia.org/wiki/Bulk_synchronous_parallel .
Many/most (?) distributed systems can be described in terms of BSP primitives.
> Paxos: https://en.wikipedia.org/wiki/Paxos_(computer_science) .
> Raft: https://en.wikipedia.org/wiki/Raft_(computer_science) #Safety
> CAP theorem: https://en.wikipedia.org/wiki/CAP_theorem .
Papers-we-love > Distributed Systems: https://github.com/papers-we-love/papers-we-love/tree/master...
awesome-distributed-systems also has many links to theory: https://github.com/theanalyst/awesome-distributed-systems
- Byzantine fault: https://en.wikipedia.org/wiki/Byzantine_fault :
> A [Byzantine fault] is a condition of a computer system, particularly distributed computing systems, where components may fail and there is imperfect information on whether a component has failed. The term takes its name from an allegory, the "Byzantine Generals Problem",[2] developed to describe a situation in which, in order to avoid catastrophic failure of the system, the system's actors must agree on a concerted strategy, but some of these actors are unreliable.
awesome-bigdata lists a number of tools: https://github.com/onurakpolat/awesome-bigdata
Practically, dask.distributed (joblib -> SLURM,), dask ML, dask-labextension (a JupyterLab extension for dask), and the Rapids.ai tools (e.g. cuDF) scale from one to many nodes.
Not without a sense of irony, as the lists above list many papers that could be readings with quizzes,
Distributed systems -> Distributed computing: https://en.wikipedia.org/wiki/Distributed_computing
Category: Distributed computing: https://en.wikipedia.org/wiki/Category:Distributed_computing
Category:Distributed_computing_architecture : https://en.wikipedia.org/wiki/Category:Distributed_computing...
DLT: Distributed Ledger Technology: https://en.wikipedia.org/wiki/Distributed_ledger
Consensus (computer science) https://en.wikipedia.org/wiki/Consensus_(computer_science)
Ask HN: How can I “work-out” critical thinking skills as I age?
As I get older, I realized I’m not as sharp as I used to be. Maybe it’s from the fatigue of juggling 2 kids, but I’m very ill prepared for interviews because I simply can’t answer “product questions” and brain teasers. It’s a skill I need, and truthfully I was never good at consultant type questions to begin with but I’m seeing a lot of these questions in Data Science interviews.
Any help or resources will be tremendously appreciated.
Problem solving: https://en.wikipedia.org/wiki/Problem_solving
Critical thinking: https://en.wikipedia.org/wiki/Critical_thinking
Computational Thinking: https://en.wikipedia.org/wiki/Computational_thinking
> 1. Problem formulation (abstraction);
> 2. Solution expression (automation);
> 3. Solution execution and evaluation (analyses).
Interviewers may be more interested in demonstrating problem solving methods and f thinking aloud than an actual solution in an anxiety-producing scenario.
https://en.wikipedia.org/wiki/Brilliant_(website) ;
> Brilliant offers guided problem-solving based courses in math, science, and engineering, based on National Science Foundation research supporting active learning.[14]
Coding Interview University: https://github.com/jwasham/coding-interview-university
Programmer Competency Matrix: https://github.com/hltbra/programmer-competency-checklist
Inference > See also: https://en.wikipedia.org/wiki/Inference
- Deductive reasoning: https://en.wikipedia.org/wiki/Deductive_reasoning
- Inductive reasoning: https://en.wikipedia.org/wiki/Inductive_reasoning
> This is the [open] textbook for the Foundations of Data Science class at UC Berkeley: "Computational and Inferential Thinking: The Foundations of Data Science" http://inferentialthinking.com/
The tragedy of FireWire: Collaborative tech torpedoed by corporations
Due to DMA (Direct Memory Access) in most implementations, IEEE 1394 ("FireWire") can be used to directly read from and write to RAM.
See: IEEE 1394 > Security issues https://en.wikipedia.org/wiki/IEEE_1394#Security_issues
FWIU, USB 3 is faster than FireWire; there are standard, interchangeable USB connectors and adapters; and USB implementations do not use DMA. https://en.wikipedia.org/wiki/USB_3.0
> FWIU, USB 3 is faster than FireWire
You've missed the point. The title of the article reads "The tragedy of FireWire" - how a good standard was victimized due to corporate politics. FireWire was 10 years ahead of USB. In 2002, FireWire 800 provided a data rate of 800 Mbps, full-duplex. On the other hand, USB 2 only had 480 Mbps, half-duplex. USB didn't catch up until 2010, after USB 3.0 was released. FireWire uses 30 V, 1.5 A power, and it supports power delivery up to 45 watts. USB didn't offer anything similar, not even the latest USB 3, until 2014 after Power Delivery and Type-C's standardization - and real applications only started to ramp up by ~2017. Both limitations of USB 2 were detrimental to some applications, e.g. using hard drives on a PC was painful due to speed and power constraints, had FireWire became more common, it would not be an issue. FireWire can also do everything Ethernet can do (in fact, 1394b's signaling is better than 100Base-TX Ethernet, and has lower radiation), networking is natively supported, you can network two computers together directly with a 1394 cable! On USB, it's only possible using an external controller. One application was NAS, a hard drive can be shared directly over 1394, and mounted as a local drive. Not possible with USB. Finally, 1394b does not have USB's 5-meter limitation due to protocol timing, and natively supports signaling over CAT-5 and fiber optics cable for a distance of ~50-100 meters - on USB, it requires a media converter running vendor-specific custom protocols and not interoperable. All of these features are standardized in 2004 in IEEE 1394b.
Of course, limitations are not inherently USB 2's failures, they have different applications, FireWire was more expensive, and USB's goal was low-cost. But the tragedy was that bad business decision prevented FireWire to achieve its full potential, slowly fading out, and eventually became irrelevant after USB 3 despite its initial good engineering.
FireWire was a tragedy.
> Due to DMA (Direct Memory Access) in most implementations, IEEE 1394 ("FireWire") can be used to directly read from and write to RAM.
This is why we need IOMMU.
You should not blame IEEE 1394 for DMA attacks just because it supports DMA. I agree that the tragedy of FireWire probably prevented a common PC exploit vector as an lucky unintentional consequence, but it's just a symptom, not the actual issue - and the symptom has came back in another form.
Most sufficiently fast hardware interfaces support DMA, and they're equally vulnerable - this includes PCI-E, ExpressCard (which exposes PCI-E), USB 4 (which exposes PCI-E port), Thunderbolt (which exposes PCI-E). Practical exploits are already widespread, not limited to arbitrary memory reads/writes, but also OptionROM arbitrary code execution if a Thunderbolt device is plugged in at boot time.
And this is not only a problem for external ports like 1394, USB 4 or Thunderbolt. Threats also exist from internal devices, like a Ethernet or Wi-Fi controller on the motherboard. While you cannot initiate a DMA transaction via Ethernet (without RDMA) or Wi-Fi since they don't expose PCI-E, the fact that those are connected directly to PCI-E implies any vulnerability in the controller firmware potentially allows an attacker to launch a DMA attack. Exploits already exist for Wi-Fi controllers.
This is the real problem. While the mechanism of IOMMU exists, CPU manufacturers and operating systems previously did little to systematically protect the system from DMA. In the past, Intel intentionally removed IOMMU functionality from low-end CPUs and motherboard chipsets in order to sell more Xeon CPUs to enterprise users. Also, while IOMMU is supported by operating systems, kernel code and drivers were not systematic audited for security - serious IOMMU bypass vulnerabilities have been previously discovered (and patched) in macOS and Linux's IOMMU implementations.
With continued proliferation of USB 4 and Thunderbolt, the lack of IOMMU and buggy driver implementation will become a bigger problem.
So far, QubesOS is most most protected operating system from DMA attacks. Untrusted hardware are assigned to a dedicated VM, and IOMMU is enforced by the hypervisor, not the individual device drivers in operating systems. Compromising a device driver in an untrusted domain does not directly expose the rest of QubesOS to attackers.
> and USB implementations do not use DMA.
USB 4 does now.
So your argument is that not security but cost is the reason that USB "won" the external device interface competition with FireWire?
Good to know that USB4 implementations are making the same mistake as FireWire implementors did in choosing performance over security . Unfortunately it looks like there will be no alternative except for maybe to use a USB3 hub (or an OS with fuzzed IOMMU and also controller firmwares)?
Could an NX bit for data coming from buses with and without DMA help at all?
Hot gluing external ports now seems a bit more rational and justified for systems where physical access is less controlled.
> So your argument is that not security, but cost is the reason that USB "won" the external device interface competition with FireWire?
Sorry, but I heavily suspect that you didn't read the original article and I suggest you to do so.
Security has nothing to do with 1394's market adoption, in the 2010s, some desktops still have 1394, and some laptops still have ExpressCard, and you don't see anyone complains about security, only some security researchers.
The reason was partially cost, but also a bad business decision by Steve Jobs, which made Intel and Microsoft stopped their efforts for pushing 1394, largely damaged it before it even got a chance for wider adoption.
> FireWire's principal creator, Apple, nearly killed it before it could appear in a single device. And eventually the Cupertino company effectively did kill FireWire, just as it seemed poised to dominate the industry.
> Intel sent its CTO to talk to Jobs about the change, but the meeting went badly. Intel decided to withdraw its support for FireWire—to pull the plug on efforts to build FireWire into its chipsets—and instead throw its weight behind USB 2.0.
> Sirkin believes that Microsoft could have reversed the new licensing policy by citing the prior signed agreement. "Microsoft must have thrown it away," he speculated, because it would have "stopped Apple in its tracks."*
--
> Good to know that USB4 implementations are making the same mistake as FireWire implementors did in choosing performance over security.
DMA is optional and can be disabled.
Speaking of usability, DMA is limited to the Thunderbolt part (which is ultimately the PCI-E part) of USB 4 standard. You don't have to use USB 4's PCI-E. Unless the USB device really needs PCI-E, like RAID storage box or a GPU, disabling DMA is not a problem. Requiring explicit user consent for PCI-E is also a solution, operating systems already does it for Thunderbolt devices. Same treatment can be applied to USB 4.
And again, calling out DMA support is only shooting the messenger while ignoring the underlying problem. DMA is essential for any high-performance system bus, and it can a problem regardless of who supported it, or whether it's connected to an external port. A Wi-Fi controller handles untrusted data, but despite being an internal device on the motherboard, it's still potentially an exploit vector for DMA attackers.
> (or an OS with fuzzed IOMMU and also controller firmwares)
Only IOMMU is needed (and possibly a good driver, but not strictly required, see QubesOS). If your IOMMU is good, you don't have to trust firmware or hardware, since arbitrary DMAs are blocked by IOMMU, just like how memory protection (MMU) works but for I/O. Memory spaces are isolated.
The IOMMU vulnerabilities I previous mentioned are already patched, and a complete bypass of IOMMU is unlikely to occur in the future. Driver bugs are still an issue, more RAM regions can be exposed by drivers than what's really needed, and potentially you can pretend to be any hardware and feed bad data to trick the most vulnerable driver to expose more RAM regions, and I expect to see more bugs.
Turn on IOMMU, and apply OS patches often, and you'll probably be fine.
I read much of the article (which assumed that "FireWire" failed because of issues with suppliers failing to work together instead of waning demand (due in part to corporate customers' knowledge of the security risks of most implementations)).
Thanks for the info on USB-4, DMA, IOMMU.
IOMMU: https://en.wikipedia.org/wiki/Input%E2%80%93output_memory_ma...
Looks like there are a number of iommu Linux kernel parameters: https://www.kernel.org/doc/html/latest/admin-guide/kernel-pa...
Wonder what the defaults are and what the comparable parameters are for common consumer OSes.
Looks like NX bit support is optional in IOMMUs.
Can I configure the amount of RAM allocated to this?
> Wonder what the defaults
See the Thunderclap FAQ, most of your questions are explained. Thunderclap is the DMA attack on Thunderbolt, and everything should be applicable to USB 4 as well, since USB 4 incorporated Thunderbolt directly.
From the FAQ,
> In macOS 10.12.4 and later, Apple addressed the specific network card vulnerability we used to achieve a root shell. However the general scope of our work still applies; in particular that Thunderbolt devices have access to all network traffic and sometimes keystrokes and framebuffer data.
> Microsoft have enabled support for the IOMMU for Thunderbolt devices in Windows 10 version 1803, which shipped in 2018. Earlier hardware upgraded to 1803 requires a firmware update from the vendor. This brings them into line with the baseline for our work, however the more complex vulnerabilities we describe remain relevant.
> Recently, Intel have contributed patches to version 5.0 of the Linux kernel (shortly to be released) that enable the IOMMU for Thunderbolt and prevent the protection-bypass vulnerability that uses the ATS feature of PCI Express.
But note that Microsoft says on its website,
> This feature does not protect against DMA attacks via 1394/FireWire, PCMCIA, CardBus, ExpressCard, and so on.
Basically, IOMMU is enabled by default in most systems now. However, only on Thunderbolt devices (and USB 4 devices, since Thunderbolt is now its subset). But not on other PCI-E ports, such as the internal PCI-E ports, or external ones like 1394 or ExpressCard. I think it's due to fears of performance and compatibility issues, there's no reason why it cannot be implemented if one insists. Also, from the macOS example, you see that bad drivers can be a problem, even with IOMMU, if a driver is bad, attackers can exploit them and trick them to give more RAM access (If my memory is correct, Linux's implementation was better, but since it's an early FAQ, I don't know if macOS has improved).
> what the comparable parameters are for common consumer OSes.
On the Linux kernel, with an Intel CPU, the kernel option "intel_iommu=on" will enable IOMMU completely for everything, internal and external, which is the most ideal option. You should see
DMAR: IOMMU enabled
IOMMU needs support from the CPU and motherboard chipset, Intel's IOMMU is called VT-D and marketed it as I/O virtualization for virtual machines. Most i3, i5, and i7 CPUs are supported. You may need to turn it on in BIOS if there's an option. It should work on most laptops, even an old Ivy Bridge laptop should work - which is good news, laptops are the most vulnerable platform. Unfortunately, on desktops, Intel intentionally removed IOMMU from high-end CPUs preferred by PC enthusiasts in the 1st (Nehalem), 2nd (Sandy Bridge), 3rd (Ivy Bridge), and 4th (Haswell) generations, possibly as a strategy to sell more Xeon chips to enterprise virtualization users (likely in line with their cripple-ECC strategy). Most high-end overclockable K-series CPUs have their IOMMU removed, while the non-K counterparts are supported. Also, in these generations, some motherboard chipsets are crippled and don't have IOMMU. Fortunately, Intel no longer does it since 5/6th gen (Broadwell/Skylake) CPUs.
No experience on AMD yet, but it should be similar. I think possibly better, my impression is that AMD doesn't intentional crippling IOMMU or ECC like Intel does (a nasty strategy...). Check the docs.
The Developer’s Guide to Audit Logs / SIEM
This article suggests that there should be separate data collection systems for: analytics, SIEM logs, and performance metrics.
The article mentions the CEF (Common Event Format) standard but not syslog or GELF or other JSON formats.
[ArcSight] Common Event Format [PDF]: https://kc.mcafee.com/resources/sites/MCAFEE/content/live/CO...
GELF: Graylog Extended Log Format: https://docs.graylog.org/en/latest/pages/gelf.html
Wikipedia > Syslog lists a few limitations of Syslog (no message delivery confirmation, though there is a reliable delivery RFC; and insufficient payload standardization) and also links to the existing Syslog RFCs. https://en.wikipedia.org/wiki/Syslog
Are push-style systems ideal for security logshipping systems? What sort of a message broker is ideal? AMQP has reliable delivery; while, for example, ZeroMQ does not and will drop messages due to resource exhaustion.
Developers simply need an API for their particular framework to non-blockingly queue and then log structs to a remote server. This typically means moving beyond a single-threaded application architecture so that the singular main [green] thread is not blocked when the remote log server is not responding.
SIEM: Security information and event management: https://en.wikipedia.org/wiki/Security_information_and_event...
Del.icio.us
The Firefox (and Chromium) bookmarks storage and sync systems still don't persist tags!
"Allow reading and writing bookmark tags" https://bugzilla.mozilla.org/show_bug.cgi?id=1225916
Notes re: how this could be standardized with JSON-LD: https://bugzilla.mozilla.org/show_bug.cgi?id=1225916#c116
The existing Web Experiment for persisting bookmark tags: https://github.com/azappella/webextension-experiment-tags/bl...
Ask HN: Recommendations for Books on Writing?
I want to propose a book club for writing as an engineer. Writing is fundamentally and critically important, but it seems that we don't emphasize it as much as we should for engineers (outside Amazon, where apparently it is a prominent member of the leadership pantheon).
I'm interested in any suggestions that HN has for great books on writing as an engineer! Accessibility and ease are important factors for a book club as well.
Technical Writing: https://en.wikipedia.org/wiki/Technical_writing
Google Technical Writing courses (1 & 2) and resources: https://developers.google.com/tech-writing :
- Google developer documentation style guide: https://developers.google.com/style
- Microsoft Writing Style Guide: https://docs.microsoft.com/en-us/style-guide/welcome/
Season of Docs is a program where applicants write documentation for open source projects: https://developers.google.com/season-of-docs/
Many open source projects are happy to accept necessary contributions of docs and editing; but do keep in mind that maintaining narrative documentation can be far more burdensome than maintaining API documentation that's kept next to the actual code. Systems like doxygen, epidoc, javadoc, and sphinx-apidoc enable developers to generate API documentation for a particular version of the software project as one or more HTML pages.
ReadTheDocs builds documentation from ReStructuredText and now also Markdown sources using Sphinx and the ReadTheDocs Docker image. ReadTheDocs organizes docs with URLs of the form <projectname>.rtfd.io/<language>/<version|latest>: https://docs.readthedocs.io/en/latest/ . The ReadTheDocs URL scheme reduces the prevalence of broken external links to documentation; though authors are indeed free to delete and rename docs pages and change which VCS tags are archived with RTD.
Write the Docs is a conference for technical documentation authors which is supported in part by ReadTheDocs: https://www.writethedocs.org/
Write the Docs > Learning Resources > All our videos and articles: https://www.writethedocs.org/topics/ :
> This page links to the topics that have been covered by conference talks or in the newsletter.
You might say that UX (User Experience) includes UI design and marketing: the objective is to imagine yourself as a customer experiencing the product or service afresh.
Writing dialogue is an activity we often associate more with creative writing exercises; where the objective is to meditate upon compassion for others.
One must imagine themself as ones/people/persons who interact with the team.
Cognitive walkthrough: https://en.wikipedia.org/wiki/Cognitive_walkthrough
The William Golding, Jung, and Joseph Campbell books on screenwriting, archetypes, and the hero's journey monomyth are excellent; if you're looking for creative writing resources.
Ask HN: How did you learn x86-64 assembly?
I'm an experienced C/C++ programmer and I occasionally look at the generated assembly to check for optimizations, loop unrolling, vectorization, etc. I understand what's going on the surface level, but I have a hard time understand what's going on in detail, especially with high optimization levels, where the compiler would do all kinds of clever tricks. I experiment with code in godbolt.org and look up the various opcodes, but I would like to take a more structured way of learning x86-64 assembly, especially when it comes to common patterns, tips and tricks, etc.
Are there any good books or tutorials you can recommend which go beyond the very beginner level?
High Level Assembly (HLA) https://en.wikipedia.org/wiki/High_Level_Assembly
> HLA was originally conceived as a tool to teach assembly language programming at the college-university level. The goal is to leverage students' existing programming knowledge when learning assembly language to get them up to speed as fast as possible. Most students taking an assembly language programming course have already been introduced to high-level control flow structures, such as IF, WHILE, FOR, etc. HLA allows students to immediately apply that programming knowledge to assembly language coding early in their course, allowing them to master other prerequisite subjects in assembly before learning how to code low-level forms of these control structures. The book The Art of Assembly Language Programming by Randall Hyde uses HLA for this purpose
Web: https://plantation-productions.com/Webster/
Book: "The Art of Assembly Language Programming" https://plantation-productions.com/Webster/www.artofasm.com/
Portable, Opensource, IA-32, Standard Library: https://sourceforge.net/projects/hla-stdlib/
"12.4 Programming in C/C++ and HLA" in the Linux 32 bit edition: https://plantation-productions.com/Webster/www.artofasm.com/...
... A chapter(s) about wider registers, WASM, and LLVM bitcode etc might be useful?
... Many awesome lists link to OllyDbg and other great resources for ASM; like such as ghidra: https://www.google.com/search?q=ollydbg+site%3Agithub.com+in...
Brain connectivity levels are equal in all mammals, including humans: study
This is outrageously misleading. To make a claim about the number of synapses between neurons based on MRI data is completely unwarranted. Voxel size (single volumetric pixel) in MRI is approximately 1mm, while synapse size is way less than a micron. You need resolution per pixel on the order of 10s of nanometers to identify synapses.
I wouldn’t be surprised if a dead salmon also has “equal” connectivity: https://www.wired.com/2009/09/fmrisalmon/
Surely there's something to learn here though. I haven't read the original paper but a quantity that's preserved across brain scales is either an artifact or a neat insight.
Your criticism reads like someone accusing economists of being outrageously misleading when they don't sample individual households but measure macro indicators. It's like saying Ramon y cajal was ridiculous because he couldn't image the neuropil effectively. Or like saying early optogenetics experiments were ridiculous because who knows if you're stimulating a neuron in a realistic manner?
And in any case, it's true that synapses are comically small relative to voxel size, but we also have some reasonable information about projection patterns and synapse number from various tracer or rabies studies with which you are no doubt familiar.
I haven't read the nature paper the press release is about and I'm not a huge fan of many d/fMRI practices or derived claims. And I've worked with enough mammalian dwi data to be skeptical of specific connection claims. But this strikes me as a rather interesting result even if you can't measure all the synapses at the right resolution: either the tractography method has connectivity conservation artifacts baked in, or there's something interesting going on.
"fNIRS Compared with other neuroimaging techniques" https://en.wikipedia.org/wiki/Functional_near-infrared_spect...
> When comparing and contrasting these devices it is important to look at the temporal resolution, spatial resolution, and the degree of immobility.
I think I missed your overall point?
OP suggests that the spatial resolution of existing MRI neuroimaging capabilities is insufficient to observe or so characterize or so generalize about neuronal activity in mammalian species. fNIRS (functional near-infrared spectroscopy) is one alternative neuroimaging capability that we could compare fMRI with according to the criteria for comparison suggested in the cited Wikipedia article: "temporal resolution, spatial resolution, and the degree of immobility".
Ask HN: Resources to start learning about quantum computing?
Hi there,
I'm an experienced software engineer (+15 years dev experience, MsC in Computer Science) and quantum computing is the first thing in my experience that is being hard to grasp/understand. I'd love to fix that ;)
What resources would you recommend to start learning about quantum computing?
Ideally resources that touch both the theoretical base and evolve to more practical usages.
"What are some good resources to learn about Quantum Computing?" https://news.ycombinator.com/item?id=16052193 https://westurner.github.io/hnlog/#comment-16052193
Launch HN: Charityvest (YC S20) – Employee charitable funds and gift matching
Stephen, Jon, and Ashby here, the co-founders of Charityvest (https://charityvest.org). We created a modern, simple, and affordable way for companies to include charitable giving in their suite of employee benefits.
We give employees their own tax-deductible charitable giving fund, like an “HSA for Charity.” They can make contributions into their fund and, from their fund, support any of the 1.4M charities in the US, all on one tax receipt.
Using the funds, we enable companies to operate gift matching programs that run on autopilot. Each donation to a charity from an employee is matched automatically by the company in our system.
A company can set up a matching gift program and launch giving funds to employees in about 10 minutes of work.
Historically, corporate charitable giving matching programs have been administratively painful to operate. Making payments to charities, maintaining tax records, and doing due diligence on charitable compliance is taxing on HR / finance teams. The necessary software to help has historically been quite expensive and not very useful for employees beyond the matching features.
This is one example of an observation Stephen made after working for years as a philanthropic consultant. Consumer fintech products aren’t built to make great giving experiences for donors. Instead, they are built for buyers — e.g., nonprofits (fundraising) or corporations (gift matching) — without a ton of consideration for the everyday user experience.
A few years back, my wife and I made a commitment to give a portion of our income away every year, and we found it administratively painful to give regularly. The tech that nonprofits typically use hardly inspires generosity — e.g., high fees, poor user flows, and questionable information flow (like tax receipts). Giving platforms try to compensate for poor functionality with bright pictures of happy kids in developing countries, but when the technology is not a good financial experience it puts a damper on things.
Charityvest started when I noticed a particular opportunity with donor-advised funds, which are tax-deductible giving funds recognized by the IRS. They are growing quickly (20% CAGR), but mainly among the high-net worth demographic. We believe they are powerful tools. They enable donors to have a giving portfolio all from one place (on one tax receipt) and have full control over their payment information/frequency, etc. Most of all, they enable a donor to split the decisions of committing to give and supporting a specific organization. Excitement about each of these decisions often strikes at different times for donors—particularly those who desire to give on a budget.
We believe everyone should have their own charitable giving fund no matter their net worth. We’ve created technology that has democratized donor-advised funds.
We also believe good technology should be available for every company, big and small. Employers can offer Charityvest for $2.49 / employee / month subscription, and we charge no fees on any of the giving — charities receive 100% of the money given.
Lastly, we send the program administrator a fun report every month to let them know all the awesome giving their company and its employees did in one dashboard. This info can be leveraged for internal culture or external brand building.
We’re just launching our workplace giving product, but we’ve already built a good portfolio of trusted customers, including Eric Ries’ (author of The Lean Startup) company, LTSE. We’ve particularly seen a number of companies use us as a meaningful part of their corporate decision to join the fight for racial justice in substantive ways.
Our endgame is that the world becomes more generous, starting with the culture of every company. We believe giving is fundamentally good and we want to build technology that encourages more of it by making it more simple and accessible.
You can check out our workplace giving product at (https://charityvest.org/workplace-giving). If you’re interested, we can get your company up and running in 10 minutes. Or, please feel free to forward us on to your HR leadership at your company.
Our giving funds are also available for free for any individual on https://charityvest.org — without gift matching and reporting. We’d invite you to check out the experience. For individuals, we make gifts of cash and stock to any charity fee-free.
Happy to share this with you all, and we’d love to know what you think.
What a great idea!
Are there two separate donations or does it add the company's name after the donor's name? Some way to notify recipients about the low cost of managing a charitable donation match program with your service would be great.
Have you encountered any charitable foundations which prefer to receive cryptoassets? Red Cross and UNICEF accept cryptocurrency donations for the children, for example.
Do you have integration with other onboarding and HR/benefits tools on your roadmap? As a potential employee, I would like to work for a place that matches charitable donations, so mentioning as much in job descriptions would be helpful.
Thanks! Our matching system issues an identical grant from the fund of the matching company. It goes out in the same grant cycle as the employee grant so they go together.
We haven't yet encountered any charity that prefers to receive cryptoassets.
We have lots of dreams about thoughtful integrations with HR software, but we want the experience to be excellent, and we want to keep the experience of our existing app excellent. We'll be balancing those priorities as we grow.
> Our matching system issues an identical grant from the fund of the matching company. It goes out in the same grant cycle as the employee grant so they go together.
So the system creates a separate transaction for the original and the matched donation with each donor's name on the respective gift?
How do users sync which elements of their HR information with your service? IDK what the monthly admin cost there is.
There are a few HR, benefits, contracts, and payroll YC companies with privacy regulation compliance and APIs https://www.ycombinator.com/companies/?query=Payroll
https://founderkit.com/people-and-recruiting/health-insuranc...
Separate data records, yes, separate payment, no. The charity receives one consolidated check in the mail each month across all donors (one payment), and the original donor's grant will be on the data sheet as well as the matching grant from the company's corporate fund (separate records).
Today, users create their accounts directly with our app, and they are affiliated with their corporation in our app via their email address. So we don't integrate with any HR information.
Administrators of the program can add and remove employees via copying and pasting email addresses (can add/remove many at a time). We aim to integrate with HR systems of record in the future to make this seamless.
Thanks for clarifying.
Do you offer a CSV containing donor information to the charity?
Do you support anonymous matched donations?
Can donors specify that a donation is strongly recommended for a specific effort?
...
3% * $1000/yr == $2.50/mo * 12mo
CSV: yes if they request we’ll happily provide.
Anonymous: yes it’s an option on our grant screen
Specific efforts: yes on the grant screen we enable donors to add a “specific need” they’d like their grant to fund.
:-)
It may be helpful to integrate with charity evaluation services to help donors assess various opportunities to give.
Charity Navigator > Evaluation method https://en.wikipedia.org/wiki/Charity_Navigator#Evaluation_m...
We love the idea of layering in rich information to help donors make informed decisions! We just want to be really thoughtful about the experience. So we plan to get there, it just may take us some time.
We Need a Yelp for Doctoral Programs
How are the data needs for such a doctoral and post-doctoral evaluation program different from the data needs for https://collegescorecard.ed.gov ?
Data: https://collegescorecard.ed.gov/data/
Data documentation: https://collegescorecard.ed.gov/data/documentation/
Reviews seem like the biggest need.
“The culture in the X department is terrible, everyone works 80 hours a week and is miserable”
“Night life is great at least, the grad students usually go out for student nights at blah blah after seminar”
“Underrated X program because at university Y, don’t be fooled we literally have 4 of the 5 top researchers in field Z”
Etc.
But it is so hard for people to give reviews. People seem to give good review when are they are very pissed off or very happy. Later seems to be a very rare case.
I have always thought that bad reviews are sufficient for all purposes, and the only reason to even have good reviews is so the reviewees don't feel overly persecuted.
Any product or service will have occasional customers who are either unreasonable or have bad luck. If you have a lot of bad reviews, then as a prospective customer you want to see if all of them fall under the two categories or whether there is a pattern that is relevant to you.
It seems to me that a review database works fine even when all the reviews are bad, and the only regulation it really needs is an effort to prevent one single individual with a vendetta from overwhelming it.
All of the World’s Money and Markets in One Visualization
> Derivatives top the list, estimated at $1 quadrillion or more in notional value according to a variety of unofficial sources.
1 Quadrillion: 1,000,000,000,000,000 (10^15)
Derivative (finance) https://en.wikipedia.org/wiki/Derivative_(finance)
Derivatives market https://en.wikipedia.org/wiki/Derivatives_market :
> The market can be divided into two, that for exchange-traded derivatives and that for over-the-counter derivatives.
Why companies lose their best innovators (2019)
https://news.ycombinator.com/item?id=23886158
> Three reasons companies lose their best innovators.
> 1. They fail to recognize and support the innovators
> 2. Innovation becomes a herculean task
> 3. Corporations don’t match rewards with outcomes
While the paragraphs under point 2 do discuss risk and the paragraphs under point 3 do discuss rewards, I'm not sure this article belongs here.
Risk and Reward.
Large corporations are able to pay people by doing things at scale; with sufficient margin at sufficient volume to justify continued investment. Risk is minimized by focusing on ROI.
Startups assume lots of risk and lots of debt and most don't make it. Liquidation preference applies as the startup team adjourns (and maybe open-sources what remains). In a large corporation, that burnt capital is reported to the board (which represents the shareholders) who aren't "gambling" per se. "You win some and you lose some" is across the street; and they don't have foosball and snacks.
How can large organizations (nonprofit, for-profit, governmental) foster intrapreneurial mindsets without just continuing to say "innovation" more times and expecting things to happen? Drink. "Innovators welcome!". Drink water.
"Intrapreneurial." What does that even mean? The employee, within their specialized department, spends resources (time, money, equipment) on something that their superior managers have not allocated funding for because they want: (a) recognition; (b) job security; (c) to save resources such as time and money; (d) to work on something else instead of this wasteful process; (e) more money.
Very few organizations have anything like "20% time". Why was 20% time thrown off the island to a new island where they had room to run? Do they have foosball? Or is the work so fun that they don't even need foosball? Or is it worth long days and nights because the potential return is enough money to retire tomorrow and then work on what?
Step 1. Steal innovators to work on our one thing
Step 2.
Step 3. Profit.
20% Project: https://en.wikipedia.org/wiki/20%25_Project
Intrapreneurship: https://en.wikipedia.org/wiki/Intrapreneurship
Internal entrepreneur: https://en.wikipedia.org/wiki/Internal_entrepreneur
CINO: Chief Innovation Officer / CTIO: Chief Technology Innovation Officer https://en.wikipedia.org/wiki/Chief_innovation_officer
... Is acquiring innovation and bringing it to scale a top-down process? How do we capture creative solutions and then allocate willing and available resources to making that happen?
awesome-ideation-tools: https://github.com/zazaalaza/awesome-ideation-tools
Powerful AI Can Now Be Trained on a Single Computer
Lots of people are focusing on this being done on a particularly powerful workstation, but the computer described seems to have power at a similar order of magnitude to the many servers which would be clustered together in a more traditional large ML computation. Either those industrial research departments could massively cut costs/increase output by just “magically keeping things in ram,” or these researchers have actually found a way to reduce the computational power that is necessary.
I find the efforts of modern academics to do ML research on relatively underpowered hardware by being more clever about it to be reminiscent of soviet researchers who, lacking anything like the access to computation of their American counterparts, were forced to be much more thorough and clever in their analysis of problems in the hope of making them tractable.
If anything it seems to me that doing the most work under constraint of resources is precisely what intelligence is about. I've always wondered why the consumption of compute resources is itself not treated as a significant part of the 'reward' in ML tasks.
At least if you're taking inspiration from biological systems, it clearly is part of the equation, a really important one even.
Ask HN: Something like Khan Academy but full curriculum for grade schoolers?
Khan Academy continually gets held up as a great resource for online courses across the age spectrum for math related subjects. With the continuing pandemic continuing to grow in the US and schools not really sure how to handle things, the GF and I are looking into other options.
Is there a recommended resource that gives unbiased (as possible) reviews for middle school (7-8th grade) curriculum? Searching these days really doesn't bring up quality, just options one has to comb through.
K12 CS Framework (ACM, Code.org, [...]) https://k12cs.org/
Computing Curricula 2020 (ACM, IEEE,) http://www.cc2020.net/
Official SAT Practice (College Board, Khan Academy) https://www.khanacademy.org/sat
http://wrdrd.github.io/docs/consulting/software-development#... (TODO: add link to cc2020 draft)
Programmer Competency Matrix: http://sijinjoseph.com/programmer-competency-matrix/ , https://competency-checklist.appspot.com/ , https://github.com/hltbra/programmer-competency-checklist
Re: Computational thinking https://westurner.github.io/hnlog/#comment-15454421
Coding Interview University: https://github.com/jwasham/coding-interview-university
AutoML-Zero: Evolving Code That Learns
They use evolutionary search [0] to create programs for image classification, specifically 'binary classification tasks extracted from CIFAR-10'. They do it from scratch, though they use a pytorch-ish programming language with differentiation baked in.
Looks like a nice summer intern project.
"AutoML-Zero: Evolving Machine Learning Algorithms From Scratch" (2020) https://arxiv.org/abs/2003.03384 https://scholar.google.com/scholar?cluster=11748751662887361...
How does this compare to MOSES (OpenCog/asmoses) or PLN? https://github.com/opencog/asmoses https://scholar.google.com/scholar?hl=en&as_sdt=0%2C43&q=%22... (2007)
SymPy - a Python library for symbolic mathematics
SymPy is the best!
In addition to the already-excellent official tutorial https://docs.sympy.org/latest/tutorial/index.html , I have written a short printable tutorial summary of the functions most useful for students https://minireference.com/static/tutorials/sympy_tutorial.pd...
For anyone interested in trying SymPy without installing, there is also the online shell https://live.sympy.org/ which works great and allows you to send entire computations as a URL, see for example this comment that links to an important linear algebra calculation: https://news.ycombinator.com/item?id=23158095
How does it compare to Matlab's symbolic toolbox, aside from not being affordable outside of academia?
NumPy for Matlab users: https://numpy.org/doc/stable/user/numpy-for-matlab-users.htm...
SymPy vs Matlab: https://github.com/sympy/sympy/wiki/SymPy-vs.-Matlab
If you then or later need to do distributed ML, it is advantageous to be working in Python. Dask Distributed, Dask-ML, RAPIDS.ai (CuDF), PyArrow, xeus-cling
Good to know: sympy does not use numpy so you can use pypy instead of python.
In my case I got a 7x speedup.
You can however use sympy to convert symbolic expressions to numeric numpy functions, using the "lambdify" feature. It's awesome because it lets you symbolically generate and manipulate a numerical program.
SymEngine https://github.com/symengine/symengine
> SymEngine is a standalone fast C++ symbolic manipulation library. Optional thin wrappers allow usage of the library from other languages, e.g.:
> [...] Python wrappers allow easy usage from Python and integration with SymPy and Sage (the symengine.py repository)
https://en.wikipedia.org/wiki/SymPy > Related Projects:
> SymEngine: a rewriting of SymPy's core in C++, in order to increase its performance. Work is currently in progress to make SymEngine the underlying engine of Sage too
How does it compare to sage math? I used sage math today and was pleasantly surprised. Even has a latex option. If you name your variables correctly, it even outputs Greek symbols and subscript correctly!
User 29athrowaway mentions in another comment that they use Sympy through Sagemath, so they probably could answer your question best. It sounds like Sympy is a subset of what Sage offers, but I'm not familiar enough with either tool to know.
Ask HN: Are there any messaging apps supporting Markdown?
I'd like to easily send formatted code, and bullet points, etc. through a messaging app without having to resort to a heavy app like Slack.
Mattermost supports CommonMark Markdown: https://docs.mattermost.com/help/messaging/formatting-text.h...
Zulip supports ~CommonMark Markdown: https://zulip.readthedocs.io/en/latest/subsystems/markdown.h...
Reddit supports Markdown. https://www.reddit.com/wiki/markdown
Discourse now supports CommonMark Markdown.
GitHub, BitBucket, GitLab and Gogs/Gitea support Markdown.
I wouldn't consider Discourse/Reddit/Github/etc. to be a messaging app per se, even if some of those have messaging functionality between users...
I digress on the category definition. Public messaging (without PM or DM features) is still messaging; and often far more useful than trying to forward 1:1 messages in order to bring additional participants onboard.
It's worth noting that GH/BB/GL have all foregone PM features; probably for the better in terms of productivity: messaging @all is likely more productive.
What vertical farming and ag startups don't understand about agriculture
My father is an ag soil chemist of 50+ years.
I'm an industrial systems eng. w/ a specialty in polymer-textile-fiber engineering. (Mostly useless skillsets in the US now)
Gonna share a few lessons here about agriculture that I try to convey to EECS, econ, Neuroscience, and the web developer crowd.
- You can only grow non-calorically dense foods in vertical farms
- It takes 10-14 kwh/1000 gallons of water to desalinate. More if it gets periodically polluted at an increasing rate.
- Large majority Agrarian populations exist because the countries are stuck in a purgatory of <1 MWh/capita annum whereby the country doesn't have scaleable nitrogen and steel manufacturing.
- Sweet potatoes and sweet potatoes are some of the highest satiety lowest input to output ratio produce. High efficiency.
- In civilizations where you are at < 1MWh/capita annum - there is not enough electricity to produce tools for farming, steel for roads, and concrete for building things. The end result is that the optimal decision is to have more children to harvest more calories per an acre.
- Property, bankruptcy, and inheritance law have an immense influence on the farmer population of a country.
I remember telling some "ag tech" VCs my insights and offering to introduce my father who has an immense amount of insight on the topic from having grown things for as long as he has....My thoughts were tossed aside.
So you can't grow potatoes vertically? Can you elaborate? Is it a function of physiology, i.e. calorie dense vegetables need far more leaves and supporting stems than can be practically stacked vertically?
I imagine space is a factor, but energy will be a big one as well. Calorie dense foods will likely need more space and energy (light) inputs. Vertical farms are very water efficient, so I don't think that matters much.
Vertical farms make a lot more sense with fresh vegetables like leafy greens that grow quickly, command high prices if grown organically, and benefit from being closer to market.
Potatoes are the exact opposite. If it ever becomes more cost effective to grow corn, wheat, and potatoes in virtual farms then outdoor agriculture is dead. While I don't agree with the article that it will never happen, it might require energy advances like fusion power or drastically higher _rural_ land values and water prices.
Greenhouses make sense long before vertical farming, just look at agriculture in the Netherlands, it's mind boggling how much they produce for such a tiny country.
Can you expand on this?
I get that to store a calorie in a potato I need to supply a calorie of energy from somewhere else.
But why is fusion power required instead of better UV lamps in my vertical farm? (Assuming I had enough electricity to run them)
The total amount of electricity to power those UV lamps should be on par with what the Sun sends to the potatoes fields. Maybe that's the reason for fusion. It didn't do the math.
Actually no not really. Plants only absorb two wavelengths of light. It's currently more efficient to convert sun into solar power via panels and then to light LEDs supplying only the wavelengths that plants use. Despite the seeming inefficiency here, the fact is that plants are even more inefficient at absorbing light not at the right wavelengths than solar panels.
Could one imagine a material that would absorb solar spectrum and emit the preferred frequencies? Something like a polymer one could stretch over fields to get more from the suns rays.
>> "Actually no not really. Plants only absorb two wavelengths of light. It's currently more efficient to convert sun into solar power via panels and then to light LEDs supplying only the wavelengths that plants use. Despite the seeming inefficiency here, the fact is that plants are even more inefficient at absorbing light not at the right wavelengths than solar panels."
> Could one imagine a material that would absorb solar spectrum and emit the preferred frequencies? Something like a polymer one could stretch over fields to get more from the suns rays.
Would you call that a "solar transmitter"?
https://en.wikipedia.org/wiki/Transmitter :
> Generators of radio waves for heating or industrial purposes, such as microwave ovens or diathermy equipment, are not usually called transmitters, even though they often have similar circuits.
Would "absorption spectroscopy" specialists have insight into whether this is possible without solar cells, energy storage, and UV LEDs? https://en.wikipedia.org/wiki/Absorption_spectroscopy
(edit) The thermal energy from sunlight (from the FREE radiation from the nuclear reaction at the center of our solar system) is also useful to and necessary for plants. There's probably a passive heat pipe / solar panel cooling solution that could harvest such heat for colder seasons and climates.
Also, UV-C is useful for sanitizing (UVGI) but not really for plant growth. https://en.wikipedia.org/wiki/Ultraviolet_germicidal_irradia... :
> UVGI can be coupled with a filtration system to sanitize air and water.
Is that necessary or desirable for plants?
https://www.lumigrow.com/learning-center/blogs/the-definitiv... :
> The light that plants predominately use for photosynthesis ranges from 400–700 nm. This range is referred to as Photosynthetically Active Radiation (PAR) and includes red, blue and green wavebands. Photomorphogenesis occurs in a wider range from approximately 260–780 nm and includes UV and far-red radiation.
Photomorphogenesis: https://en.wikipedia.org/wiki/Photomorphogenesis
PAR: Photosynthetically active radiation: https://en.wikipedia.org/wiki/Photosynthetically_active_radi...
Grow light: https://en.wikipedia.org/wiki/Grow_light
Are there bioluminescent e.g. algae which emit PAR and/or UV? Algae can feed off of waste industrial gases.
Bioluminescence > Light production: https://en.wikipedia.org/wiki/Bioluminescence#Light_producti...
Biophoton: https://en.wikipedia.org/wiki/Biophoton
Chemiluminescence: https://en.wikipedia.org/wiki/Chemiluminescence
Electrochemiluminescence: https://en.wikipedia.org/wiki/Electrochemiluminescence
Quantum dot display / "QLED": https://en.wikipedia.org/wiki/Quantum_dot_display
Could be possible? Analyzing the inputs and outputs is useful in natural systems, as well.
Ask HN: What are your go to SaaS products for startups/MVPs?
Looking for some inspiration. Ive done a lot of MVPs/Early-stage apps over the years and I tend to lean on the same SaaS portfolio for mails, text gateways, payment etc, but Im sure Ive missed a few valuable additions.
Here's a few I use: Mails: Mailchimp / Mandrill Payment: Paylike Search: Algolia
https://StackShare.io and https://FounderKit.com are great places to find reviews of SaaS services:
> mails,
https://founderkit.com/growth-marketing/email-marketing/revi...
https://stackshare.io/email-marketing
https://zapier.com/learn/email-marketing/
> text gateways,
https://founderkit.com/apis/sms/reviews
https://stackshare.io/voice-and-sms
> payments
https://founderkit.com/apis/credit-card-processing/reviews
https://stackshare.io/payment-services
Both have categories:
https://stackshare.io/categories
https://founderkit.com/reviews
If you're looking for market/competition research, I'd recommend https://coscout.com as well, for info about companies, funding, competitors, tech stack etc
It's in alpha right now, so you'll have to get in the waitlist, though they're only a few weeks away from launching AFAIK.
Full disclosure: A friend of mine is building this :)
Long term viability of SaaS solutions is definitely worth researching.
Is this something that's going to get acquired and be extinguished?
What are our switching costs?
How do we get our data in a format that can be: read into our data warehouse/lake and imported into an alternate service if necessary in the future?
How does Coscout compare to e.g. Crunchbase, PitchBook (Morningstar), YCharts, AngelList?
Ask HN: Do you read aloud or silently in your minds?
Most times while reading a new topic that I am not familiar with, I tend to read aloud in my mind. Yet that changes based on the content and the way it is written.
When I'm focused, I notice that reading silently helps increase my reading speed and cognition, like everything is flowing in.
Other times I don't seem to understand anything if I'm not reading it aloud in my mind.
Has anyone noticed such a thing and if so can you share any tips or information you've learned about this behavior
Subvocalization https://en.wikipedia.org/wiki/Subvocalization
Speed reading https://en.wikipedia.org/wiki/Speed_reading
Ask HN: How do you deploy a Django app in 2020?
Hi. I'm a mid-level software engineer trying to deploy a small (2000 max users) django app to production.
If I Google: How to deploy a Django app. I get 10+ different answers.
Can anyone on HN help me please.
You'll probably get 10 different answers here too.
I use uwsgi in emperor mode. I have a Makefile that runs a few SSH commands on production to clone my git repo at a specific commit, builds a virtual environment from requirements.txt, and atomically swaps a symlink to a uwsgi socket. Nginx in front of it all. It's pretty much the Python equivalent of FTPing HTML files to a web server.
Uwsgi is terrifying because it has so many options (and I wonder how secure it is), but it has never failed me. Packaging and containerizing things sounds cool, but I just can't justify spending time on it when my setup works fine (I'm a solo dev).
If you have only one production server, dokku is "A docker-powered PaaS that helps you build and manage the lifecycle of applications." Dokku supports Heroku buildpack deployment (buildstep), Procfiles, Dockerfile deployment, Docker image deployment, git deployment (gitreceive), or tarfile deployments. https://github.com/dokku/dokku
There are a number of plugins for Dokku. Dokku ships with the nginx plugin as the HTTP frontend proxy. Dokku supports SSL certs with the certs plugin.
When you need to move to more than one server, what do you do? There's now a dokku-scheduler-kubernetes plugin which can do HA (high availability) which is worth reading about before you develop and document your own deployment workflow. https://github.com/dokku/dokku-scheduler-kubernetes
I also always put build, test, and deployment commands in a Makefile.
Package it; as a container or as containers that install a RPM/DEB/APK/Condapkg/Pythonpkg (possibly containing a zipapp). Zipapps are fast.
If you have any non-python dependencies, a Pythonpkg only solves for part of the packaging needs.
Producing a packaged artifact should be easy and part of your CI build script.
Here's the cookiecutter-django production docker-compose.yml with containers for django, celery, postgres, redis, and traefik as a load balancer: https://github.com/pydanny/cookiecutter-django/blob/master/%...
Cookiecutter-django also includes a Procfile.
With k8s, you have an ingress (~load balancer + SSL termination proxy) other than traefik.
You can generate k8s YML from docker-compose.yml with Kompose.
I just found this which describes using GitLab CI with Helm: https://davidmburke.com/2020/01/24/deploy-django-with-helm-t...
What is the command to scale up or down? Do you need a geodistributed setup (on multiple providers' clouds)? Who has those credentials and experience?
How do you do red/green or rolling deployments?
Can you run tests in a copy of production?
Can you deploy when the tests that run on git commit pass?
What runs the database migrations in production; while users are using the site?
If something deletes the whole production setup or the bus factor is 1, how long does it take to redeploy from zero; and how much manual work does it take?
CI + Ansible + Terraform + Kubernetes.
Whatever tools you settle on, django-eviron for a 12 Factor App may be advisable. https://github.com/joke2k/django-environ
The Twelve-Factor App: https://12factor.net/
Containers from first principles
I recently discovered systemd-nspawn and was amazed at how lightweight a basic container can be.
"Docker Without Docker" (2015) explains /sbin/init and systemd-nspawn. Systemd did not exist when docker was first created. https://chimeracoder.github.io/docker-without-docker/
That's not what I remember. Wikipedia backs up that recollection as well [0], marking systemd's initial release in 2010. Docker's initial release is listed as 2013 [1]. Maybe that was true of the dotCloud internal releases, and certainly, not all of the EL and other Linux distros had not adopted systemd during 2010 - 2013. Certainly after 2014 or 2015, systemd had spread to the major Linux distros so Docker could have chosen to take a systemd-based approach at that point.
Are there other systemd + containers solutions?
"Chapter 4. Running containers as Systemd services with Podmam" https://access.redhat.com/documentation/en-us/red_hat_enterp...
AFAIU, when running containers with systemd:
- logs go to journald by default
- there's no docker-compose for just the [name-prefixed] containers in the docker-compose.yml,
- you can use systemd unit template parametrization
- it's not as easy to collect metrics on every container on the system without a read-only docker socket: how many containers are running, how much RAM quota are they assigned and utilizing? What are the filesystem and port mappings?
- you can run containers as non-root
- you can run containers in systemd timer units
- you use runC to handle seccomp
... You can do cgroups and namespaces with just systemd; but keeping chroots/images upgraded is outside the scope of systemd: where is the ideal boundary between systemd and containers?
See this comment regarding per-container MAC MCS labels: https://news.ycombinator.com/item?id=23430959
There's much additional complexity that justifies k8s / OpenShift: when would I want to manage containers with just systemd units?
> Many people might think the word “container” has a specific meaning within the Linux kernel; however the kernel has no notion of a “container”. The word has been synonymous with a variety of Linux tooling which when applied give the resemblance of what we expect a container to be.
Before LXC ( https://LinuxContainers.org ) and CNCF ( https://landscape.cncf.io/ ) and OCI ( https://opencontainers.org/ ), for shared-kernel VPS hosting ("virtual private server"; root on a shared box), there was OpenVZ (which requires a patched kernel and AFAIU still has features, like bursting, not present in cgroups).
Docker no longer has an LXC driver: libcontainer (opencontainers/runc) is the story now. The LXC docs have a great list of utilized kernel features that's also still true for docker-engine = runC + moby. The LXC docs: https://linuxcontainers.org/lxc/introduction/ :
> Current LXC uses the following kernel features to contain processes:
> ## Kernel namespaces (ipc, uts, mount, pid, network and user)
>> Namespaces are a feature of the Linux kernel that partitions kernel resources such that one set of processes sees one set of resources while another set of processes sees a different set of resources. https://en.wikipedia.org/wiki/Linux_namespaces
> ## Apparmor and SELinux profiles https://en.wikipedia.org/wiki/AppArmor / https://en.wikipedia.org/wiki/Security-Enhanced_Linux
udica is an interesting tool for creating SELinux policies for containers.
Is it possible for each container to run confined with a different SELinux label?
> ## Seccomp policies https://en.wikipedia.org/wiki/Seccomp
See below re: Seccomp.
> ## Chroots (using pivot_root) https://en.wikipedia.org/wiki/Chroot
Chroots and symlinks, Chroots and bind mounts, Chroots and overlay filesystems, Chroots and SELinux context labels.
FWIU, Chroots are a native feature of filesystem syscalls in Fuchsia.
> ## Kernel capabilities
https://wiki.archlinux.org/index.php/Capabilities :
>> "Capabilities (POSIX 1003.1e, capabilities(7)) provide fine-grained control over superuser permissions, allowing use of the root user to be avoided. Software developers are encouraged to replace uses of the powerful setuid attribute in a system binary with a more minimal set of capabilities. Many packages make use of capabilities, such as CAP_NET_RAW being used for the ping binary provided by iputils. This enables e.g. ping to be run by a normal user (as with the setuid method), while at the same time limiting the security consequences of a potential vulnerability in ping."
> ## CGroups (control groups)* https://en.wikipedia.org/wiki/Cgroups
Control groups enable per-process (and to thus per-container) resource quotas. Other than limiting the impact of resource exhaustion, cgroups are not a security feature of the Linux kernel.
Here's a helpful explainer of the differences between some of these kernel features; which, combined, have become somewhat ubiquitous:
From "Formally add support for SELinux" (k3s #1372) https://github.com/rancher/k3s/issues/1372#issuecomment-5817... :
> https://blog.openshift.com/securing-kubernetes/*
>> The main thing to understand about SELinux integration with OpenShift is that, by default, OpenShift runs each container as a random uid and is isolated with SELinux MCS labels. The easiest way of thinking about MCS labels is they are a dynamic way of getting SELinux separation without having to create policy files and run restorecon.*
>> If you are wondering why we need SELinux and namespaces at the same time, the way I view it is namespaces provide the nice abstraction but are not designed from a security first perspective. SELinux is the brick wall that’s going to stop you if you manage to break out of (accidentally or on purpose) from the namespace abstraction.
>> CGroups is the remaining piece of the puzzle. Its primary purpose isn’t security, but I list it because it regulates that different containers stay within their allotted space for compute resources (cpu, memory, I/O). So without cgroups, you can’t be confident your application won’t be stomped on by another application on the same node.
From Wikipedia: https://en.wikipedia.org/wiki/Seccomp ::
> seccomp (short for secure computing mode) is a computer security facility in the Linux kernel. seccomp allows a process to make a one-way transition into a "secure" state where it cannot make any system calls except exit(), sigreturn(), read() and write() to already-open file descriptors. Should it attempt any other system calls, the kernel will terminate the process with SIGKILL or SIGSYS.[1][2] In this sense, it does not virtualize the system's resources but isolates the process from them entirely.
... SELinux is one implementation of MAC (Mandatory Access Controls) that is built upon the LSM (Linux Security Modules) support in the Linux kernel. Some distros include policy sets for Docker hosts and lots of other packages that could be installed; see: "Formally add support for SELinux" (k3s #1372) https://github.com/rancher/k3s/issues/1372#issuecomment-5817...
How many people did it take to build the Great Pyramid?
> The potential energy of the pyramid—the energy needed to lift the mass above ground level—is simply the product of acceleration due to gravity, mass, and the center of mass, which in a pyramid is one-quarter of its height. The mass cannot be pinpointed because it depends on the specific densities of the Tura limestone and mortar that were used to build the structure; I am assuming a mean of 2.6 metric tons per cubic meter, hence a total mass of about 6.75 million metric tons. That means the pyramid’s potential energy is about 2.4 trillion joules.
In "Lost Technologies of the Great Pyramid" (2010) and "The Great Pyramid Prosperity Machine: Why the Great Pyramid was Built!" (2011), Steven Myers contends that the people who built the pyramids were master hydrologists who built a series of locks from the Nile all the way up the sides of the pyramids and pumped water up to a pool of water on the topmost level; where they used buoyancy and mechanical leverage by way of a floating barge crane in order to place blocks. This would explain how and why the pyramids are water tight, why explosive residue has been found in specific chambers, and why boats have been found buried at the bases of the pyramids.
https://www.amazon.com/dp/B0045Y26CC/
There are videos: http://www.thepump.org/video-series-2
https://www.youtube.com/playlist?list=PLt_DvKGJ_QLYvJ3IdVKXU...
I'm not aware of other explanations for how friction could have been overcome in setting the blocks such that they are watertight (in the later Egyptian pyramids).
AFAIU, the pyramids of South America appear to be of different - possibly older - construction methods.
Solar’s Future is Insanely Cheap
Power markets are more complicated than most people realize. One thing to note is that solar power can be cheaper than gas, but still not be economic. The fact of the matter is that an intermittent kWh is not as valuable as an on-demand reliable kWh to a utility who's number 1 priority is reliability. Even as solar is acquired at lower and lower prices, if evening power is generated by expensive and inefficient gas turbines, the customer might not see costs go down (and emissions might not go down either!). Solar is clearly economic in many markets, but we'll never get grid emissions in a place like California down much more without storage.
As mentioned in the article, solar power is insanely cheap. So while you are right that a kWh of solar is not, on average, as valuable as a kWh of semidispatchable coal, the insane cheapness means that at some point, it is economical to massively overbuild renewable and decommission fossil fuel plants. You provide no evidence for the implausible claim that solar causes emissions to go up: your argument is only that the emissions reducing effect is muted, which may be true.
Also, while storage would be helpful, it is not the only way to enable a renewable transition. Additional transmission is enormously helpful: as the sun goes down on California, the wind is picking up in the Midwest. And don’t forget demand response: if smart thermostats received price signals (maybe we should precool this house...) that would alleviate the evening ramp-up issue.
So I claim we’ll need less storage than “a whole day’s usage”. But the learning curve applies to batteries as well! This storage won’t cost as much anyway.
The whole issue of intermittency is overrated. While a single solar panel might generate intermittently, the solar fleet across a whole state generates more predictably.
I am predicting that grid emissions will come down a lot over the coming decades. Partially I’m predicting the past: they’ve already come down, a lot!
> if smart thermostats received price signals (maybe we should precool this house...) that would alleviate the evening ramp-up issue.
Is there an existing model for retail intraday rates? Would intraday rates be desirable for all market participants?
"Add area for curtailment data?" https://github.com/tmrowco/electricitymap-contrib/issues/236...
Demo of an OpenAI language model applied to code generation [video]
So that's basically program synthesis from natural language (ish) specifications (i.e. the comments).
I can see this being a useful tool [1]. However, I don't expect any ability for innovation. At best this is like having an exceptionally smart autocomplete function that can look up code snippets on SO for you (provided those code snippets are no longer than one line).
That's not to say that it can't write new code, that nobody has quite written before in the same way. But in order for a tool like this to be useful it must stick as close as possible to what is expected- or it will slow development down rather than helping it. Which means it can only do what has already been done before.
For instance- don't expect this to come up with a new sorting algorithm, out of the blue, or to be able to write good code to solve a certain problem when the majority of code solving that problem on github happens to be pretty bad.
In other words: everyone can relax. This will not take your job. Or mine.
____________
[1] I apologise to the people who know me and who will now be falling off their chairs. OK down there?
I think you are underselling the potential of a model which deeply understand programming. Imagine combining such a model with something like AutoML-Zero: https://arxiv.org/abs/2003.03384 It may not be 'creative', but used as tab-completion, it's not being rewarded or incentivized or used in any way which would expose its abilities towards creating a new sort algorithm.
I agree on the tab-completion part. Something like Gmail's smart-compose could have potentially huge benefits here.
But I'm not sure about the "deeply understand programming" part. Language modelling and "AI", in its current form, uncovers only statistical correlations and barely scratches the surface of what "understanding" is. This has restricted deployment of majority of academic research into the real-world and this, I believe, is no different and will work only in constrained settings.
Edit: typo
It would be nice to have an AI that could write unit tests, or look over your code and understand and explain where you might have bugs.
> look over your code and understand and explain where you might have bugs.
This would certainly be interesting. I'm not aware of active research going on in this area (any pointers would be helpful!).
This would require an agent to have thorough understanding of the logic you're trying to implement, and locate the piece of code where it silently fails. For this you'd again need a training dataset where the input is a piece of code and the supervision signal (the output) is location of the bug. I could imagine some sort of self-supervision to tackle this initially where you'd intentionally introduce bugs in your code to generate training data. But not sure how far this can go!
1. Generate test cases from function/class/method definitions.
2. Generate test cases from fuzz results.
3. Run tests and walk outward from symbols around relevant stacktrace frames (line numbers,).
4. Mutate and run the test again.
...
Model-based Testing (MBT) https://en.wikipedia.org/wiki/Model-based_testing
> Models can also be constructed from completed systems
> At best this is like having an exceptionally smart autocomplete function that can look up code snippets on SO for you (provided those code snippets are no longer than one line).
Yeah, all it could do for you is autocomplete around what it thinks the specification might be at that point in time.
> But what if Andy gets another dinosaur, a mean one? -- Toy Story (1995)
Future of the human climate niche
How many degrees Celsius hotter would that be for billions of people in 50 years?
> The Paris Agreement's long-term temperature goal is to keep the increase in global average temperature to well below 2 °C above pre-industrial levels; and to pursue efforts to limit the increase to 1.5 °C, recognizing that this would substantially reduce the risks and impacts of climate change. This should be done by reducing emissions as soon as possible, in order to "achieve a balance between anthropogenic emissions by sources and removals by sinks of greenhouse gases" in the second half of the 21st century. It also aims to increase the ability of parties to adapt to the adverse impacts of climate change, and make "finance flows consistent with a pathway towards low greenhouse gas emissions and climate-resilient development."
> Under the Paris Agreement, each country must determine, plan, and regularly report on the contribution that it undertakes to mitigate global warming. [6] No mechanism forces [7] a country to set a specific emissions target by a specific date, [8] but each target should go beyond previously set targets.
And then this is what was decided:
> In June 2017, U.S. President Donald Trump announced his intention to withdraw the United States from the agreement. Under the agreement, the earliest effective date of withdrawal for the U.S. is November 2020, shortly before the end of President Trump's 2016 term. In practice, changes in United States policy that are contrary to the Paris Agreement have already been put in place.[9]
https://en.wikipedia.org/wiki/Paris_Agreement
Ask HN: Best resources for non-technical founders to understand hacker mindset?
Background: technical founder wondering what reading to recommend to a business focused founder for them to grok the hacker mindset. I've thought perhaps Mythical Man Month and How To Become A Hacker (Eric Raymond essay) but not sure they're quite right.
Any suggestions?
(In case it helps an analogue in the mathematical world might be A Mathematician's Apology or Gödel, Escher, Bach.)
Let me sum up all possible books about understanding the "hacker" (terrible word by the way, because of multiple meanings, which meaning are we talking about?) mindset, to a management perspective:
1) True "hackers" value knowledge over money.
2) True "hackers" value doing things once and doing them right, no matter how long that takes. (Compare to the business mindset of "we need it now", or "we needed it yesterday")
3) True "hackers" value taking ownership in their work, that is, whatever they work on becomes an extension of themselves, much like an artist working on a work of art.
4) True "hackers" are not about work-arounds. If/when work-arounds are used, it's because there there's an artificial timeframe (as might be found in the corporate world), and there's a lack of understanding in the infrastructure which created the need for that work-around.
But, all of these virtues run counter to the demands of business, which constantly wants more things done faster, cheaper, with more features, more complexity, less testing, and doesn't want to worry about problems that may be caused by all of those things in the future (less accountability) -- as long as customer revenue can be collected today.
You see, a true "hacker's" values -- are completely different than those of big business...
And business people wonder why there's stress and burnout among tech people...
> 3) True "hackers" value taking ownership in their work, that is, whatever they work on becomes an extension of themselves, much like an artist working on a work of art
There's something to be said about owning your work, but I have to disagree that unhealthy attachment to work products is a universal attribute of technical founder hackers. It's not a kid, it's a thing that was supposed to be the best use of the resources and information available at the time.
I must have confused this point with vanity and retention in projecting my own counterproductive anti-patterns.
Prolific is not the objective for a true hacker, but not me but a guy I know mentioned something about starting projects and seeing the next 5 years of potentially happily working on that project, too.
Dissecting the code responsible for the Bitcoin halving
> The difficulty of the calculations are determined by how many zeroes need to be at the front. [...]
The difficulty is actually not determined by the number of zeroes (as was initially the case).
https://en.bitcoinwiki.org/wiki/Difficulty_in_Mining :
> The Bitcoin network has a global block difficulty. Valid blocks must have a hash below this target. Mining pools also have a pool-specific share difficulty setting a lower limit for shares.
"Less than" instead of "count leading zeroes" makes it possible for the difficulty to be less broadly adjusted in a difficulty retargeting.
Difficulty retargetings occur after up to 2016 blocks (~10 minutes, assuming the mining pool doesn't suddenly disappear resulting in longer block times that could make it take months to get to 2016 blocks according to "What would happen if 90% of the Bitcoin miners suddenly stopped mining?" https://bitcoin.stackexchange.com/questions/22308/what-would... )
Difficulty is adjusted up or down (every up to 2016 blocks) in order to keep the block time to ~10 minutes.
The block reward halving occurs every ~4 years (210,000 blocks).
Relatedly, Moore's law observes/predicts that processing power (as measured by transistor count per unit) will double every 2 years while price stays the same. Is energy efficiency independent of transistor count? https://en.wikipedia.org/wiki/Moore%27s_law
Ask HN: Does mounting servers parallel with the temperature gradient trap heat?
Heat rises. Is heat trapped in the rack? Would mounting servers sideways (vertically) allow heat to transfer out of the rack?
Many systems have taken the vertical mount approach approach over the years: Blade servers, routers, modems, and various gaming systems.
Horizontally-mounted: parallel with the floor
Vertically-mounted: perpendicular to the floor
[deleted]
Thermodynamics https://en.wikipedia.org/wiki/Thermodynamics
Are engine cylinders ever mounted horizontally? Why or why not?
> Heat rises.
Warmer air is less dense / more buoyant; so it floats.
"Does hot air really rise?" https://physics.stackexchange.com/questions/6329/does-hot-ai...
- Water ice floats because – somewhat uniquely – solid water is less dense than liquid water.
> Is heat trapped in the rack?
Probably.
> Would mounting servers sideways (vertically) allow heat to transfer out of the rack?
How could we find studies that have already tested this hypothesis?
What does the 'rc' in `.bashrc`, etc. mean?
For me, the hard part is remembering if .bashrc is supposed to source .bash_profile or vice versa.
info bash -n "Bash Startup Files"
Google ditched tipping feature for donating money to sites
> When asked, Google confirmed that the designs were an internal idea it explored last year but decided not to pursue as part of [Google Contributor] and Google Funding Choices, which lets sites ask visitors to disable ad blockers, or instead buy a subscription or pay a per page fee to remove ads.
Could this be built on Web Monetization API (ILP (Interledger Protocol)) and e.g. Google Pay as one of many possible payment/card/cryptocurrency processing backends; just like Coil is built on Web Monetization API?
Innovating on Web Monetization: Coil and Firefox Reality
Coil: $5/mo, Content creators get a proportional cut of that amount according to what is browsed with the browser extension enabled or the Puma browser, and Private: No Tracking
> Coil sends payments via the Interledger Protocol, which allows any currency to be used for sending and receiving.
It looks like the Web Monetization API is not yet listed on the Website Monetization Wikipedia page: https://en.wikipedia.org/wiki/Website_monetization
Quoting from earlier this week:
> Web Monetization API (ILP: Interledger Protocol)
>> A JavaScript browser API which allows the creation of a payment stream from the user agent to the website.
>> Web Monetization is being proposed as a #W3C standard at the Web Platform Incubator Community Group.
> https://webmonetization.org/
> Interledger: Web Monetization API https://interledger.org/rfcs/0028-web-monetization/
Ask HN: Recommendations for online essay grading systems?
Which automated essay grading systems would you recommend? Are they open source?
How can we identify biases in these objective systems?
What are your experiences with these systems as authors and graders?
Who else remembers using the Flesch-Kincaid Grade Level metric in Word to evaluate school essays? https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readabi...
Imagine my surprise when I learned that this metric is not one that was created for authors to maximize: reading ease for the widest audience is not an objective in some deparments, but a requirement.
What metrics do and should online essay grading systems present? As continuous feedback to authors, or as final judgement?
I'm reminded of a time in highschool when an essay that I wrote was flagged by an automated essay verification engine as plagiarism. I certainly hadn't plagiarized, and it was up to me to refute that each identified keyword-similar internet resource on the internet was not an uncited source of my paper. I disengaged. I later wrote an essay about how keyword search tools could be helpful to students doing original research. True story.
Decades later, I would guess that human review is still advisable.
This need of mine to have others validate my unpaid work has nothing to do with that traumatic experience.
I still harbor this belief in myself: that what I have to say is worth money to others, and that - someday - I'll pay a journal to consider my ScholarlyArticle for publishing in their prestigious publication with maybe even threaded peer review (and #StructuredPremises linking to Datasets and CreativeWorks that my #LinkedMetaAnalyses are predicated upon). Someday, I'll develop an online persona as a scholar, as a teacher, maybe someday as a TA or an associate professor and connect my CV to any or all of the social networks for academics. I'll work to minimize the costs of interviewing and searching public records. My research will be valued and funded.
Or maybe, like 20% time, I'll find time and money on the side for such worthwhile investigations; and what I produce will be of value to others: more than just an exercise in hearing myself speak.
In my years of internet communications, I've encountered quite a few patrons; lurkers; participants; and ne'er-do-wells who'll order 5 free waters, plaster their posters to the walls, harass paying customers, and just walk out like nothing's going to happen. Moderation costs time and money; and it's a dirty job that sometimes pays okay. There are various systems for grading these comments, these essays, these NewsArticles, these ScholarlyArticles. Human review is still advisable.
> How can we identify biases in these objective systems?
Modern "journalism" recognizes that it's not a one-way monologue but a dialogue: people want to comment. Ignorantly, helpfully, relevantly, insightfully, experiencedly. What separates the "article part" from the "comments part" of the dialogue? Typesetting, CSS, citations, quality of argumentation?
Ask HN: Systems for supporting Evidence-Based Policy?
What tools and services would you recommend for evidence-based policy tasks like meta-analysis, solution criteria development, and planned evaluations according to the given criteria?
Are they open source? Do they work with linked open data?
> Ask HN: Systems for supporting Evidence-Based Policy?
> What tools and services would you recommend for evidence-based policy tasks like meta-analysis, solution criteria development, and planned evaluations according to the given criteria?
> Are they open source? Do they work with linked open data?
I suppose I should clarify that citizens, consumers, voters, and journalists are not acceptable answers
Facebook, Google to be forced to share ad revenue with Australian media
What if Google and Facebook just said “No, thank you.”? I think Google and Facebook have more power in this relationship, which might say something about the tech giants. The only possible thing the Aussies could do is block the websites, and that sounds like political suicide to me. I’m guessing the companies will decide to play ball, but if other countries start to follow suit, it might be interesting to see what happens if they start pushing back.
There would be no political suicide there. It's so easy to make Google/Facebook look evil in this case if they don't comply. Just say they are not paying taxes and getting our money abroad etc.
But there is 0% chance of Google/Facebook not complying.
> I think Google and Facebook have more power in this relationship
Not even close.
Don't the French and Spanish examples indicate otherwise? Spain's link tax in 2014 led to the shutting-down of Google News Spain, and France's attempt to charge for Google News snippets led to the removal of those snippets for French sites (followed by a dip in traffic for pubs that hurt them far more than it hurt Google). What makes you think that FB/G don't have the power here?
If you don't want them to index your content and send you free traffic, you can already specify that in your robots.txt; for free. https://en.wikipedia.org/wiki/Robots_exclusion_standard
There are no ads on Google News.
There is an apparent glut of online news: supply exceeds demand and so the price has fallen.
> There are no ads on Google News.
This is the ironic part of the policy. You're right about the glut, and there was a glut in print 20 years ago, too. Google's definitely hurt businesses, but it's usually though disintermediating them (think Yelp). Google News and search don't really do that for news--snippets and headlines aren't the same as an article. I find it hard to believe that ABC's brand isn't strong enough for them to pull their content from Google and expect people to go to abc.net.au; I just don't think than can sell enough ads to give the content away.
By hurt, do you mean competed with by effectively utilizing technology to help people find information about the world from multiple sources.
There are very many news aggregators and most do serve ads next to the headlines they index. I assume that people typically link out from news aggregation sites more than into vertically-integrated services.
Perhaps the content producers / information service providers could develop additional revenue streams in order to subsidize a news aggregation public service. Micropayments (BAT, Web Monetization (ILP)), ads, paywalls, and public and private grants are sources of revenue for content producers.
I think it's disingenuous to blame news aggregation sites for the unprofitability of extremely redundant journalism. What happened to journalism? Internet. Excessive ads. Aren't we all writers these days.
Unfortunately they killed the "most cited" and was it "most in-depth" source analysis functions of Google News; and now we're stuck with regurgitated news wires and press releases and all of these eyewitness mobile phone videos with two-bit banal commentary and also punditry. How the world has changed.
So, as far as scientific experiments are concerned, it might be interesting to see what the impact of de-listing from free time sites X, Y, and Z is.
Do the papers in Australia and France now intend to compensate journal ScholarlyArticle authors whose work they summarize and hopefully at least cite the titles and URLs of, or the journals themselves?
France rules Google must pay news firms for content
Website monetization https://en.wikipedia.org/wiki/Website_monetization
Web Monetization API (ILP: Interledger Protocol)
> A JavaScript browser API which allows the creation of a payment stream from the user agent to the website.
> Web Monetization is being proposed as a #W3C standard at the Web Platform Incubator Community Group.
Interledger: Web Monetization API https://interledger.org/rfcs/0028-web-monetization/
Khan Academy, for example, accepts BAT (Basic Attention Token) micropayments/microdonations that e.g. Brave browser users can opt to share with the content producers and indexers. https://en.wikipedia.org/wiki/Brave_(web_browser)#Basic_Atte...
Web Monetization w/ Interledger should enable any payments system with low enough transaction costs ("ledger-agnostic, currency agnostic") to be used to pay/tip/donate to content producers who are producing unsensational, unbiased content that people want to pay for.
Paywalls/subscriptions and ads are two other approaches to funding quality journalism.
Should journalists pay ScholarlyArticle authors whose studies they publish summaries of without even citing the DOI/URL and Title; or the journals said ScholarlyArticles are published in? https://schema.org/ScholarlyArticle
Adafruit Thermal Camera Imager for Fever Screening
> Thermal Camera Imager for Fever Screening with USB Video Output - UTi165K. PRODUCT ID: 4579 https://www.adafruit.com/product/4579
> This video camera takes photos of temperatures! This camera is specifically tuned to work in the 30˚C~45˚C / 86˚F~113˚ F range with 0.5˚C / 1˚ F accuracy, so it's excellent for human temperature & fever detection. In fact, this thermal camera is often used by companies/airports/hotels/malls to do a first-pass fever check: If any person has a temperature of over 99˚F an alarm goes off so you can do a secondary check with an accurate handheld temperature meter.
> You may have seen thermal 'FLIR' cameras used to find air leaks in homes, but those cameras have a very wide temperature range, so they're not as accurate in the narrow range used for fever-scanning. This camera is designed specifically for that purpose!
... USB Type-C, SD Card; no price listed yet?
The end of an Era – changing every single instance of a 32-bit time_t in Linux
Thanks!
Year 2038 Problem > Solutions is already updated re: 5.6. https://en.wikipedia.org/wiki/Year_2038_problem
Ask HN: What's the ROI of Y Combinator investments?
To calculate the ROI of YC investments, we could find the terms of the YC investments (x for y%, preference) and find the exit rate (what % of companies exit).
We could search for 'ROI of ycombinator investments' and find valuation numbers from a number of years ago.
From the first page of search results, we'd then learn about "return on capital" and how the standard YC seed terms have changed over the years.
Return on capital: https://en.wikipedia.org/wiki/Return_on_capital
From the See also section of this Wikipedia page, we might discover "Cash flow return on investment" and "Rate of return on a portfolio"
From the "rate of return" Wikipedia page, we might learn that "The return on investment (ROI) is return per dollar invested. It is a measure of investment performance, as opposed to size (c.f. return on equity, return on assets, return on capital employed)." and that "The annualized return of an investment depends on whether or not the return, including interest and dividends, from one period is reinvested in the next period. " https://en.wikipedia.org/wiki/Rate_of_return
From the YCombinator Wikipedia page, we might read that "The combined valuation of the top YC companies was over $155 billion as of October, 2019. [4]" and that "As of late 2019, Y Combinator had invested in >2,000 companies [37], most of which are for-profit. Non-profit organizations can also participate in the main YC program. [38]" and then read about "seed accelerators" and then "business incubators" in search of appropriate metrics for comparing VC performance. https://en.wikipedia.org/wiki/Y_Combinator
ROI is such a frou frou statistic anyway. What does that even mean, ROI? In any case, YC itself is not a public company, per se, AFAICT, so, it's not so easy as going to https://YCharts.com, entering the equity symbol, clicking on "Key Stats", and scrolling down to "Profitability" to review [Gross | EBITDA | Operating] [Profit] Margin.
The LTSE (Long-Term Stock Exchange) is where people who are in this for real are really doing it now.
Microsoft announces Money in Excel powered by Plaid
This looks really useful.
At first glance, I found a number of ways to push transaction data into Google Sheets from the Plaid API:
build-your-own-mint (NodeJS, CircleCI) https://github.com/yyx990803/build-your-own-mint
go-plaid: https://github.com/ebcrowder/go-plaid
Presumably, like the GOOGLEFINANCE function, there's some way to pull data from an API with just Apps Script (~JS) without an auxiliary serverless function to get the txs from Plaid and post to the gsheets API?
Lora-based device-to-device smartphone communication for crisis scenarios [pdf]
Sudomesh has been working on one of these devices, disaster radio: https://disaster.radio
I like the solar integration. My ideal product would be a programmable LoRa transceiver integrates into a waterproof battery bank with solar, gps and a small touch screen. Should be about the size of a large handheld which could mounted on a house, tree, mast or carry it in a pocket/bag. Apps could be, SOS messages, chat and weather information. If it’s extensible (e.g. something like an app store or package manager), I’m sure folks will dream up additional uses.
Sort of like a more hackable/accessible phone and long range radio.
Unfortunately, the Earl tablet never made it to market: https://blog.the-ebook-reader.com/2015/01/26/video-update-ab...
Earl specs: Waterproof; Solar charging; eInk; ANT+; NFC; VHF/UHF transceiver (GMRS, PMR446, UHFCB); GPS; Sensors: Accelerometer, Gyroscope, Magnetometer, Temperature, Barometer, Humidity; AM/FM/SW/LW/WB
LTE, LoRa, 5G, and Hostapd would be great
Being able to plug it into a powerbank and antennas for use as a fixed or portable e.g. BATMAN mesh relay would be great
"LoRa+WiFi ClusterDuck Protocol by Project OWL for Disaster Relief" https://news.ycombinator.com/item?id=22707267
> An opkg (for e.g. OpenWRT) with this mesh software would make it possible to use WiFi/LTE routers with a LoRa transmitter/receiver connected over e.g. USB or Mini-PCIe.
LoRa+WiFi ClusterDuck Protocol by Project OWL for Disaster Relief
> Project OWL (Organization, Whereabouts, and Logistics) creates a mesh network of Internet of Things (IoT) devices called DuckLinks. These Wi-Fi-enabled devices can be deployed or activated in disaster areas to quickly re-establish connectivity and improve communication between first responders and civilians in need.
> In OWL, a central portal connects to solar- and battery-powered, water-resistant DuckLinks. These create a Local Area Network (LAN). In turn, these power up a Wi-Fi captive portal using low-frequency Long-range Radio (LoRa) for Internet connectivity. LoRA has a greater range, about 10km, than cellular networks.
...
> You don't actually need a DuckLink device. The open-source OWL firmware can quickly turn a cheap wireless device into a DuckLink using the -- I swear I'm not making this up -- ClusterDuck Protocol. This is a mesh network node, which can hook up to any other near-by Ducks.
> OWL is more than just hardware and firmware. It's also a cloud-based analytic program. The OWL Data Management Software can be used to facilitate organization, whereabouts, and logistics for disaster response.
Homepage: http://clusterduckprotocol.org/
GitHub: https://github.com/Code-and-Response/ClusterDuck-Protocol
The Linux Foundation > Code and Response https://www.linuxfoundation.org/projects/code-and-response/
An opkg (for e.g. OpenWRT) with this mesh software would make it possible to use WiFi/LTE routers with a LoRa transmitter/receiver connected over e.g. USB or Mini-PCIe.
... cc'ing from https://twitter.com/westurner/status/1238859774567026688 :
OpenWRT is a Make-based embedded Linux distro w/ LuCI (Lua + JSON + UCI) web interface).
#OpenWRT runs on RaspberryPis, ARM, x86, ARM, MIPS; there's a Docker image. OpenWRT Supported Devices: https://openwrt.org/supported_devices
OpenWRT uses opkg packages: https://openwrt.org/docs/guide-user/additional-software/opkg
I searched for "Lora" in OpenWRT/packages: lora-gateway-hal opkg package: https://github.com/openwrt/packages/blob/master/net/lora-gat...
lora-packet-forwarder opkg package (w/ UCI integration): https://github.com/openwrt/packages/pull/8320
https://github.com/xueliu/lora-feed :
> Semtech packages and ChirpStack [(LoRaserver)] Network Server stack for OpenWRT
> > [In addition to providing node2node/2net connectivity, #batman-adv can bridge VLANs over a mesh (or link), such as for “trusted” client, guest, IoT, and mgmt networks. It provides an easy-to-configure alternative to other approaches to “backhaul”, […]] https://openwrt.org/docs/guide-user/network/wifi/mesh/batman
> I have a few different [quad-core, MIMO] ARM devices without 4G. TIL that the @GLiNetWifi devices ship with OpenWRT firmware (and a mobile config app) and some have 1-2 (Mini-PCIe) 4G w/ SIM slots. Also, @turris_cz has OpenWRT w/ LXC in the kernel build. https://t.co/Rz0Uu5uHJQ
A Visual Debugger for Jupyter
Source: jupyterlab/debugger https://github.com/jupyterlab/debugger
When will jupyter have "highlight and execute" functionality? The cell concept is fine, but I'm constantly copy pasting snippets of code into new cells to get that "incremental" coding approach...
So, I went looking for the answer to this because in the past I've installed the scratchpad extension by installing jupyter_contrib_nbextensions, but those don't work with JupyterLab because there's a new extension model for JupyterLab that requires node and npm.
Turns out that with JupyterLab, all you have to to is right-click and select "New Console for Notebook" and it opens a console pane below the notebook already attached to the notebook kernel. You can also instead do File > New > Console and select a kernel listed under "Use Kernel From Other Session".
The "New action runInConsole to allow line by line execution of cell content" "PR adds a notebook command `notebook:run-in-console`" but you have to add the associated keyboard shortcut to your config yourself; e.g. `Ctrl Shift Enter` or `Ctrl-G` that calls `notebook:run-in-console`. https://github.com/jupyterlab/jupyterlab/pull/4330
"In Jupyter Lab, execute editor code in Python console" describes how to add the associated keyboard shortcut to your config: https://stackoverflow.com/questions/38648286/in-jupyter-lab-...
Ask HN: What's the Equivalent of 'Hello, World' for a Quantum Computer?
The 'Hello,World' program is one of the simplest programs to demonstrate how to go about writing a program in a new programming language.
What is an equivalent simple program which demonstrates how to write a very simple program for a quantum computer?
I have tried (and failed) to imagine such a program. Can somebody who has actually used a quantum computer show us an actual quantum computer program?
"Getting Started with Qiskit" https://qiskit.org/documentation/getting_started.html
Qiskit / qiskit-community-tutorials > "1 Hello, Quantum World with Qiskit" https://github.com/Qiskit/qiskit-community-tutorials#1-hello...
"Quantum basics with Q#" https://docs.microsoft.com/en-us/quantum/quickstart?tabs=tab...
Qutip notebooks: https://github.com/qutip/qutip#run-notebooks-online
Jupyter Notebooks labeled quantum-computing: https://github.com/topics/quantum-computing?l=jupyter+notebo...
Ask HN: Communication platforms for intermittent disaster relief?
Are there good platforms for disaster relief that work well with intermittent connectivity (i.e. spotty 3G/4G/WiFi/LoRa)?
How can major networks improve in terms of e.g. indicating message delivery status, most recent sync time, sync throttling status due to load, optionally downloading images/audio/video, referring people to local places and/or forms for help with basic needs, etc?
What are some tools that app developers can use to simulate intermittent connectivity when running tests?
DroneAid: A Symbol Language and ML model for indicating needs to drones, planes
From the README https://github.com/Code-and-Response/DroneAid :
> The DroneAid Symbol Language provides a way for those affected by natural disasters to express their needs and make them visible to drones, planes, and satellites when traditional communications are not available.
> Victims can use a pre-packaged symbol kit that has been manufactured and distributed to them, or recreate the symbols manually with whatever materials they have available.
> These symbols include those below, which represent a subset of the icons provided by The United Nations Office for the Coordination of Humanitarian Affairs (OCHA). These can be complemented with numbers to quantify need, such as the number or people who need water.
Each of the symbols are drawn within a triangle pointing up:
- Immediate Help Needed (orange; downward triangle \n SOS),
- Shelter Needed (cyan; like a guy standing in a tall pentagon without a floor),
- OK: No Help Needed (green; upward triangle \n OK),
- First Aid Kit Needed (yellow; briefcase with a first aid cross),
- Water Needed (blue; rain droplet), Area with Children in Need (lilac; baby looking thing with a diaper on),
- Food Needed (red; pan with wheat drawn above it),
- Area with Elderly in Need (purple; person with a cane)
So, we're going to need some artists; something to write large things with; some orange, cyan, green, yellow, blue, lilac, red, and purple things; some people who can tell me the difference between lilac (light purple: babies) and purple (darker purple: old people); and some drones that can capture location and imagery.
Note that DroneAid is also a project of The Linux Foundation Code and Response organization.
Ask HN: Computer Science/History Books?
Hi guys, can you recommend interesting books on Computer Science or computer history (similar to Dealers of Lightning) to read on this quarantine times? I really like that subject and am looking for something to keep myself away from TV at night.
Thank you.
The Information: a History, a Theory, a Flood by James Gleick is great. Starts with Ada Lovelace/Charles Babbage and goes on from there, I found it fascinating.
"The Information: A History, a Theory, a Flood" starts with "1 | Drums That Talk" re: African drum messaging; a complex coding scheme:
> Here was a messaging system that outpaced the best couriers, the fastest horses on good roads with way stations and relays.
https://en.wikipedia.org/wiki/The_Information:_A_History,_a_...
https://www.goodreads.com/book/show/8701960-the-information
From "Polynesian People Used Binary Numbers 600 Years Ago" https://www.scientificamerican.com/article/polynesian-people... :
> Binary arithmetic, the basis of all virtually digital computation today, is usually said to have been invented at the start of the eighteenth century by the German mathematician Gottfried Leibniz. But a study now shows that a kind of binary system was already in use 300 years earlier among the people of the tiny Pacific island of Mangareva in French Polynesia.
Open-source security tools for cloud and container applications
It's odd they don't list OPA Gatekeeper, which is probably the best tool for enforcing security and other best practices in Kubernetes clusters.
List of CNCF open source security projects without the blog post: https://landscape.cncf.io/category=security-compliance&forma...
> List of CNCF open source security projects without the blog post: https://landscape.cncf.io/category=security-compliance&forma...
Thanks for this.
YC Companies Responding to Covid-19
Are life sciences and healthcare familiar verticals for YC?
Good to see money and talent going to such good use.
(Edit) Here's the YC Companies list; which doesn't yet list these new investments:
Biomedical vertical: https://www.ycombinator.com/companies/?vertical=Biomedical
Healthcare vertical: https://www.ycombinator.com/companies/?vertical=Healthcare
"The Y Combinator Database" https://www.ycdb.co/
Show HN: Neh – Execute any script or program from Nginx location directives
And we've come back round full circle from CGI scripts, although separating out headers and body on different fds sounds neat
Many things we’ve dispensed with long ago seem to be forgotten and the younger coders are learning for themselves the hard way. Aka the same way we did.
One of the things I'm wondering however: How popular is CGI if I don't find it with the query "run script on nginx location directive".
Nginx probably somewhat-deliberately has FastCGI but not regular CGI for a number of reasons.
CGI has process-per-request overhead.
CGI typically runs processes as the user the webserver is running as; said processes can generally read and write to the unsandboxed address space of the calling process (such as x.509 private certs).
Just about any app can be (D)DOS'd. That requires less resources with the process-per-request overhead of CGI.
In order to prevent resource exhaustion due to e.g someone benignly hitting reload a bunch of times and thus creating multiple GET requests, applications should enqueue task messages which a limited number of workers retrieve from a (durable) FIFO or priority queue and update the status of.
Websockets may or may not scale better than long-polling for streaming stdout to a client.
Interesting and really informative! Thanks for sharing!
Ask HN: How can a intermediate-beginner learn Unix/Linux and programming?
For a long time, I’ve been in an awkward position with my knowledge of computers. I know basic JavaScript (syntax and booleans and nothing more). I’ve learned the bare basics of Linux from setting up a Pi-hole. I understand the concept of secure hashes. I even know some regex.
The problem is, I know so little that I can’t actually do anything with this knowledge. I suppose I’m looking for a tutorial that will teach me to be comfortable with the command line and a Unix environment, while also teaching me to code a language. Where should I start?
a really good book is Richard Stevens Advanced Programming in the UNIX environment[1]. (sounds daunting but it's not too bad when you combine it with another introductory text on C). If you stick with C you'll eventually know UNIX/Linux a lot deeper than anyone. It takes time though so go easy on yourself.
Also check github for a bunch of repos that contain biolerplate code that is used in most deamons illustrating signal handling, forking, etc.[2]
Also I suggest taking some open source projects from Daniel Bernstein (DJB) as examples on how to write secure code. qmail, djbdns, daemontools ... there are lots of great ideas there[3]
Write a couple of simple programs that utilize your code and expand from there, learn the plumbing, e.g. write a Makefile, learn autotools/autoconf, how to write tests, how to use gdb, how to create shared libs and link a program LD_LIBRARY_PATH/ldconfig, etc ...
Most important think about something that you like to write and go from there. Back in the late 90ies I studied RFC's (MIME, SMTP, etc) and then implemented things that I could actually use myself. Here is a recent project I did some weeks ago (an extreme edge-case for security maximalists which will never be used by anyone other then myself, ... but I really enjoyed writing this and learned a bunch of new stuff while doing so)[4]
if you need ideas or help with looking at your code, don't hesitate and send me a mail.
[1] https://en.wikipedia.org/wiki/Advanced_Programming_in_the_Un...
> Also check github for a bunch of repos that contain biolerplate code that is used in most deamons illustrating signal handling, forking, etc.[2]
docker-systemctl-replacement is a (partial) reimplementation of systemd as one python script that can be run as the init process of a container that's helpful for understanding how systemd handles processes: https://github.com/gdraheim/docker-systemctl-replacement/blo...
systemd is written in C: https://github.com/systemd/systemd
> examples of how to write secure code
awesome-safety-critical > Coding Guidelines https://github.com/stanislaw/awesome-safety-critical/blob/ma...
Math Symbols Explained with Python
Average of a finite series: There's a statistics module in Python 3.4+:
X = [1, 2, 3]
from statistics import mean, fmean
mean(X)
# may or may not be preferable to
sum(X) / len(X)
Product of a terminating iterable:
import operator
from functools import reduce
# from itertools import accumulate
reduce(operator.mul, X)
from numpy import linalg as LA
LA.norm(X)
Function domains and ranges can be specified and checked at compile-time with type annotations or at runtime with type()/isinstance() or with something like pycontracts or icontracts for checking preconditions and postconditions.
Dot product:
Y = [4, 5, 6]
np.dot(X, Y)
Unit vector:
X / np.linalg.norm(X)Ask HN: Is there way you can covert smartphone to a no contact thermometer?
Wondering is there an infrared dongle that can convert your phone to a no contact thermometer to read body temperature?
Infrared thermometer: https://en.wikipedia.org/wiki/Infrared_thermometer
Thermography: https://en.wikipedia.org/wiki/Thermography
IDK what the standard error is for medical temperature estimation with an e.g. FLIR ONE thermal imaging camera for an Android/iOS device. https://www.flir.com/applications/home-outdoor/
I'd imagine that sanitization would be crucial for any clinical setting.
(Edit) "Prediction of brain tissue temperature using near-infrared spectroscopy" (2017) Neurophotonics https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5469395/
"Nirs body temperature" https://scholar.google.com/scholar?hl=en&as_sdt=0%2C43&q=nir...
"Infrared body temperature" https://scholar.google.com/scholar?hl=en&as_sdt=0%2C43&q=inf...
"Infrared thermometer iOS" https://m.alibaba.com/trade/search?SearchText=infrared%20the...
"Infrared thermometer Android" https://alibaba.com/trade/search?SearchText=infrared%20therm...
Employee Scheduling
From "Ask HN: What algorithms should I research to code a conference scheduling app" https://news.ycombinator.com/item?id=15267804 :
> Resource scheduling, CSP (Constraint Satisfaction programming)
CSP: https://en.wikipedia.org/wiki/Constraint_satisfaction_proble...
Scheduling (production processes):
https://en.wikipedia.org/wiki/Scheduling_(production_process...
Scheduling (computing):
https://en.wikipedia.org/wiki/Scheduling_(computing)
... To an OS, a process thread has a priority and sometimes a CPU affinity.
From http://markmail.org/search/?q=list%3Aorg.python.omaha+pysche... :
Pyschedule:
- Src: https://github.com/timnon/pyschedule
From https://github.com/timnon/pyschedule :
> pyschedule is python package to compute resource-constrained task schedules. Some features are:
- precedence relations: e.g. task A should be done before task B
- resource requirements: e.g. task A can be done by resource X or Y
- resource capacities: e.g. resource X can only process a few tasks
Previous use-cases include:
- school timetables: assign teachers to classes
- beer brewing: assign equipment to brewing stages
- sport schedules: assign stadiums to games
... https://en.wikipedia.org/wiki/Slurm_Workload_Manager :
> Slurm is the workload manager on about 60% of the TOP500 supercomputers.[1]
Slurm uses a best fit algorithm based on Hilbert curve scheduling or fat tree network topology in order to optimize locality of task assignments on parallel computers.[2]
... https://en.wikipedia.org/wiki/Hilbert_curve_scheduling :
> [...] the Hilbert curve scheduling method turns a multidimensional task allocation problem into a one-dimensional space filling problem using Hilbert curves, assigning related tasks to locations with higher levels of proximity.[1] Other space filling curves may also be used in various computing applications for similar purposes.[2]
Show HN: Simulation-based high school physics course notes
I love the train demo for general & special relativity, they intuitively explain the theories from frames of reference (ground/train).
This helped me instantly understand the 2 theories, great work!
WebAssembly brings extensibility to network proxies
FWIW, Ethereum WASM (ewasm) has a cost (in "particles" ("gas")) for each WebAssembly opcode. [1]
Opcode costs help to incentivize efficient code.
ewasm/design /README.md [2] links to the complete WebAssembly instruction set. [3]
[1] https://ewasm.readthedocs.io/en/mkdocs/determining_wasm_gas_...
[2] https://github.com/ewasm/design/blob/master/README.md
[3] https://webassembly.github.io/spec/core/appendix/index-instr...
Pandemic Ventilator Project
> https://www.projectopenair.org/
From https://app.jogl.io/project/121#about
>> Current Status of the project
>> The main bottleneck currently (2020-03-13) is organization / management.
>> […] This is an organization of experts and hobbyists from around the globe.
The link https://app.jogl.io/project/121#about doesn't go anywhere anymore. And there are no results for "ventilator" in the app.jogl.io search.
I see a "Ventilator Project" heading?
(edit) here's the link to their 'Ventilator' document: https://docs.google.com/document/d/1RDihfZIOEYs60kPEIVDe7gms...
Low-cost ventilator wins Sloan health care prize (2019)
Relating to coronavirus (COVID-19), if you require a ventilator, your survival chances are already slim.
> The median time from illness onset (ie, before admission) to discharge was 22·0 days (IQR 18·0–25·0), whereas the median time to death was 18·5 days (15·0–22·0; table 2). 32 patients required invasive mechanical ventilation, of whom 31 (97%) died. The median time from illness onset to invasive mechanical ventilation was 14·5 days (12·0–19·0). Extracorporeal membrane oxygenation was used in three patients, none of whom survived. Sepsis was the most frequently observed complication, followed by respiratory failure, ARDS, heart failure, and septic shock (table 2). Half of non-survivors experienced a secondary infection, and ventilator-associated pneumonia occurred in ten (31%) of 32 patients requiring invasive mechanical ventilation. The frequency of complications were higher in non-survivors than survivors (table 2)
https://www.thelancet.com/action/showPdf?pii=S0140-6736%2820...
Ventilator availability is limiting our ability to get care to the most people we can.
True, but at least Italy claims that the main bottleneck in ventilator availability isn't in the machines themselves but in trained doctors and nurses to support patients on ventilators.
AI can detect coronavirus from CT scans in twenty seconds
Is it possible to detect coronavirus with NIRS (Near-Infrared Spectroscopy)? https://en.wikipedia.org/wiki/Near-infrared_spectroscopy
FWIU, the equipment costs and scan times are lower with NIRS than with CT or MRI? And infrared is zero rads?
(Edit) I think it was this or the TED video that had the sweet demo: "The Science of Visible Thought & Our Translucent Selves | Mary Lou Jepsen | SU Global Summit" https://youtu.be/IRCXNBzfeC4
Are these devices in production?
AutoML-Zero: Evolving machine learning algorithms from scratch
Next:
- Autosuggest database tables to use
- Automatically reserve parallel computing resources
- Autodetect data health issues and auto fix them
- Autodetect concept drift and auto fix it
- Auto engineer features and interactions
- Autodetect leakage and fix it
- Autodetect unfairness and auto fix it
- Autocreate more weakly-labelled training data
- Autocreate descriptive statistics and model eval stats
- Autocreate monitoring
- Autocreate regulations reports
- Autocreate a data infra pipeline
- Autocreate a prediction serving endpoint
- Auto setup a meeting with relevant stakeholders on Google Calendar
- Auto deploy on Google Cloud
- Automatically buy carbon offset
- Auto fire your in-house data scientists
Would be funny but most of those things are already on AutoML Tables, including the carbon offset
> Would be funny but most of those things are already on AutoML Tables, including the carbon offset
GCP datacenters are 100% offset with PPAs. Are you referring to different functionality for costing AutoML instructions in terms of carbon?
...
I'd add:
- Setup a Jupyter Notebook environment
> Jupyter Notebooks are one of the most popular development tools for data scientists. They enable you to create interactive, shareable notebooks with code snippets and markdown for explanations. Without leaving Google Cloud's hosted notebook environment, AI Platform Notebooks, you can leverage the power of AutoML technology.
> There are several benefits of using AutoML technology from a notebook. Each step and setting can be codified so that it runs the same every time by everyone. Also, it's common, even with AutoML, to need to manipulate the source data before training the model with it. By using a notebook, you can use common tools like pandas and numpy to preprocess the data in the same workflow. Finally, you have the option of creating a model with another framework, and ensemble that together with the AutoML model, for potentially better results.
https://cloud.google.com/blog/products/ai-machine-learning/u...
This sounds like the sort of thing that would be useful outside of data science. Which leads to the question of whether it needs to be generalized, or redone differently for different specializations. Which in turn seems like the sort of question that it's tricky to answer with AI.
> This sounds like the sort of thing that would be useful outside of data science.
The instruction/operation costing or the computational essay/notebook environment setup?
Ethereum ("gas") and EOS have per-instruction costing. SingularityNET is a marketplace for AI solutions hosted on a blockchain, where you pay for AI/ML services with the SingularityNET AGI token. E.g. GridCoin and CureCoin compensate compute resource donations with their own tokens; which also have a floating exchange rate.
TLJH: "The Littlest JupyterHub" describes how to setup multi-user JupyterHub with e.g. Docker spawners that isolate workloads running with shared resources like GPUs and TPUs: http://tljh.jupyter.org/en/latest/
"Zero to BinderHub" describes how to setup BinderHub on a k8s cluster: https://binderhub.readthedocs.io/en/latest/zero-to-binderhub...
The notebook/procedure thing. Like, doesn't everybody everywhere operate on a basis of mixed manual/automated procedures, where it needs to fluidly transition from one to another, yet be controlled and recorded and verified and structured?
REES is one solution to reproducibility of the computational environment.
> BinderHub ( https://mybinder.org/ ) creates docker containers from {git repos, Zenodo, FigShare,} and launches them in free cloud instances also running JupyterLab by building containers with repo2docker (with REES (Reproducible Execution Environment Specification)). This means that all I have to do is add an environment.yml to my git repo in order to get Binder support so that people can just click on the badge in the README to launch JupyterLab with all of the dependencies installed.
> REES supports a number of dependency specifications: requirements.txt, Pipfile.lock, environment.yml, aptSources, postBuild. With an environment.yml, I can install the necessary CPython/PyPy version and everything else.
REES: https://repo2docker.readthedocs.io/en/latest/specification.h...
REES configuration files: https://repo2docker.readthedocs.io/en/latest/config_files.ht...
Storing a container built with repo2docker in a container registry is one way to increase the likelihood that it'll be possible to run the same analysis pipeline with the same data and get the same results years later.
...
Pachyderm ( https://pachyderm.io/platform/ ) does Data Versioning, Data Pipelines (with commands that each run in a container), and Data Lineage (~ "data provenance"). What other platforms are there for versioning data and recording data provenance?
...
Recording manual procedures is an area where we've somewhat departed from the "write in a lab notebook with a pen" practice. CoCalc records all (collaborative) inputs to the notebook with a timeslider for review.
In practice, people use notebooks for displaying generated charts, manual exploratory analyses (which does introduce bias), for demonstrating APIs, and for teaching.
Is JupyterLab an ideal IDE? Nope, not by a longshot. nbdev makes it easier to write a function in a notebook, sync it to a module, edit it with a more complete data-science IDE (like RStudio, VSCode, Spyder, etc), and then copy it back into the notebook. https://github.com/fastai/nbdev
> What other platforms are there for versioning data and recording data provenance?
Quilt also versions data and data pipelines: https://medium.com/pytorch/how-to-iterate-faster-in-machine-...
https://github.com/quiltdata/quilt (Python)
Poor data scientists, now whose heads get cut when things go wrong and companies lose billions?
In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.
“What are you doing?”, asked Minsky.
“I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied.
“Why is the net wired randomly?”, asked Minsky.
“I do not want it to have any preconceptions of how to play”, Sussman said.
Minsky then shut his eyes.
“Why do you close your eyes?”, Sussman asked his teacher.
“So that the room will be empty.”
At that moment, Sussman was enlightened.
Is this an argument in favor of unjustified magic constant arbitrary priors?
A sufficiently large amount of random data contains all the magic constants you could want.
AutoML-Zero aims to automatically discover computer programs that can solve machine learning tasks, starting from empty or random programs and using only basic math operations.
If this system is not using human bias, who is it choosing what good program is? Surely, human labeling data involves humans adding their bias to the data?
It seems like AlphaGoZero was able to do just end-to-end ML because it was able to use a very clear and "objective" standard, whether a program wins or loses at the game of Go.
Would this approach only deal with similarly unambiguous problems?
Edit: also, AlphaGoZero was one of the most ML ever created (at least at the time of its creation). How much computing resources would this require for more fully general learning? Will there be a limit to such an approach?
Options for giving math talks and lectures online
One option: screencast development of a Jupyter notebook.
Jupyter Notebook supports LaTeX (MathTeX) and inline charts. You can create graded notebooks with nbgrader and/or with CoCalc (which records all (optionally multi-user) input such that you can replay it with a time slider).
Jupyter notebooks can be saved to HTML slides with reveal.js, but if you want to execute code cells within a slide, you'll need to install RISE: https://rise.readthedocs.io/en/stable/
Here are the docs for CoCalc Course Management; Handouts, Assignments, nbgrader: https://doc.cocalc.com/teaching-course-management.html
Here are the docs for nbgrader: https://nbgrader.readthedocs.io/en/stable/
You can also grade Jupyter notebooks in Open edX:
> Auto-grade a student assignment created as a Jupyter notebook, using the nbgrader Jupyter extension, and write the score in the Open edX gradebook
https://github.com/ibleducation/jupyter-edx-grader-xblock
Or just show the Jupyter notebook within an edX course: https://github.com/ibleducation/jupyter-edx-viewer-xblock
There are also ways to integrate Jupyter notebooks with various LMS / LRS systems (like Canvas, Blackboard, etc) "nbgrader and LMS / LRS; LTI, xAPI" on the "Teaching with Jupyter Notebooks" mailing list: https://groups.google.com/forum/#!topic/jupyter-education/_U...
"Teaching and Learning with Jupyter" ("An open book about Jupyter and its use in teaching and learning.") https://jupyter4edu.github.io/jupyter-edu-book/
> TLJH: "The Littlest JupyterHub" describes how to setup multi-user JupyterHub with e.g. Docker spawners that isolate workloads running with shared resources like GPUs and TPUs: http://tljh.jupyter.org/en/latest/
> "Zero to BinderHub" describes how to setup BinderHub on a k8s cluster: https://binderhub.readthedocs.io/en/latest/zero-to-binderhub...
If you create a git repository with REES-compatible dependency specification file(s), students can generate a container with all of the same software at home with repo2docker or with BinderHub.
> REES is one solution to reproducibility of the computational environment.
>> BinderHub ( https://mybinder.org/ ) creates docker containers from {git repos, Zenodo, FigShare,} and launches them in free cloud instances also running JupyterLab by building containers with repo2docker (with REES (Reproducible Execution Environment Specification)). This means that all I have to do is add an environment.yml to my git repo in order to get Binder support so that people can just click on the badge in the README to launch JupyterLab with all of the dependencies installed.
>> REES supports a number of dependency specifications: requirements.txt, Pipfile.lock, environment.yml, aptSources, postBuild. With an environment.yml, I can install the necessary CPython/PyPy version and everything else.
> REES: https://repo2docker.readthedocs.io/en/latest/specification.h...
> REES configuration files: https://repo2docker.readthedocs.io/en/latest/config_files.ht...
> Storing a container built with repo2docker in a container registry is one way to increase the likelihood that it'll be possible to run the same analysis pipeline with the same data and get the same results years later.
Aerogel from fruit biowaste produces ultracapacitors
> "Aerogel from fruit biowaste produces ultracapacitors with high energy density and stability" (2020) https://www.sciencedirect.com/science/article/pii/S2352152X1...
Years ago, I remember reading about supercapacitor electrodes made from what would be waste hemp bast fiber. They used graphene as a control. And IIRC, the natural branching structure in hemp (the strongest natural fiber) was considered ideal for an electrode.
"Hemp Carbon Makes Supercapacitors Superfast" https://www.asme.org/topics-resources/content/hemp-carbon-ma...
How do the costs and performance compare? Graphene, hemp, durian, jackfruit
While graphene production costs have fallen due to lots of recent research, IIUC all graphene production is hazardous due to graphene's ability to cross the lungs and the blood-brain barrier?
All my life, I've heard people go on and on about the magical properties of hemp.
It's hilarious.
They have these material science level arguments about hemp, yet the same people would be hard pressed to find an alternative use for cotton aside from clothes, or wood pulp aside from ikea furniture.
It's a desert topping, it's a driveway sealant. And the government won't let us have it!
Hemp textiles are rough, but antimicrobial/antibacterial: hemp textiles resist growth of pneumonia and staph.
AFAIU, when they blend hemp with e.g. rayon it's good enough for underwear, sheets, scrubs.
The government is getting the heck out of the way of hemp, a great rotation crop that can be used for soul remediation.
damn if hemp doesn't make a come back.. from cbd to ultracaps..
Hopefully but it’s only been legal to grow hemp in the US since 2018. https://www.vox.com/the-goods/2018/12/13/18139678/cbd-indust...
Technically, the 2013 farm bill (signed into law in 2014) authorized growing hemp for state-registered research purposes. https://www.votehemp.com/laws-and-legislation/federal-legisl...
Turns out UC Berkeley's got an approach for brewing cannabinoids (and I think terpenes) from yeast, and a company in Germany has a provisional patent application to brew cannabinoids from bacteria. We could be absorbing carbon ("sequestering" carbon) and coal ash acid rain with mostly fields of industrial hemp for which there are indeed thousands of uses.
Ask HN: How to Take Good Notes?
I want to improve my note-taking skill. I've started writing a text file with notes from class, however, I don't have a systematic way of writing. This means at this point I just wrote down, arbitrarily, things the professor said, things the professor wrote, how I understood the information, and everything else, mostly all over the place.
I'm wondering if anyone developed a system like this I could adapt to myself, and how did they do it.
I think you ought to consider whether you should take notes at all. Notetaking is great for remembering actions that you have comitted to doing, or if you need to spread information to people who didn't participate in a meeting. Managers need to do a lot of notetaking.
However, taking notes seriously hinder your ability to engage with the material and build true understanding as you are listening, which would have helped you remember the material right away. If you are in school or are an individual contributor in a company I think you ought to stop taking notes all together.
If you need notes for future practice I would advice you to write them after the meeting/lecture. Actively recalling things from memory is the best form practice.
> In 2009, psychologist Jackie Andrade asked 40 people to monitor a 2-½ minute dull and rambling voice mail message. Half of the group doodled while they did this (they shaded in a shape), and the other half did not. They were not aware that their memories would be tested after the call. Surprisingly, when both groups were asked to recall details from the call, those that doodled were better at paying attention to the message and recalling the details. They recalled 29% more information! https://www.health.harvard.edu/blog/the-thinking-benefits-of...
https://en.wikipedia.org/wiki/Doodle#Effects_on_memory references the same study.
Related articles on GScholar: https://scholar.google.com/scholar?q=related:YVG_-PKhNH4J:sc...
> dull and rambling voice mail message
This is in no way a realistic study. A dull and rambling voice speaking about some random thing not related to you or your work will make people disengage. Doodling presumably keeps people from totally spacing out.
Many lectures and meetings may be experienced as similarly dross and irrelevant and a waste of time (though you can't expect people to just read the necessary information ahead of time, as flipped classrooms expect of committed learners).
What would be a better experimental design for measuring effect on memory retention of passively-absorbed lectures?
Ask HN: STEM toy for a 3 years old?
Hello! Can the HN community recommend me a STEM toy (or similar that would educate and entertain him) for my 3 yo boy? He's highly curious but I can't find many things to play with him :( The things that I like bore him and the things that he likes bore me (or are way too messy and dangerous to let him do them)...
"12 Awesome (& Educational) STEM Subscription Boxes for Kids" https://stemeducationguide.com/subscription-boxes-for-kids/
Tape measure with big numbers, ruler(s)
Measuring cup, water, ice.
"Melissa & Doug Sunny Patch Dilly Dally Tootle Turtle Target Game (Active Play & Outdoor, Two Color Self-Sticking Bean Bags, Great Gift for Girls and Boys - Best for 3, 4, 5, and 6 Year Olds)"
Set of wooden blocks in a wood box; such as "Melissa & Doug Standard Unit Blocks"
...
https://sugarlabs.org/ , GCompris mouse and keyboard games with a trackpad and a mouse, ABCMouse, Khan Academy Kids, Code.org, ScratchJr (5-7), K12 Computer Science Framework https://k12cs.org/
OpenAPI v3.1 and JSON Schema 2019-09
In all honesty, isn't json schema and open api more or less reinventing wsdl/soap/rpc? It feels like we've come full circle now and replaced <> with {} (and on the way lost a lot of mature xml tooling).
Defusedxml lists a number of XML "Attack vectors: billion laughs / exponential entity expansion, quadratic blowup entity expansion, external entity expansion (remote), external entity expansion (local file), DTD retrieval" https://pypi.org/project/defusedxml/
Are there similar vulnerabilities in JSON parsers (that would then also need to be monkeypatched)?
Sure, but the page itself says, that it's mostly based on some uncommon features. Since we are converging to the feature list of XML/WSDL with JSON, what makes you think that some JSON (i'm using JSON as placeholder for the next big thing in the JSON space) parsing libraries won't have bugs? Isn't that the danger of the added complexity?
So, most likely some of those uncommon features won't be added to JSON but why didn't "we" as a community then not "just" invent "XML light" or disable those dangerous but sometimes used features behind better defaults?
So, over the years all i've seen is some mumble jumble over how XML is too complicated and too noisy and big, atleast i used to say that, but what if that complexity came from decades of usage, from real requirements? Why wouldn't we arrive in the same situation 10 or 20 years from now? And if size matters so much, why do we ship megabytes of code for simple websites? And why didn't we choose protobuffers over JSON?
And what makes you think that the next generation won't think that JSON is too complicated (because naturally, complexity will increase with new requirements and features). So, i suppose we'll see a JSON replacement with different brackets ("TOML Schema"?) in the future. Or maybe it's time for a binary representation again, just to be replaced with the next "editor friendly and humanly readable" format.
It all feels like, we as an industry took a step backward for years (instead of improving the given tools and specs) just to head to the same level of complexity all the while throwing away lots and lots of mature tooling of XML processing.
P.S.: Same goes probably for ASN.1 vs. XML Schemas... So now we are in the 3rd generation of schema based data representation: ASN.1 -> XML -> JSON..
P.P.S.: I'm not arguing that we all should use XML now, i'm reflecting over the historical course and what might be ahead of us. Clearly, XML is declining steadily and has "lost".
Yeah, expressing complex types with only primitives in a portable way is still unfortunately a challenge. For example, how do we encode a datetime without ISO8601 and a schema definition; or, how do we encode complex numbers with units like "1j+1"; or "1.01 USD/meter"?
Fortunately, we can use XSD with RDFS and RDFS with JSON-LD.
LDP: Linked Data Platform and Solid: social linked data (and JSON-LD) are the newer W3C specs for HTTP APIs.
For one, pagination is a native feature of LDP.
JSON-LD is just the same old RDF data model that the W3C promotes unsuccessfully for that last 2 decades, repackaged with trendy JSON syntax. It's almost as unreadable as the XML serialization and still put very little value on the table. It shines however with people who like to do over-engineering instead of delivering a product.
It's astounding how often people make claims like this.
There is a whole lot of RDF Linked Data; and it links together without needing ad-hoc implementations of schema-specific relations.
I'll just link to the Linked Open Data Cloud again, for yet another hater that's probably never done anything for data interoperability: https://lod-cloud.net/
That's a success.
the only thing I miss are the comments
JSON5 supports comments: https://json5.org/
But who supports json5?
You can use json5 for comments and compile it to json.
You can use (yaml, hjson, jsonnet) for the same purpose. What sets json 5 apart?
YAML, hjson and jsonnet do not validate existing json blobs. You can "use" ROT13 for the same purpose, but it's incidental in what you are trying to do.
I would plead that everyone stop using YAML. It's terrible at everything.
jsonnet is a template language for a serialization format. Who would choose that nightmare?
Ecmascript isn't big on extending the language orthogonally, so hjson is eventually going to be superceded by a ES*.
This is a niche concern that has an optimal path. Go with a validation schema designed for applying to a serialization format, which has widespread library support.
>YAML ... do not validate existing json blobs
YAML 1.2 is a strict superset of JSON. What do you mean by "validate" here?
>I would plead that everyone stop using YAML. It's terrible at everything.
Fair enough.
>jsonnet is a template language for a serialization format. Who would choose that nightmare?
People who are tired of Jinja2 and YAML but don't want to jump to a general purpose programming language?
>widespread library support
JSON5 is implemented in Javascript, Go, and C#. Not sure how "widespread" that is. Rust, C, Python, Lua, Java, and Haskell are missing out on the fun.
Git for Node.js and the browser using libgit2 compiled to WebAssembly
This looks useful. Are there pending standards for other browser storage mechanisms than an in-memory FS?
Would it be a security risk to grant limited local filesystem access by domain; with a storage quota?
... To answer my own question, it looks like the FileSystem API is still experimental and only browser extensions can request access to the actual filesystem: https://developer.mozilla.org/en-US/docs/Web/API/FileSystem
Actually, looks like emscripten has support for a few different options, including IndexedDB[1]. I have a use case where we'd like to both use the local filesystem via Node APIs as well as in-memory in the browser. I asked the author of wasm-git and it looks like this is possible with a custom build[2].
[1]: https://emscripten.org/docs/api_reference/Filesystem-API.htm...
Scientists use ML to find an antibiotic able to kill superbugs in mice
> That is an especially pressing challenge in the development of new antibiotics, because a lack of economic incentives has caused pharmaceutical companies to pull back from the search for badly needed treatments. Each year in the U.S., drug-resistant bacteria and fungi cause more than 2.8 million infections and 35,000 deaths, with more than a third of fatalities attributable to C. diff, according to the the Centers for Disease Control and Prevention.
How big does the market have to be to commercially viable for research and development? Nearly 3M potential patients at a couple hundred dollars per course is nearing a $1B/year.
The second-order costs avoided by treatments developed so innovatingly could be included in a "value to society" estimation.
"Acknowledgements" lists the grant funders for this federally-funded open access study.
"A Deep Learning Approach to Antibiotic Discovery" (2020) https://doi.org/10.1016/j.cell.2020.01.021
> Mutant generation
> Chemprop code is available at: https://github.com/swansonk14/chemprop
> Message Passing Neural Networks for Molecule Property Prediction
> A web-based version of the antibiotic prediction model described herein is available at: http://chemprop.csail.mit.edu/
> This website can be used to predict molecular properties using a Message Passing Neural Network (MPNN). In order to make predictions, an MPNN first needs to be trained on a dataset containing molecules along with known property values for each molecule. Once the MPNN is trained, it can be used to predict those same properties on any new molecules.
Shit – An implementation of Git using POSIX shell
Hiya HN. I was ranting on Mastodon earlier today because I feel like people learn git the wrong way - from the outside in, instead of the inside out. I reasoned that git internals are pretty simple and easy to understand, and that the supposedly obtuse interface makes a lot more sense when you approach it with an understanding of the fundamentals in hand. I said that the internals were so simple that you could implement a workable version of git using only shell scripts inside of an afternoon. So I wrapped up what I was working on and set out to prove it.
Five hours later, it had turned into less of a simple explanation of "look how simple these primitives are, we can create them with only a dozen lines of shell scripting!" and more into "oh fuck, I didn't realize that the git index is a binary file format". Then it became a personal challenge to try and make it work anyway, despite POSIX shell scripts clearly being totally unsuitable for manipulating that kind of data.
Anyway, this is awful, don't use it for anything, don't read the code, don't look at it, just don't.
> the supposedly obtuse interface makes a lot more sense when you approach it with an understanding of the fundamentals in hand.
Agreed. I always said the best git tutorial is https://www.sbf5.com/~cduan/technical/git/
> The conclusion I draw from this is that you can only really use Git if you understand how Git works. Merely memorizing which commands you should run at what times will work in the short run, but it’s only a matter of time before you get stuck or, worse, break something.
I never really understood Git until I read this tutorial: https://github.com/susam/gitpr
Things began to click for me as soon as I read this in its intro section:
> Beginners to this workflow should always remember that a Git branch is not a container of commits, but rather a lightweight moving pointer that points to a commit in the commit history.
A---B---C
↑
(master)
A---B---C---D
↑
(master)
This "moving pointer" model of Git branches led me to instant enlightenment. Now I can apply this model to other complicated operations too like conflict resolution during rebase, interactive rebase, force pushes, etc.
If I had to select a single most important concept in Git, I would say it is this: "A branch is merely a pointer to the tip of a series of commits."
And you can see this structure if you add to any "git log" command "--graph --oneline --decorate --color". IIRC some of those are unnecessary in recent versions of git, I just remember needing all of them at the point I started using it regularly.
I have a bash function for it (with a ton of other customizations, but it boils down to this):
function pwlog() {
git log "$@" --graph --oneline --decorate --color | less -SEXIER
}
pwlog --all -20
You can also set $GIT_PAGER/core.pager/$PAGER and create an alias to accomplish this:
#export PAGER='less -SEXIER'
#export GIT_PAGER='less -SEXIER'
git config --global core.pager 'less -SEXIER'
git config --global alias.l 'log --graph --oneline --decorate --color'
# git diff ~/.gitconfig
git l
> The order of preference is the $GIT_PAGER environment variable, then core.pager configuration, then $PAGER, and then the default chosen at compile time (usually less).
HTTP 402: Payment Required
The RFC offers no reasoning for this besides
> The 402 (Payment Required) status code is reserved for future use.
Isn't this obviously domain-specific and something that should not be part of the transfer protocol?
> The new W3C Payment Request API [4] makes it easy for browsers to offer a standard (and probably(?) already accessible) interface for the payment data entry screen, at least. https://www.w3.org/TR/payment-request/
Salesforce Sustainability Cloud Becomes Generally Available
> - Reduce emissions with trusted analytics from a trusted platform. Analyzing carbon emissions from energy usage and company travel can be daunting and time-consuming. But with all your data flowing directly onto one platform, you can efficiently quantify your carbon footprint. Formulate a climate action plan for your company from a single source of truth, built on our trusted and secure data platform.
> - Take action with data-driven insights. Prove to customers, employees, and potential investors your commitment to carbon-conscious and sustainable practices. Offer regulatory agencies a clear snapshot of your energy usage patterns. Extrapolate energy consumption and track carbon emissions with cutting-edge analytics — and take action.
> - Tackle carbon accounting audits in weeks instead of months. Carbon analysis can be an overwhelming time commitment, even a barrier to action for companies that want to get it right. Use preloaded datasets from the U.S. EPA, IPCC, and others to accurately assess your carbon accounting. Streamline your data gathering and climate action plan with embedded guides and user flows.
> - Empower decision makers with executive-ready dashboard data. Evaluate corporate environmental impact with rich data visualization and dashboards. Track energy patterns and emission trends, then make the business case to executives. Once an organization understands its carbon footprint, decision makers can begin to drive sustainability solutions.
Are there similar services for Sustainability Reporting and accountability? https://en.wikipedia.org/wiki/Sustainability_reporting
Httpx: A next-generation HTTP client for Python
> Fully type annotated.
This is a huge win compare to requests. AFAIKT requests is too flexible (read: easier to misuse) and difficult to add type annotation now. The author of requests gave a horrible type annotation example here [0].
IMO at this time when you evaluate a new Python library before adopting, "having type annotation" should be as important as "having decent unit test coverage".
The huge win against requests is that Https is fully async.. you can download 20 files in parallel without not too much effort. Throughout in python asyncio is amazing, something similar to node or perhaps better... That's the main point.
FWIW, requests3 has "Type-annotations for all public-facing APIs", asyncio, HTTP/2, connection pooling, timeouts, etc https://github.com/kennethreitz/requests3
Sorry, have you checked the source? Are these features there or only announced? Has requests added a timeout by default finally?
It looks like requests is now owned by PSF. https://github.com/psf/requests
But IDK why requests3 wasn't transferred as well, and why issues appear to be disabled on the repo now.
The docs reference a timeout arg (that appears to default to the socket default timeout) for connect and/or read https://3.python-requests.org/user/advanced/#timeouts
And the tests reference a timeout argument. If that doesn't work, I wonder how much work it would be to send a PR (instead of just talking s to Ken and not contributing any code)
>But IDK why requests3 wasn't transferred as well, and
That's thing... Who knows..
TIL requests3 beta works with httpx as a backend: https://github.com/not-kennethreitz/team/issues/21#issuecomm...
If requests3 is installed, `import requests' imports requests3
BlackRock CEO: Climate Crisis Will Reshape Finance
+1. From the letter: https://www.blackrock.com/us/individual/larry-fink-ceo-lette...
> The money we manage is not our own. It belongs to people in dozens of countries trying to finance long-term goals like retirement. And we have a deep responsibility to these institutions and individuals – who are shareholders in your company and thousands of others – to promote long-term value.
> Climate change has become a defining factor in companies’ long-term prospects. Last September, when millions of people took to the streets to demand action on climate change, many of them emphasized the significant and lasting impact that it will have on economic growth and prosperity – a risk that markets to date have been slower to reflect. But awareness is rapidly changing, and I believe we are on the edge of a fundamental reshaping of finance.
> The evidence on climate risk is compelling investors to reassess core assumptions about modern finance. Research from a wide range of organizations – including the UN’s Intergovernmental Panel on Climate Change, the BlackRock Investment Institute, and many others, including new studies from McKinsey on the socioeconomic implications of physical climate risk – is deepening our understanding of how climate risk will impact both our physical world and the global system that finances economic growth.
Environmental, social and corporate governance > Responsible investment: https://en.wikipedia.org/wiki/Environmental,_social_and_corp...
Corporate social responsibility: https://en.wikipedia.org/wiki/Corporate_social_responsibilit...
UN-supported PRI: Principles for Responsible Investment (2,350 signatories (2019-04)) https://en.wikipedia.org/wiki/Principles_for_Responsible_Inv...
A lot of complex “scalable” systems can be done with a simple, single C++ server
Many developers severely underestimate how much workload can be served by a single modern server and high-quality C++ systems code. I've scaled distributed workloads 10x by moving them to a single server and a different software architecture more suited for scale-up, dramatically reducing system complexity as a bonus. The number of compute workloads I see that actually need scale-out is vanishingly small even in industries known for their data intensity. You can often serve millions of requests per second from a single server even when most of your data model resides on disk.
We've become so accustomed to extremely inefficient software systems that we've lost all perspective on what is possible.
Can you expand on this? I have some pretty massive compute loads that need to be scaled onto a cluster with 100+ workers for most computations. This is after I use a library called dask that graphically does its own mapreduce optimisation inside its modules. This is all for a relatively small 250GB raw data file that I keep in a csv (and need to convert to SQL at some point).
Are you saying this can be optimised to fit inside a single 10 core server in terms of compute loads?
Don't know why you're being downvoted but I'll assume your question is genuine.
You use a cluster when your data and compute requirements are large and parallel enough that the tax paid on network latency trumps the 10-20X speedup you get on SSD and 1000X speedup you get from just keeping data in RAM.
250 Gigs is tiny enough that you could probably much get better performance running on high memory instance in AWS or GCP. You'll generally have to write your own multiprocessing code though which is fairly simple - your existing library may also be able to support it.
I once actually ran this kind of workload on just my laptop using a compiled language that performed better than pyspark on a cluster.
I'd love to keep it in RAM if I could. The problem is, the library I'm familiar with (pandas) typically seems to take more memory than the original csv file once it loads it onto memory. I know this is due to bad data types but in certain cases, I cannot get around those.
However, even if I could load it all into memory at once, and assuming it takes 200 gb, I'm still using a master's student access to a cluster. So I get preempted like it's nobody's business. Hence why I prefer a smaller memory footprint even if I take up cpus at variable rates through a single execution.
I did try to write my own multiprocessing code for this, but the operations are sometimes too complicated (like groupby) for me to rewrite everything from the ground up. If I'm not reliant on serial data communication between processes (like you'd need to sort a column), I can get it done pretty easily. In fact, I wrote my data cleaning code with this and cleaned up the entire file in half an hour because single chunks didn't rely on others.
However, if you have some idea of how to run these computational loads in parallel in python or any other language on single compute instances (like the size of a laptop's memory of 16 gb), I'd really love to see it. Thanks.
Numpy supports memory mapping `ndarrays` which can back a DataFrame in pandas. This lets you access a dataset far larger than will fit in RAM as if it lived in RAM. Provided it's on fast SSD storage you'll have speedy access to the data and can process huge chunks at once.
Can you provide a link to this please? My current knowledge is that all numpy data lives in memory, and pandas itself has a feature to fragment any data into iterables so I can read upto my memory limit. I cannot use this feature due to the serial nature of some of the operations that I alluded to (I'd have to almost rewrite the entire library for some of these complicated operations like groupby and sorting).
I do have fast SSD storage because it's on the scratch drive of a cluster and from what I've seen it can do ~300-400 MB/s easily. I haven't had a chance to test more than that since I'm mostly memory constrained in much of my testing.
My current attempt is to push this data into a pure database handling system like SQL so that I can query it. But like I said, I work with a less-than-stellar set of tools and I have to literally set up a postgres server from ground up to write to it. Which shouldn't be a big deal except when it's on a non-root user and I have to keep remapping dependencies (took 5-6 hours to set it up on the instance I have access to).
My other option was to write the entire 250 GB to a sqllite database using the sqlalchemy library in Python, but that seems to fail whether I do it with parallel writes, or serial writes. In both cases, it fails after I create ~64-70 tables.
Dask groupby example: https://examples.dask.org/dataframes/02-groupby.html
> Generally speaking, Dask.dataframe groupby-aggregations are roughly same performance as Pandas groupby-aggregations, just more scalable.
The dask.distributed scheduler can also run on one high-RAM instance (with threads or processes) https://docs.dask.org/en/latest/setup.html
Pandas docs > Ecosystem > Out-of-core: https://pandas.pydata.org/pandas-docs/stable/ecosystem.html#...
Reading from Parquet into Apache Arrow is much faster than CSV because the data can just be directly loaded into RAM. https://ursalabs.org/blog/2019-10-columnar-perf/
If you have GPU instances, cuDF has a Pandas-like API on top of Apache Arrow. https://github.com/rapidsai/cudf
> Built based on the Apache Arrow columnar memory format, cuDF is a GPU DataFrame library for loading, joining, aggregating, filtering, and otherwise manipulating data.
> cuDF provides a pandas-like API that will be familiar to data engineers & data scientists, so they can use it to easily accelerate their workflows without going into the details of CUDA programming.
Dask-ML makes scalable scikit-learn, XGBoost, TensorFlow really easy. https://dask-ml.readthedocs.io/en/latest/
... re: the OT: While it's possible to write C++ code that's really fast, it's generally inflexible, expensive to develop, and dangerous for devs with experience in their respective domains of experience to write. Much saner to put a Python API on top and optimize that during compilation.
There are a few C++ frameworks in the top quartile of the TechEmpower framework benchmarks. https://www.techempower.com/benchmarks/
Hardware/hosting is relatively cheap. Developers and memory vulnerabilities aren't.
Unfortunately, I'm not sure what's wrong with dask, but it doesn't work properly on my cluster. I tested it on an exceedingly simple operation - find all unique values in a very big column (5 billion rows, but I know for a fact that there are only 500-502 unique values in there). With a 100 workers, it still failed. Now this is an embarrassingly parallel operation that can be implemented trivially. So I'm not sure if there's a problem with my cluster or if dask just does not work with slurm clusters very well.
https://docs.dask.org/en/latest/setup/hpc.html says dask-jobqueue handles "PBS, SLURM, LSF, SGE and other resource managers"
"Dask on HPC, what works and what doesn't" https://github.com/dask/dask-blog/issues/5
Maybe you should spend some time developing a job visualization system for end users from scratch, for end users with lots of C, JS, and HTML experience https://jobqueue.dask.org/en/latest/interactive.html
Yes, I think the library still isn't ready for non HPC experts to use without tinkering. I don't have that level of expertise. I'm a data person who can handle working tools.
Warren Buffett is spending billions to make Iowa 'the Saudi Arabia of wind'
It's both cost-rational and environment-rational to invest heavily in clean energy (with or without the comparatively paltry tax incentives).
The long-term costs of climate change and inaction are unfortunately still mostly external costs to energy producers. We should expect that to change as we start developing competencies in evaluating the costs and frequency of weather disasters exacerbated by anthropogenic climate change. We all get to pay for floods, fires, tornados, hurricanes, landslides, blizzards, and the gosh darn heat.
Insurance firms clearly see these costs. Our military sees the costs of responding to natural disasters. Local economies see the costs of months and years spent on disaster relief; on just getting back up to speed so that they can generate profit from selling goods and services (and pay taxes to support disaster relief efforts essential to operational readiness).
The cost per kilowatt hour of wind (and solar) energy is now lower than operating existing dirty energy plants that dump soot on our crops, air, and water.
With wind, they talk about the "alligator curve". With solar, it's the "duck curve". Grid-scale energy storage is necessary for reaching 100% renewable energy as soon as possible.
Iowa's renewable energy tax incentives are logically aligned with international long-term goals:
UN Sustainable Development Goal 7: Affordable and Clean Energy https://www.globalgoals.org/7-affordable-and-clean-energy
Goal 13: Climate Action https://www.globalgoals.org/13-climate-action
SDG Target 12.6: "Encourage companies to adopt sustainable practices and sustainability reporting" (CSR; e.g. GRI Sustainability Reporting Standards that we can score portfolios with)
https://www.undp.org/content/undp/en/home/sustainable-develo... :
> Rationalize inefficient fossil-fuel subsidies that encourage wasteful consumption by removing market distortions, in accordance with national circumstances, including by restructuring taxation and phasing out those harmful subsidies, where they exist, to reflect their environmental impacts, taking fully into account the specific needs and conditions of developing countries and minimizing the possible adverse impacts on their development in a manner that protects the poor and the affected communities
...
> Thanks. How can I say "try and only run this [computational workload] in zones with 100% PPA offsets or 100% directly sourced #CleanEnergy"? #Goal7 #Goal11 #Goal12 #Goal13 #GlobalGoals #SDGs
It makes good business sense to invest in clean energy to take advantage of tax incentives, minimize future costs to other business units (e.g. insurance, taxes), and earn the support of investors choosing portfolios with long term environmental (and thus economic) sustainability as a primary objective.
Scientists Likely Found Way to Grow New Teeth for Patients
"Scientists Have Discovered a Drug That Fixes Cavities and Regrows Teeth" https://futurism.com/neoscope/scientists-have-discovered-thi...
Tideglusib https://en.wikipedia.org/wiki/Tideglusib
> Tooth repair mechanisms that promotes dentine reinforcement of a sponge structure until the sponge biodegrades, leaving a solid dentine structure. In 2016, the results of animal studies were reported in which 0.14 mm holes in mouse teeth were permanently filled.
Very interesting mechanism.
GSK3 inhibitors are interesting, but we don't understand much the subtypes that are likely to exist: "GSK-3 appears to both promote and inhibit apoptosis, and this regulation varies depending on the specific molecular and cellular context."
Still, I believe better GSK3i will find a role in autologous bone-marrow grafting therapies to fight senescence: with TERC/TERT overexpression (the opposite of DeGrey WILT ideas) to send new stem cells to the tissues - just like how you can find the donors chromosomes in most tissues of grafted patient.
Announcing the New PubMed
I'm actually part of the team developing the new PubMed. Very curious and interested to know what the hacker news community thinks and feels about their experience. https://pubmed.gov/labs
This looks great: I like the search timeline, the ability to easily search for free full-text meta-analyses (a selection bias we should all be aware of), the MeSH term listing in a reasonably-sized font, and that there's schema.org/ImageObject metadata within the page, but there's no [Medical]ScholarlyArticle metadata?
I've worked with Google Scholar (:o) [1], Semantic Scholar (Allen Institute for AI) [2], Meta (Chan Zuckerberg Institute) [3], Zotero, Mendeley and a number of other tools for indexing and extracting metadata and graph relations from https://schema.org/ScholarlyArticle and MedicalScholarlyArticles . Without RDFa (or Microdata, or JSON-LD) in PDF, there's a lot of parsing that has to go down in order to get a graph from the citations in the article. Each service adds value to this graph of resources. Pushing forward on publishing linked research that's reproducible (#LinkedResearch, #LinkedReproducibility) is a worthwhile investment in meta-research that we have barely yet addressed:
> http://Schema.org/NewsArticle .citation: https://schema.org/citation ... Wouldn't it be great if NewsArticles linked to the ScholarlyArticle and/or Notebook CreativeWorks that they're .about (with reified relations)?
> A practical use case: Alice wants to publish a ScholarlyArticle [1] (in HTML with structured data, as a PDF) predicated upon Datasets [2] (as CSV, CSVW JSONLD, XLSX (DataDownload)) with static HTML (and no special HTTP headers). 1 https://schema.org/ScholarlyArticle 2 https://schema.org/Dataset*
> B wants to build a meta analysis: to collect a # of ScholarlyArticles and Dataset DataDownloads; review study controls and data; merge, join, & concatenate Datasets if appropriate, and inductively or deductively infer a conclusion and suggestions for further studies of variance*
The Linked Open Data Cloud shows the edges, the relations, the structured data links between very many (life sciences) datasets: https://lod-cloud.net/ . https://5stardata.info/en/ lists TimBL's suggested 5-start deployment schema for Open Data; which culuminates in publishing linked open data in non-proprietary formats that uses URIs to describe and link to things.
Could any of these [1][2][3][4][5] services cross-link the described resources, given a common URI identifier such as https://schema.org/identifier and/or https://schema.org/url ? ORCID is a service for generating stable identifiers for researchers and publishers who have names in common but different emails. W3C DID solves for this need in a different way.
When I check an article result page with the OpenLink OSDS extension (or any of a number of other tools for extracting structured data from HTML pages (and documents!) https://github.com/CodeForAntarctica/codeforantarctica.githu... ), there could be quite a bit more data there for search engines, browser extensions, and meta-research tools.
Is this something like ElasticSearch on the backend? It is possible to store JSON-LD documents in the search index. I threw together elasticsearchjsonld to "Generate JSON-LD @contexts from ElasticSearch JSON Mappings" for the OpenFDA FAERS data a few year ago. That's not GraphQL or SPARQL, but it's something and it's Linked Data.
re: "Canada's Decision To Make Public More Clinical Trial Data Puts Pressure On FDA" https://news.ycombinator.com/item?id=21232183
> We really could get more out of this data through international collaboration and through linked data (e.g. URIs for columns). See: "Open, and Linked, FDA data" https://github.com/FDA/openfda/issues/5#issuecomment-5392966... and "ENH: Adverse Event Count / 'Use' Count Heatmap" https://github.com/FDA/openfda/issues/49 . With sales/usage counts, we'd have a denominator with which we could calculate relative hazard.
W3C Web Annotations handle threaded comments and highlights; reviewing the reviewers is left as an exercise for the reader. Does Zotero still make it easy to save the bibliographic metadata for one or more ScholarlyArticles from PubMed to a collection in the cloud (and add metadata/annotations)?
Sorry to toot my own horn here. Great job on this. This opens up many new opportunities for research.
[1] https://scholar.google.com
[2] https://www.semanticscholar.org/
Pubmed publishes its dataset for download. Its rather large but update files come frequently. Its amazing. I beleive NIH adds the MESH terms.
ftp://ftp.ncbi.nlm.nih.gov/pubmed/
We had someone do a project with it. downloaded the dataset and used it and create a tool to do some searches that we found useful to find colaborators: (last author, working on a specific gene, paper counts, most recent).
Searching by Mesh Terms across species, and search with orthologs.
The dataset sometimes has a hard time disambiguating names (I think the european dataset assigns Ids to names)
To make sure your feedback is heard, please use "Feedback button" found in bottom right corner of https://pubmed.gov/labs
Ask HN: Is it worth it to learn C in 2020?
The GNU/Linux kernel, FreeBSD kernel, Windows kernel, MacOS kernel, Python, Ruby, Perl, PHP, NodeJS, and NumPy are all written in C. If you want to review and contribute code, you'd need to learn C.
There are a number of coding guidelines e.g. for safety-critical systems where bounded running time and resource consumption are essential. These coding guidelines and standards are basically only available for C, C++, and Ada. https://github.com/stanislaw/awesome-safety-critical/blob/ma...
Even though modern languages have garbage-collection that runs whenever it feels like it, It's helpful to learn about memory management in C (or C++). You'll appreciate object destructor methods that free memory and sockets and file handles that much more. Reference cycles in object graphs are easier to handle with modern C++ than with C. Are there RAII (Resource Acquisition is Initialization) "smart pointers" that track reference counts in C?
Without OO namespacing, in C, function names are often prefixed with namespaces. How many ways could a struct be initialized? When can I free that memory?
When strace prints a syscall, what is that?
Is it necessary to learn C? Somebody needs to maintain and improve the C-based foundation for most of our OSs and very many of our fancy scripting languages. C can be very unforgiving: it's really easy to do it wrong, and there's a lot to keep in mind at once: the cognitive burden is higher with C (and then still with ASM and WebASM) than with an interpreted (or compiled) duck-typed 3GL scripting language with first-class functions.
What's a good progression that includes syntax, finding and reading the libc docs, Make/CMake/Autotools, secure recommended compiler flags for GCC (CPPFLAGS, CFLAGS, LDFLAGS) and LLVM Clang?
C: https://learnxinyminutes.com/docs/c/
C++: https://learnxinyminutes.com/docs/c++/
Links to the docs for Libc and other tools: https://westurner.github.io/tools/#libc
xeus-cling is a Jupyter kernel for C++ (and most of C) that works with nbgrader. https://github.com/QuantStack/xeus-cling
What's a better unit-testing library for C/C++ than gtest? https://github.com/google/googletest/
Amazing comment, thank you for your detailed input and the awesome resources. I want to get into C because I like network and system programming but lack the underlying knowledge and experience. These links will help me start on that new path. Very much appreciated sir :)
For network programming, you might consider asynchronous programming with coroutines. C++20 has them and they're already supported in LLVM. For C, there are a number of implementations of coroutines: https://en.wikipedia.org/wiki/Coroutine#Implementations_for_...
> Once a second call stack has been obtained with one of the methods listed above, the setjmp and longjmp functions in the standard C library can then be used to implement the switches between coroutines. These functions save and restore, respectively, the stack pointer, program counter, callee-saved registers, and any other internal state as required by the ABI, such that returning to a coroutine after having yielded restores all the state that would be restored upon returning from a function call. Minimalist implementations, which do not piggyback off the setjmp and longjmp functions, may achieve the same result via a small block of inline assembly which swaps merely the stack pointer and program counter, and clobbers all other registers. This can be significantly faster, as setjmp and longjmp must conservatively store all registers which may be in use according to the ABI, whereas the clobber method allows the compiler to store (by spilling to the stack) only what it knows is actually in use.
CPython's asyncio implementation (originally codenamed 'tulip') is written in C and IMHO much easier to use than callbacks like Twisted and JS before Promises and the inclusion of tulip-like async/await keywords in ECMAscript. Uvloop - based on libuv, like Node - is apparently the fastest asyncio event loop. CPython Asyncio C module source: https://github.com/python/cpython/blob/master/Modules/_async... Asyncio docs: https://docs.python.org/3/library/asyncio.html
(When things like file or network I/O are I/O bound, the program can yield to allow other asynchronous coroutines ('async') to run on that core. With network programming, we're typically waiting for things to send or reply.)
Return-oriented-programming > Return-into-library technique is an interesting read regarding system programming :) https://en.wikipedia.org/wiki/Return-oriented_programming#Re...
Free and Open-Source Mathematics Textbooks
This is a good list of books. Unfortunately many of the links are broken? Probably just my luck, but the first few "with Sage" books I excitedly selected unfortunately 404'd. I'll send an email.
> Moreover, the American Institute of Mathematics maintains a list of approved open-source textbooks. https://aimath.org/textbooks/approved-textbooks/
I also like the (free) Green Tea Press books: Think Stats, Think Bayes, Think DSP, Think Complexity, Modeling and Simulation in Python, Think Python 2e: How To Think Like a Computer Scientist https://greenteapress.com/wp/
And IDK how many times I've recommended the book for the OCW "Mathematics for Computer Science" course: https://ocw.mit.edu/courses/electrical-engineering-and-compu...
There may be a newer edition than the 2017 version of the book: https://courses.csail.mit.edu/6.042/spring17/mcs.pdf
Does anyone have a good strategy for archiving and downloading the textbooks from aimath.org? They are all excellently formatted in HTML, but I am not certain what the best way to get the complete book would be.
When I have books in form of webpages I normally write a small crawler in python, extract the text div with beautifulsoup, add <hn> tags for chapter names and throw them all together in html form. Add a cover image and combine everything with pandoc.
Nothing fancy but works reliable in an automated fashion
I've found MIT's "Mathematics for Computer Science" actually hard to read, and in my opinion it cannot really be called a proper book, more like a collection of notes, as it skips over many details, jumping to conclusions, definitions, results, without giving you an intuitive explanation. In that regards it's actually hard to follow.
It's fast-going. You may want to start with something more introductory, like:
Levin http://discrete.openmathbooks.org/dmoi3.html
Hammack https://www.people.vcu.edu/~rhammack/BookOfProof/
the early parts of Newstead https://infinitedescent.xyz/
Make CPython segfault in 5 lines of code
Applications Are Now Open for YC Startup School – Starts in January
I'm presently living in a fairly rural part of the US due to an illness in the family. In the town of 14,000 I currently reside in it's pretty difficult to network in a meaningful way and talk about my company with folks that can give guidance and feedback. Really excited for Startup School and so grateful that YC has put this together.
> In the town of 14,000 I currently reside in it's pretty difficult to network in a meaningful way and talk about my company with folks that can give guidance and feedback
GitLab and Zapier are examples of all remote former YC companies.
"GitLab Handbook" https://about.gitlab.com/handbook/
"The Ultimate Guide to Remote Work: Lessons from a team of over 200 remote workers" https://zapier.com/learn/remote-work/
Were those companies remote when they did YC? I can't recall exactly where offhand but I seem to remember Paul Graham (or was it Sam Altman?) saying that they were strongly against remote teams in the early stages. It was a few years ago though, I wonder if their opinions have changed.
Startup School is now designed as a remote program.
It'd be interesting to hear from them about building all remote team culture with transparency and accountability. Are text-chat "digital stand up meetings" with quality transcripts of each team member's responses to the three questions enough? ( Yesterday / Today and Tomorrow / Obstacles // What did I do since the last time we met? What will I do before the next time we meet? What obstacles are blocking my progress? )
Or are there longer term planning sessions focusing on a plan for delivering value on a far longer term than first getting the MVP down and maximizing marginal profit by minimizing costs?
‘Adulting’ is hard. UC Berkeley has a class for that
+1 for Life Skills for Adulting and also Home Economics including Family and Meal Planning.
A bunch of resources from "Consumer science (a.k.a. home economics) as a college major" https://news.ycombinator.com/item?id=17894632 : CS 007: Personal Finance for Engineers, r/personalfinance/wiki, Healthy Eating Plate, Khan Academy > Science > Health and Medicine
And also, Instant Pot. The Instant Pot pressure cooker is your key to nutrient preservation and ultimate happiness.
Founder came back after 8 years to rewrite flash photoshop in canvas/WebGL
Five cities account for vast majority of growth in U.S. tech jobs: study
> Boston, San Francisco, San Jose, Seattle, and San Diego—accounted for more than 90% of the nation’s innovation-sector growth during the years 2005 to 2017
Why 2005 and why'd it drop off post-2017?
Also surprised DC and NYC aren't on the list. Lots of new companies and growth in those areas, but I guess they may not qualify as "innovation-sector growth".
> To that end, the present paper proposes that Congress assemble and award to a select set of metropolitan areas a major package of federal innovation inputs and supports that would accelerate their innovation-sector scale-up. Along these lines, we envision Congress establishing a rigorous competitive process by which the most promising eight to 10 potential growth centers would receive substantial financial and regulatory support for 10 years to become self-sustaining new innovation centers. Such an initiative would not only bring significant economic opportunity to more parts of the nation, but also boost U.S. competitiveness on the global stage.
"Potential growth centers" sounds promising.
Don’t Blame Tech Bros for the Housing Crisis
If there is demand for housing, we would expect people to be finding land and building housing unless there are policies that prevent this (and/or long commutes that people don't want to suffer) or higher-value opportunities.
If the city wanted residential areas (over commercial tax revenue giants), the city should have zoned residential.
The people elect city leaders. The people all want affordable housing.
With $4.5b from corporations and nowhere to build but out or up, high rise residential is the most likely outcome. (Which is typical for dense urban areas that have prioritized and attracted corporate tax revenue over affordable housing)
... Effing scooter bros with their scooters and their gold rush money and their tiny houses.
[Edit: more than] One company says "I will pay you $10,000 to leave the Bay Area / Silicon Valley" Because there's a lot of tech talent (because universities and opportunities) but ridiculously high expenses.
What an effectual headline from NY.
Docker is just static linking for millenials
No, LXC does quite a bit more than static linking. An inability to recognize that likely has nothing to do with generation.
Can you launch a process in a chroot, with cgroups? Okay, now upgrade everything it's linked with (without breaking the host/build system)
Configure a host-only network for a few processes – running in separate cgroups – without DHCP.
Criticize Docker? Rootless builds and containers are essentially impossible. Buildah and podman make rootless builds possible without a socket. Like sysvinit, though, IDK how well centralized logging (and logshipping, and logged crashes and restarts) works without that socket.
Given comments like this, it's likely that you've never built a chroot for a different distro. Or launchd a process with cgroups.
Show HN: Bamboolib – A GUI for Pandas (Python Data Science)
This looks excellent. The ability to generate the Python code for the pandas dataframe transformations looks to be more useful than OpenRefine, TBH.
How much work would it be to use Dask (and Dask-ML) as a backend?
I see the OneHotEncoder button. Have you considered integration with Yellowbrick? They've probably already implemented a few of your near-future and someday roadmap items involving hyperparameter selection and model selection and visualization? https://www.scikit-yb.org/en/latest/
This video shows more of the advanced bamboolib features: https://youtu.be/I0a58h1OCcg
The live histogram rebinning looks useful. Recently I read about a 'shadowgram' / ~KDE approach with very many possible bin widths translucently overlaid in one chart. https://stats.stackexchange.com/questions/68999/how-to-smear...
Yellowbrick also has a bin width optimization visualization in yellowbrick.target.binning.BalancedBinningReference: https://www.scikit-yb.org/en/latest/api/target/binning.html
Great work.
Thank you for your feedback and support :) Are you currently using OpenRefine?
We are currently thinking about providing other dataframe libraries like dask or pyspark and similar. However, we are a little bit unsure on how to make sure that there is user demand before we implement it. It is not a complete rewrite but it would require some additional abstractions at some points in the library. And we need to check if some features might not be available any more. Would dask support be a reason to buy for you?
Great hint with yellowbrick and yes, we are considering some of those features as well if there is a useful place in the library.
In general, we are also thinking about ways how you can extend the library for yourself so that you can add your own analyses/charts of choice and then they will come up again the right point in time. In case that this is useful.
In the past, I've looked at OpenRefine and Jupyter integration. Once I've learned to do data transformation with pandas and sklearn with code, I'll report back to you.
Pandas-profiling has a number of cool descriptive statistics features as well. https://github.com/pandas-profiling/pandas-profiling
There's a new IterativeImputer in Scikit-learn 0.22 that it'd be cool to see visualizations of. https://twitter.com/TedPetrou/status/1197150813707108352 https://scikit-learn.org/stable/modules/impute.html
A plugin model would be cool; though configuring the container every time wouldn't be fun. Some ideas about how we could create a desktop version of binderhub in order to launch REES-compatible environments on our own resources: https://github.com/westurner/nbhandler/issues/1
Dask has only a subset of Pandas available.
Could you send me a link to the docs where they say which ones are not included in Pandas? Would love to take a closer look at his.
Set difference and/or intersection of dir(pd.DataFrame) and dir(dask.DataFrame) with inspect.getargspec and inspect.doc would be a useful document for either or both projects.
pyfilemods generates a ReStructuredText document with introspected API comparisons. "Identify and compare Python file functions/methods and attributes from os, os.path, shutil, pathlib, and path.py" https://github.com/westurner/pyfilemods
Battery-Electric Heavy-Duty Equipment: It's Sort of Like a Cybertruck
> They’ve created a single platform that can be easily modified to do any number of jobs. For instance, their flagship product, the Dannar 4.00, can accept over 250 attachments from CAT, John Deere, or Bobcat. […] Having interoperability with so many different types of equipment, one platform can easily perform many tasks over the course of a year. This is a huge win for cash strapped municipalities. Why would a company or municipality opt to have a backhoe parked all winter long when it could be doing another job?
Does it have regenerative brakes?
Tools for turning descriptions into diagrams: text-to-picture resources
There's a sphinx extension (sphinxcontrib-sdedit) [1] for sdedit ("Quick Sequence Diagram Editor") which does require Java [2].
CSR: Corporate Social Responsibility
> Proponents argue that corporations increase long-term profits by operating with a CSR perspective, while critics argue that CSR distracts from businesses' economic role.
... The 3 Pillars of Corporate Sustainability: Environmental, Social, Economic https://www.investopedia.com/articles/investing/100515/three...
Three dimensions of sustainability: (Environment (Society (Economy))) https://en.wikipedia.org/wiki/Sustainability#Three_dimension...
What are some of the corporate sustainability reporting standards?
How can I score a candidate portfolio with sustainability metrics in order to impact invest with maximum impact?
> What are some of the corporate sustainability reporting standards?
From https://en.wikipedia.org/wiki/Sustainability_reporting#Initi... :
>> Organizations can improve their sustainability performance by measuring (EthicalQuote (CEQ)), monitoring and reporting on it, helping them have a positive impact on society, the economy, and a sustainable future. The key drivers for the quality of sustainability reports are the guidelines of the Global Reporting Initiative (GRI),[3] (ACCA) award schemes or rankings. The GRI Sustainability Reporting Guidelines enable all organizations worldwide to assess their sustainability performance and disclose the results in a similar way to financial reporting.[4] The largest database of corporate sustainability reports can be found on the website of the United Nations Global Compact initiative.
The GRI (Global Reporting Initiative) Standards are now aligned with the UN Sustainable Development Goals (#GlobalGoals). https://en.wikipedia.org/wiki/Global_Reporting_Initiative
>> In 2017, 63 percent of the largest 100 companies (N100), and 75 percent of the Global Fortune 250 (G250) reported applying the GRI reporting framework.[3]
> How can I score a candidate portfolio with sustainability metrics in order to impact invest with maximum impact?
Does anybody have solutions for this? AFAIU, existing cleantech funds are more hand-picked than screened according to sustainability fundamentals.
GTD Tickler file – a proposal for text file format
Taskwarrior is also built upon the todo.txt format. [1]
Taskw supports various task dates – { due: scheduled: wait: until: recur: } [2]
Taskw supports various named dates like soq/eocq, som/eom (start/end of [current] quarter, start/end of month), tomorrow, later [3]
Taskw recurring tasks (recur:) use the duration syntax: weekly/wk/w, monthly/mo, quarterly/qtr, yearly/yr, … [4]
Pandas has a "date offset" "frequency string" microsyntax that supports business days, quarters, and years; e.g. BQuarterEnd, BQuarterBegin [5]
IDK how usable by other tools these date string parsers are.
W/ just a text editor, having `todo.txt`, `daily.todo.txt`, and `weekly.todo.txt` (and `cleanhome.todo.txt` and `hygiene.todo.txt` with "## heading" tasks that get lost @where +sorting) works okay.
I have physical 43 folders, too: A 12 month and a 31 day expanding file. [6]
[2] https://taskwarrior.org/docs/using_dates.html
[3] https://taskwarrior.org/docs/named_dates.html
[4] https://taskwarrior.org/docs/durations.html
[5] https://pandas.pydata.org/pandas-docs/stable/user_guide/time...
Ask HN: Any suggestion on how to test CLI applications?
Hello HN!
I've been looking at alternatives on how to test command line applications, specifically, for example, exit codes, output messages and whatnot. I've seen "bats" https://github.com/sstephenson/bats and Bazel for testing but I'm curious as what other tools people use in a day to day basis. UI testing is nice with tools like Cypress.io and maybe there's something out there that isn't as popular but it's useful.
Thoughts?
pytest-docker-pexpect: https://github.com/nvbn/pytest-docker-pexpect
Pexpect: https://pexpect.readthedocs.io/en/stable/
pytest with subprocess.popen (or Sarge) may be sufficient for checking return codes and checking stdout and stderr output streams. Pytest has tmp_path and tmpdir fixtures that provide less test isolation than Docker containers: http://doc.pytest.org/en/latest/tmpdir.html
sarge.Capture.expect() takes a regex and returns None if there's no match: https://sarge.readthedocs.io/en/latest/tutorial.html#looking...
The Golden Butterfly and the All Weather Portfolio
The Golden Butterfly (is a modified All Weather Portfolio)
> Stocks: 20% Domestic Large Cap Fund (Vanguard’s VTI or Goldman Sach’s JUST), 20% Domestic Small Cap Value (Vanguard’s VBR)
> Bonds: 20% Long Term (Vanguard’s BLV), 20% Short Term (Vanguard’s BSV)
> Real Assets: 20% Gold (SPDR’s GLD)
The All Weather Portfolio:
> Stocks: 30% Domestic Total Stock Market (VG total stock)
> Bonds: 40% Long Term, 15% Intermediate-Term
> Real Assets: 7.5% Commodities, 7.5% Gold
Canada's Decision To Make Public More Clinical Trial Data Puts Pressure On FDA
We really could get more out of this data through international collaboration and through linked data (e.g. URIs for columns). See: "Open, and Linked, FDA data" https://github.com/FDA/openfda/issues/5#issuecomment-5392966... and "ENH: Adverse Event Count / 'Use' Count Heatmap" https://github.com/FDA/openfda/issues/49
With sales/usage counts, we'd have a denominator with which we could calculate relative hazard.
Python Alternative to Docker
Shiv does not solve for what containers and Docker/Podman/Buildah/Containerd solve for: re-launching processes at boot and failure, launching processes in chroots or cgroups (with least privileges), limiting access to network ports, limiting access to the host filesystem, building chroots / images, [...]
You can run build tools like shiv with a RUN instruction in a Dockerfile and get some caching.
You can build a zipapp with shiv (in a build container) and run the zipapp in a container.
Should the zipapp contain the test suite(s) and test_requires so that the tests can be run in an environment most similar to production?
It's much easier to develop with code on the filesystem (instead of in a zipapp).
It's definitely faster to read the whole zipapp into RAM than to stat and read each imported module from the filesystem once at startup.
There may be a better post title than the current "Python Alternative to Docker"? Shiv is a packaging utility for building Python zipapps. Shiv is not an alternative to process isolation with containers (or VMs)
$6B United Nations Agency Launches Bitcoin, Ethereum Crypto Fund
"UNICEF launches Cryptocurrency Fund: UN Children’s agency becomes first UN Organization to hold and make transactions in cryptocurrency" https://www.unicef.org/press-releases/unicef-launches-crypto...
From https://www.unicefusa.org/ :
> UNICEF USA helps save and protect the world's most vulnerable children. UNICEF USA is rated one of the best charities to donate to: 89% of every dollar spent goes directly to help children.
Timsort, the Python sorting algorithm
Here are the Python 3 docs for sorting [1], in-place list.sort() [2], and sorted() [3] (which makes a sorted copy of the references). And the Timsort Wikipedia page [4].
[1] https://docs.python.org/3/howto/sorting.html#sort-stability-...
[2] https://docs.python.org/3/library/stdtypes.html#list.sort
Supreme Court allows blind people to sue retailers if websites aren't accessible
"a11y": Accessibility
https://a11yproject.com/ has patterns, a checklist for checking web accessibility, resources, and events.
awesome-a11y has a list of a number of great resources for developing accessible applications: https://github.com/brunopulis/awesome-a11y
In terms of W3C specifications [1], you've got: WAI-ARIA (Web Accessibility Initiative: Accessibile Rich Internet Applications) [2], and WCAG: Web Content Accessibility Guidelines [3]. The new W3C Payment Request API [4] makes it easy for browsers to offer a standard (and probably(?) already accessible) interface for the payment data entry screen, at least.
There are a number of automated accessibility testing platforms. "[W3C WAI] Web Accessibility Evaluation Tools List" [5] lists quite a few. Can someone recommend a good accessibility testing tools? Is Google Lighthouse (now included with Chrome Devtools and as a standalone script) a good tool for accessibility reviews?
[1] https://github.com/brunopulis/awesome-a11y/blob/master/topic...
[2] https://www.w3.org/TR/using-aria/
[3] https://www.w3.org/WAI/standards-guidelines/wcag/
Streamlit: Turn a Python script into an interactive data analysis tool
Cool!
requests_cache caches HTML requests into one SQLite database. [1] pandas-datareader can cache external data requests with requests-cache. [2]
dask.cache can do opportunistic caching (of 2GB of data). [3]
How does streamlit compare to jupyter voila dashboards (with widgets and callbacks)? They just launched a new separate github org for the project. [4] There's a gallery of voila dashboard examples. [5]
> Voila serves live Jupyter notebooks including Jupyter interactive widgets.
> Unlike the usual HTML-converted notebooks, each user connecting to the Voila tornado application gets a dedicated Jupyter kernel which can execute the callbacks to changes in Jupyter interactive widgets.
> - By default, voila disallows execute requests from the front-end, preventing execution of arbitrary code.
[1] https://github.com/reclosedev/requests-cache
[2] https://pandas-datareader.readthedocs.io/en/latest/cache.htm...
[3] https://docs.dask.org/en/latest/caching.html
[4] https://github.com/voila-dashboards/voila
[5] https://blog.jupyter.org/a-gallery-of-voil%C3%A0-examples-a2...
Acess control and resource exhaustion are challenges with building any {Flask, framework_x,} app [from Jupyter notebooks]. First it's "HTTP Digest authentication should be enough for now"; then it's "let's use SSO and LDAP" (and review every release); then it's "why is it so sloww?". JupyterHub has authentication backends, spawners, and per-user-container/vm resource limits.
> Each user on your JupyterHub gets a slice of memory and CPU to use. There are two ways to specify how much users get to use: resource guarantees and resource limits. [6]
[6] https://zero-to-jupyterhub.readthedocs.io/en/latest/user-res...
Some notes re: voila and JupyterHub:
> The reason for having a single instance running voila only is to allow non JupyterHub users to have access to the dashboards. So without going through the Hub auth flow.
> What are the requirements in your case? Voila can be installed in the single user Docker image, so that each user can also use it on their own server (as a server extension for example). [7]
Scott’s Supreme Quantum Supremacy FAQ
Who even asked these questions?
I question this. All of this.
Literally people on HN have been asking many of these questions for years whenever QC is discussed.
Ask HN: How do you handle/maintain local Python environments?
I'm having some trouble figuring out how to handle my local Python. I'm not asking about 2 vs 3 - that ship has sailed - I'm confused on which binary to be using. From the way I see it, there's at least 4 different Pythons I could be using:
1 - Python shipped with OS X/Ubuntu
2 - brew/apt install python
3 - Anaconda
4 - Getting Python from https://www.python.org/downloads/
And that's before getting into how you get numpy et al installed. What's the general consensus on which to use? It seems like the OS X default is compiled with Clang while brew's version is with GCC. I've been working through this book [1] and found this thread [2]. I really want to make sure I'm using fast/optimized linear algebra libraries, is there an easy way to make sure? I use Python for learning data science/bioinformatics, learning MicroPython for embedded, and general automation stuff - is it possible to have one environment that performs well for all of these?
[1] https://www.amazon.com/Python-Data-Analysis-Wrangling-IPython/dp/1449319793
[2] https://www.reddit.com/r/Python/comments/46r8u0/numpylinalgsolve_is_6x_faster_on_my_mac_than_on/
I just use Anaconda. It's basically the final word in monolithic Python distributions.
For data science/numerical computation, all batteries included. It also has fast optimized linear algebra (MKL) plus extras like dask (paralellization), numba, out of the box. No fuss no muss. No need to fiddle with anything.
Everything else is a "pip install" or "conda install" away. Virtual envs? Has it. Run different Python versions on the same machine via conda environments? Has it. Web dev with Django etc.? All there. Need to containerize? miniconda.
The only downside? It's quite big and takes a while to install. But it's a one time cost.
I also prefer conda for the same reasons.
Precompiled MKL is really nice. Conda and conda-forge now build for aarch64. There are very few wheels for aarch64 on PyPI. Conda can install things like Qt (IPython-qt, spyder,) and NodeJS (JupyterLab extensions).
If I want to switch python versions for a given condaenv (instead of just creating a new condaenv for a different CPython/PyPy version), I can just run e.g. `conda install -y python=3.7` and it'll reinstall everything in the depgraph that depended on the previous python version.
I always just install miniconda instead of the whole anaconda distribution. I always create condaenvs (and avoid installing anything in the root condaenv) so that I can `conda-env export -f environment.yml` and clean that up.
BinderHub ( https://mybinder.org/ ) creates docker containers from {git repos, Zenodo, FigShare,} and launches them in free cloud instances also running JupyterLab by building containers with repo2docker (with REES (Reproducible Execution Environment Specification)). This means that all I have to do is add an environment.yml to my git repo in order to get Binder support so that people can just click on the badge in the README to launch JupyterLab with all of the dependencies installed.
REES supports a number of dependency specifications: requirements.txt, Pipfile.lock, environment.yml, aptSources, postBuild. With an environment.yml, I can install the necessary CPython/PyPy version and everything else.
...
In my dotfiles, I have a setup_miniconda.sh script that installs miniconda into per-CPython-version CONDA_ROOT and then creates a CONDA_ENVS_PATH for the condaenvs. It may be overkill because I could just specify a different python version for all of the conda envs in one CONDA_ENVS_PATH, but it keeps things relatively organized and easily diffable: CONDA_ROOT="~/-wrk/-conda37" CONDA_ENVS_PATH="~/-wrk/-ce37"
I run `_setup_conda 37; workon_conda|wec dotfiles` to work on the ~/-wrk/-ce37/dotfiles condaenv and set _WRD=~/-wrk/-ce37/dotfiles/src/dotfiles.
Similarly, for virtualenvwrapper virtualenvs, I run `WORKON_HOME=~/-wrk/-ve37 workon|we dotfiles` to set all of the venv cdaliases; i.e. then _WRD="~/-wrk/-ve37/dotfiles/src/dotfiles" and I can just type `cdwrd|cdw` to cd to the working directory. (Some of the other cdaliases are: {cdwrk, cdve|cdce, cdvirtualenv|cdv, cdsrc|cds}. So far, I have implemented cdalias support for bash, IPython, and vim)
One nice thing about defining _WRD is I can run `makew <tab>` and `gitw` to `cd $_WRD; make <tab>` and `git -C $_WRD` without having to change directory and then `cd -` to return to where I was.
So, for development, I use a combination of virtualenvwrapper, pipsi, conda, and some shell scripts in my dotfiles that I should get around to releasing and maintaining someday. https://westurner.github.io/dotfiles/venv
For publishing projects, I like environment.yml because of the REES support.
Is the era of the $100 graphing calculator coming to an end?
For $100, you can buy a Pinebook with an 11" or 14" screen, a multitouch trackpad, gigabytes of storage, WiFi, a keyboard without a numpad, and an ARM processor.
On this machine, you can create reproducible analyses with JupyterLab; do arithmetic with Python; work with multidimensional arrays with NumPy, SciPy, Pandas, xarray, Dask; do machine learning with Statsmodels, Scikit-learn, Dask-ML, TPOT; create books of these notebooks (containing code and notes (in Markdown, which easily transformed to HTML) and LaTeX equations) with jupyter-book, nbsphinx, git + BinderHub; store the revision history of your discoveries; publish what you've discovered and learned to public or private git repositories; and complete graded exercises with nbgrader.
But the task is to prepare for a world of mental arithmetic, no validation, no tests, no reference materials, and no search engines; and CAS (Computer Algebra Systems) systems like SymPy and Sage are not allowed.
On this machine, you can run write code, write papers, build spreadsheets and/or Jupyter notebooks, run physical simulations, explore the stars, and play games and watch videos. Videos like: Khan Academy videos and exercises that you can watch and do, with validation, until you've achieved mastery and move on to the next task on your todo.txt list.
But the task is to preserve your creativity and natural curiosity despite the compulsory education system's demands for quality control and allocative efficiency; in an environment where drama and popularity are the solutions to relatedness and acceptance needs.
I have three of these $100 calculators in my toolbox. It's been so long since I've powered them on that I'm concerned that the rechargeable AAA batteries are leaking battery acid.
For $100, you can buy an ARM notebook and install conda and conda-forge packages and build sweet visualizations to collaborate with colleagues on (with Seaborn (matplotlib), HoloViews, Altair, Plotly)
"You must buy a $100 calculator that only runs BASIC and ASM, and only use it for arithmetic so that we can measure you."
Hand tools are fun, but please don't waste any more of my compulsory time.
I would LOVE to live in a world where Jupyter notebooks are the defacto standard for post-primary education. It would take some vision and leadership to bring about that sort of change, but would far better prepare kids for the world they will be running one day.
Reinventing Home Directories
Full Disclosure: I am violently opposed to systemd and pulseaudio (before it was taken away from Poettering).
However, I think I'm seeing a pattern, Poettering identifies real issues and actually tries to fix them.
But it seems like his ambition is impeding his ability to consider those who do not want to approach the fix his way. It's never a measured approach, it /feels/ like a "lets write it now and figure out all the problems down the line and fix them" which, for core pieces of software is a dangerous mindset.
After watching the intro here I agree with him, the current state of things is not ideal, and nobody _wants_ to touch user account management on Linux. On MacOS for example, it's much more thought out, and I think Linux could do it better.
As much as I hate poetterings specific approaches (and, subsequent lockout) I have to give credit to him identifying these issues and actually trying to fix them, it's more than I do. Then again I am a lowly sysadmin type.
I like the tone of your comment, unlike many others here. But to your remark:
"But it seems like his ambition is impeding his ability to consider those who do not want to approach the fix his way. It's never a measured approach, it /feels/ like a "lets write it now and figure out all the problems down the line and fix them" which, for core pieces of software is a dangerous mindset."
I'd like to say that he forces nobody and so we should not complain about his creative endeavors. The market will decide, and you, will decide.
I disagree, you can argue the semantics of if it's him specifically or if it's another project that decides it will only target systemd/pulseaudio. But the fact that if I want a stable distro for server usage in 2019 I _must_ use a systemd distro speaks to the fact that I am indeed forced to use it.
Maybe that's just adoption though, typically the sysadmin community considers the "safe" options to be CentOS/RHEL or Debian Stable. Both of which default to systemd. (and no sysadmin is going to change the init system on production servers unless it presents some abject nightmare, short of eating 30% CPU/Mem-- it's core software, it wont be touched).
Projects like Devuan are simply not considered safe choices by the majority of the sysadmin community, and there's no alternative for RHEL-esque sites.
So, semantically: it's not poetterings fault. But at the same time, I'm still forced.
Sorry, but you'll need to find a word different from "forced"
What word would you recommend?
If I must do something then I am forced? no? -- and I'm not being facetious I'm genuinely curious.
If I wish to keep using production grade linux distributions (that my company has been using for 15+ years) then I must adopt systemd.
I guess I could convince my 25,000+ person organisation to change everything to a Debian based OS and subsequently move them to Debian, but that's not likely.
Why do you think that is? Why have production grade Linux distributions all chosen to adopt systemd?
With SysV init, how do you securely launch processes in cgroups, such that they'll consistently restart when the process happens to terminate, with stdout and stderr logged with consistent timestamps, with process dependency models that allow for faster boots due to parallelization?
(edit)
Journalctl is far better than `tail -f /var/log/starstar` and parsing all of those timestamps and inconsistently escaped logfile formats. There's no good way to modify everything in /etc/init.d in order to log to syslog-ng or rsyslog. Systemd and journalctl solve for that; for unified logging.
IMO, there's no question that systemd is the better way and I have zero nostalgia for spawning everything from a probably a shell specified in an /etc/init.d shebang without process restarting (and logging thereof), cgroups, and consistent logging.
> Why do you think that is? Why have production grade Linux distributions all chosen to adopt systemd?
Because RH funds a lot of software development, and Poettering works for them.
> Journalctl is far better than `tail -f /var/log/starstar`
Except when journald shits the bed and corrupts its own log file. Or when you want ship logs off-machine, yet journald does not support any standards-compliant transport so you end up have to log a second logging daemon (e.g., rsyslog)—in which case what's the damn point of the first? Or when you do a "service mysql start" and it dies, but there is no output anywhere of what went wrong, and you have to guess that you have a typo in your "my.cnf".
But other than that Mrs. Lincoln, how was the play?
> ... in an /etc/init.d shebang without process restarting (and logging thereof), cgroups, and consistent logging.
The problem is not systemd-as-init-replacement. The problem is systemd-as-kitchen-sink. udevd is an "essential" part of systemd? Really? It can't be it's own stand-alone project? Really?
When journald logfile corruption occurs, it's detected and it starts writing a new logfile.
When flatfile logfile corruption occurs, it's not detected and there are multiple logfile formats to contend with. And multiple haphazard logrotate configs.
Here's how to use a separate process to ship journald logs - from one file handle - to a remote logging service: https://unix.stackexchange.com/questions/394822/how-should-i...
While there is a systemd-journal-remote, it's not necessary for journald to try and replicate what's already solved and tested in rsyslog and syslog-ng.
It's quite a bit more work to add every new service to the syslog-ng or rsyslog configuration than to just ship one journald log.
Furthermore, service start/stop events are already in the same stream (with the same timestamp format) with the services' stdout and stderr.
Why hasn't anyone written fsck for corrupted journald recovery?
...
I have not needed to makedev and chown and chatted and chcon anything in very many years. When you accidentally newbishly delete something from a static /dev and rebooting doesn't work and you have no idea what the major minor is or was, it sucks bad.
When you're trying to boot a system on a different machine but it doesn't work because the NIC is in a different bus, it's really annoying to have to symlink /dev or modify /etc. With udevd, all you need to do is define a rule to map the busid device name to e.g. eth0. I can remember encountering the devfs race condition resulting in eth0 and eth1 being mapped to different devices on different boots; which was dangerous because firewall rules are applied to device names.
Udev has been in the kernel since 2.6.
"What problems does udev actually solve?" https://superuser.com/questions/686774/what-problems-does-ud...
With integrated udev and systemd, I have no reason to run a separate hotplugd with a different config format (again with no cgroup support) and a different logstream.
Perhaps ironically, here's a link to the presentation PDF that was posted yesterday: https://news.ycombinator.com/item?id=21036020
And my comments there:
> What a good idea.
> Here's the hyperlinkified link to the {systemd-homed.service, systemd-userdbd.service, homectl, userdbctl} sources from the PDF: https://github.com/poettering/systemd/tree/homed
> Hadn't heard of varlink: https://varlink.org/
> Is there a FIPS-like subset of the most-widely-available LUKS configs? Otherwise home directories won't work on systems that have a limited set of LUKS modules.
Serverless: slower and more expensive
It'd be interesting to see how much this same workload would cost with e.g. OpenFaaS on k8s with autoscaling to zero; but there also you'd need to include maintenance costs like OS and FaaS stack upgrades. https://docs.openfaas.com/architecture/autoscaling/
Entropy can be used to understand systems
Maximum entropy: https://en.wikipedia.org/wiki/Maximum_entropy
Here's a quote of a tweet about a (my own): comment on a schema:BlogPost: https://twitter.com/westurner/status/1048125281146421249:
> “When Bayes, Ockham, and Shannon come together to define machine learning” https://towardsdatascience.com/when-bayes-ockham-and-shannon...
> Comment: "How does this relate to the Principle of Maximum Entropy? How does Minimum Description Length relate to Kolmogorov Complexity?"
Thanks for sharing! Despite the fact that Shannon's "A Mathematical Theory of Communication" is so accessible, I find that most in our field (stats/ML) don't often think through information-theoretic tools in a "first principles way."
Yes, KL divergences show up everywhere, but they are not derived from scratch often enough. Maybe I'm stifled by my campus bubble though :)
New Query Language for Graph Databases to Become International Standard
Graph query languages are nice and all, but what about Linked Data here? Queries of schemaless graphs miss lots of data because without a schema this graph calls it "color" and that graph calls it "colour" and that graph calls it "色" or "カラー". (Of course this is also an issue even when there is a defined schema; but it's hardly possible to just happen to have comprehensible inter or even intra-organizational cohesion without e.g. RDFS and/or OWL and/or SHACL for describing (and changing) the shape of the data)
So, the task is then to compile schema-aware SPARQL to GQL or GraphQL or SQL or interminable recursive SQL queries or whatever it is.
For GraphQL, there's GraphQL-LD (which somewhat unfortunately contains a hashtag-indeterminate dash). I cite this in full here because it's very relevant to the GQL task at hand:
"GraphQL-LD: Linked Data Querying with GraphQL" (2018) https://comunica.github.io/Article-ISWC2018-Demo-GraphQlLD/
> GraphQL is a query language that has proven to be a popular among developers. In 2015, the GraphQL framework [3] was introduced by Facebook as an alternative way of querying data through interfaces. Since then, GraphQL has been gaining increasing attention among developers, partly due to its simplicity in usage, and its large collection of supporting tools. One major disadvantage of GraphQL compared to SPARQL is the fact that it has no notion of semantics, i.e., it requires an interface-specific schema. This therefore makes it difficult to combine GraphQL data that originates from different sources. This is then further complicated by the fact that GraphQL has no notion of global identifiers, which is possible in RDF through the use of URIs. Furthermore, GraphQL is however not as expressive as SPARQL, as GraphQL queries represent trees [4], and not full graphs as in SPARQL.
> In this work, we introduce GraphQL-LD, an approach for extending GraphQL queries with a JSON-LD context [5], so that they can be used to evaluate queries over RDF data. This results in a query language that is less expressive than SPARQL, but can still achieve many of the typical data retrieval tasks in applications. Our approach consists of an algorithm that translates GraphQL-LD queries to SPARQL algebra [6]. This allows such queries to be used as an alternative input to SPARQL engines, and thereby opens up the world of RDF data to the large amount of people that already know GraphQL. Furthermore, results can be translated into the GraphQL-prescribed shapes. The only additional requirement is their queries would now also need a JSON-LD context, which could be provided by external domain experts.
> In related work, HyperGraphQL [7] was introduced as a way to expose access to RDF sources through GraphQL queries and emit results as JSON-LD. The difference with our approach is that HyperGraphQL requires a service to be set up that acts as a intermediary between the GraphQL client and the RDF sources. Instead, our approach enables agents to directly query RDF sources by translating GraphQL queries client-side.
All of these RDFS vocabularies and OWL ontologies provide structure that minimizes the costs of merging and/or querying multiple datasets: https://lov.linkeddata.es/dataset/lov/
All of these schema.org/Dataset s in the "Linked Open Data Cloud" are easier to query than a schemaless graph: https://lod-cloud.net/ . Though one can query schemaless graphs with SPARQL, as well.
For reference, RDFLib has a bunch of RDF graph implementations over various key/value and SQL store backends. RDFLib-sqlachemy does query parametrization correctly in order to minimize the risk of query injection. FOR THE RECORD, SQL Injection is the CWE Top 25 #1 most prevalent security weakness; which is something that any new spec and implementation should really consider before launching anything other than an e.g. overly-verbose JSON-based query language that people end up bolting a micro-DSL onto. https://github.com/RDFLib/rdflib-sqlalchemy
Most practically, I frequently want to read a graph of objects into RAM; update, extend, and interlink; and then transactionally save the delta back to the store. This requires a few things: (1) an efficient binary serialization protocol like Apache Arrow (SIMD), Parquet, or any of the BSON binary JSONs; (2) a transactional local store that can be manually synchronized with the remote store until it's consistent.
SPARQL Update was somewhat of an out-of-scope afterthought. Here's SPARQL 1.1 Update: https://www.w3.org/TR/sparql11-update/
Here's SOLID, which could be implemented with SPARQL on GQL, too; though all the re-serialization really shouldn't be necessary for EAV triples with a named graph URI identifier: https://solidproject.org/
5 star data: PDF -> XLS -> CSV -> RDF (GQL, AFAIU (but with no URIs(!?))) -> LOD https://5stardata.info/en/
Linked Data tends to live in a semantic web world that has a lot of open world assumptions. While there are a few systems like this out there, there aren't many. More practically focused systems collapse this worldview down into a much simpler model, and property graphs suit just fine.
There's nothing wrong with enabling linked data use cases, but you don't need RDF+SPARQL+OWL and the like to do that.
The "semantic web stack" I think has been shown by time and implementation experience to be an elegant set of standards and solutions for problems that very few real world systems want to tackle. In the intervening 2 full generations of tech development that have happened since a lot of those standards were born, some of the underlying stuff too (most particularly XML and XML-NS) went from indispensable to just plain irritating.
"Ask HN: Something like Khan Academy but full curriculum for grade schoolers?" [through undergrads] https://news.ycombinator.com/item?id=23794001
"Ask HN: How to introduce someone to programming concepts during 12-hour drive?" https://news.ycombinator.com/item?id=15454071
"Ask HN: Any detailed explanation of computer science" https://news.ycombinator.com/item?id=15270458 : topologically-sorted? Information Theory and Constructor Theory are probably at the top:
> A bottom-up (topologically sorted) computer science curriculum (a depth-first traversal of a Thing graph) ontology would be a great teaching resource.
> One could start with e.g. "Outline of Computer Science", add concept dependency edges, and then topologically (and alphabetically or chronologically) sort.
> https://en.wikipedia.org/wiki/Outline_of_computer_science
> There are many potential starting points and traversals toward specialization for such a curriculum graph of schema:Things/skos:Concepts with URIs.
> How to handle classical computation as a "collapsed" subset of quantum computation? Maybe Constructor Theory?
> https://en.wikipedia.org/wiki/Constructor_theory
https://westurner.github.io/hnlog/ ... Ctrl-F "interview", "curriculum"