Contents^
Items^
Show HN: OtterTune – Automated Database Tuning Service for RDS MySQL/Postgres
Yo. OtterTune is a database optimization service. It uses machine learning to automatically tune your MySQL and Postgres configuration (i.e., RDS parameter groups) to improve performance and reduce costs. It does this by only looking at your database's runtime metrics (e.g., INNODB_METRICS, pg_stat_database, CloudWatch). We don't need to examine sensitive queries or user tables. We spun this project out of my research group at Carnegie Mellon University in 2020.
This week we've announced that OtterTune is now available to the public. We are offering everyone a starter account to try it out on their Postgres RDS or MySQL RDS databases (all versions, AWS US AZs only). We have seen OtterTune achieve 2-4x performance improvements and 50% cost reductions for these databases compared to using Amazon's default RDS configuration.
I am happy to answer any questions that you may have about how OtterTune works here.
-- Andy
================
More Info:
* 5min Demo Video: https://ottertune.com/blog/ottertune-explained-in-five-minutes
* Free Account Sign-up: https://ottertune.com/try
I know this Show HN is about RDS specifically, but the site suggests this works elsewhere, too; any issues with pointing OtterTune at DBs making heavy use of extensions (e.g. Timescale, Citus) or nonstandard deployment approaches (k8s, patroni, vitess)?
We have deployed OtterTune for on-prem databases (baremetal, containers). But we are not offering that to everyone right now because organizations manage their configurations for these deployments in a bunch of different ways (Github actions, Chef/Puppet, Terraform). The nice thing about RDS is that they have a single API for updating configurations (https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...).
As for the Postgres-derivates that you mentioned (Timescale, Citus), we have not tested OtterTune for them yet. But there is no reason they shouldn't also work because they expose the same metrics API (pg_stat_database) and config knobs. There are just way more people using Postgres (RDS, Aurora), so we are focusing on them right now.
What about OpenStack Trove DBaaS? OpenStack Trove is like an open source self-hosted Amazon RDS or Google CloudSQL. https://docs.openstack.org/trove/latest/
FWIU, Trove supports 10+ databases including MySQL and PostgreSQL.
AFAIU, there are sound reasons to host containers with e.g. OpenStack VMs instead of a k8s scheduler with a proper SAN and just figure out how to redundantly and resiliently sync - replicate, synchronize, primary/secondary, nodeprocd - and tune given the CAP theorem and the given DB implementation(s)?
Here's the official Ansible role for Trove, which provisions various e.g. SQL databases on an OpenStack cloud: https://github.com/openstack/openstack-ansible-os_trove
Despite having just 5.8% sales, over 38% of bug reports come from Linux
Notably, of those bug reports, fewer than 1% (only 3 bugs) were specific to the Linux version of the game. That is, over 99% of the bugs reported by Linux gamers also affected players in other platforms. Moreover (quoting from the OP):
> The report quality [from Linux users] is stellar. I mean we have all seen bug reports like: “it crashes for me after a few hours”. Do you know what a developer can do with such a report? Feel sorry at best. You can’t really fix any bug unless you can replicate it, see it with your own eyes, peek inside and finally see that it’s fixed. And with bug reports from Linux players is just something else. You get all the software/os versions, all the logs, you get core dumps and you get replication steps. Sometimes I got with the player over discord and we quickly iterated a few versions with progressive fixes to isolate the problem. You just don’t get that kind of engagement from anyone else.
Are there any good articles on writing helpful bug reports? Whenever I submit a bug I try to give as much detail and be as specific as possible but I always imagine some dev somewhere reading it rolling their eyes at the unnecessary paragraphs I’ve submitted.
From "Post-surgical deaths in Scotland drop by a third, attributed to a checklist" https://westurner.github.io/hnlog/#comment-19684376 https://news.ycombinator.com/item?id=19686470 :
> GitHub and GitLab support task checklists in Markdown and also project boards [...]
> GitHub and GitLab support (multiple) Issue and Pull Request templates:
> Default: /.github/ISSUE_TEMPLATE.md || Configure in web interface
> /.github/ISSUE_TEMPLATE/Name.md || /.gitlab/issue_templates/Name.md
> Default: /.github/PULL_REQUEST_TEMPLATE.md || Configure in web interface
> /.github/PULL_REQUEST_TEMPLATE/Name.md || /.gitlab/merge_request_templates/Name.md
> There are template templates in awesome-github-templates [1] and checklist template templates in github-issue-templates [2].
Arrow DataFusion includes Ballista, which does SIMD and GPU vectorized ops
From the Ballista README:
> How does this compare to Apache Spark? Ballista implements a similar design to Apache Spark, but there are some key differences.
> - The choice of Rust as the main execution language means that memory usage is deterministic and avoids the overhead of GC pauses.
> - Ballista is designed from the ground up to use columnar data, enabling a number of efficiencies such as vectorized processing (SIMD and GPU) and efficient compression. Although Spark does have some columnar support, it is still largely row-based today.
> - The combination of Rust and Arrow provides excellent memory efficiency and memory usage can be 5x - 10x lower than Apache Spark in some cases, which means that more processing can fit on a single node, reducing the overhead of distributed compute.
> - The use of Apache Arrow as the memory model and network protocol means that data can be exchanged between executors in any programming language with minimal serialization overhead.
Previous article from when Ballista was a separate repo from arrow-datafusion: "Ballista: Distributed compute platform implemented in Rust using Apache Arrow" https://news.ycombinator.com/item?id=25824399
Parsing gigabytes of JSON per second
The project described in the paper is https://simdjson.org/.
Source: https://github.com/simdjson/simdjson
PyPI: https://pypi.org/project/pysimdjson/
There's a rust port: https://github.com/simd-lite/simd-json
... From ijson https://pypi.org/project/ijson/#id3 which supports streaming JSON:
> Ijson provides several implementations of the actual parsing in the form of backends located in ijson/backends: [yajl2_c, yajl2_cffi, yajl2, yajl, python]
Fed to ban policymakers from owning individual stocks
The relatively low salaries compared to the huge amounts of power is almost begging for insider trading, influence peddling, etc.
> The salary of a Congress member varies based on the job title of the congressman or senator. Most senators, representatives, delegates and the resident commissioner from Puerto Rico make a salary of $174,000 per year.
https://www.indeed.com/career-advice/pay-salary/congressman-...
Then again, some people have near insatiable greed and even at a $1M/year salary, some would be looking for ways to further boost their income at the boundaries of ethics, or beyond.
This is about the Fed banning it's own policymakers from owning them, not about congress.
Would the Fed actually have the authority to even do that, regulate what securities Congress members are allowed to hold?
“The Fed” as in the Federal Reserve: no.
“The Fed” as in the Federal Government: yes.
"Blind Trust" > "Use by US government officials to avoid conflicts of interest" https://en.wikipedia.org/wiki/Blind_trust
... If you want to help, you must throw all of your startup equity away.
... No, you may not co-brand with that company (which is not complicit with your agenda).
... Besides, I'm not even eligible for duty: you can't hire me.
... Maybe I could be more helpful from competitive private industry.
... How can a government hire prima donna talent like Iron Man?
... Is it criminal to start a solvent, sustainable business to solve government problems, for that one customer?
... Which operations can a government - operating with or without competition - solve most energy-efficiently and thus cost-effectively? Looks like single-payer healthcare and IDK what else?
(Edit)
US Digital Services Playbook: https://github.com/usds/playbook
From https://www.nist.gov/itl/applied-cybersecurity/nice/nice-fra... :
> "NIST Special Publication 800-181 revision 1, the Workforce Framework for Cybersecurity (NICE Framework), provides a set of building blocks for describing the tasks, knowledge, and skills that are needed to perform cybersecurity work performed by individuals and teams. Through these building blocks, the NICE Framework enables organizations to develop their workforces to perform cybersecurity work, and it helps learners to explore cybersecurity work and to engage in appropriate learning activities to develop their knowledge and skills.
From "NIST Special Publication 800-181 Revision 1: Workforce Framework for Cybersecurity (NICE Framework)" (2020) https://doi.org/10.6028/NIST.SP.800-181r1:
> 3.1 Using Existing Task, Knowledge, and Skill (TKS) Statements
(Edit) FedNOW should - like mCBDC - really consider implementing Interledger Protocol (ILP) for RTGS "Real-Time Gross Settlement" https://interledger.org/developer-tools/get-started/overview...
From https://interledger.org/rfcs/0032-peering-clearing-settlemen... :
> Peering, Clearing and Settling; The Interledger network is a graph of nodes (connectors) that have peered with one another by establishing a means of exchanging ILP packets and a means of paying one another for the successful forwarding and delivery of the packets.
Fed or no, wouldn't you think there'd be money in solving for the https://performance.gov Goals ( https://www.usaspending.gov/ ) and the #GlobalGoals (UN Sustainable Development Goals) -aligned GRI Corporate Sustainability Report? #CSR #ESG #SustyReporting
Hardened wood as a renewable alternative to steel and plastic
From "Hemp Wood: A Comprehensive Guide" https://www.buildwithrise.com/stories/hempwood-the-sustainab... :
> HempWood is priced competitively to similar cuts of black walnut. You can purchase 72" HempWood boards for between $13 and $40 as of the date of publishing. HempWood also sells carving blocks, cabinets, and kits to make your own table. Prices for table kits range from $175 to $300. Jul 5, 2021 […]
> Is Hemp Wood Healthy? Due to its organic roots and soy-based adhesive, hemp wood is naturally non-toxic and doesn't contain VOCs, making it a healthier choice for interior building.
> Hemp wood has also been tested to have a decreased likelihood of warping and twisting. Its design is free of any of the knots common in other hardwoods to reduce wood waste.
FWIU, hempcrete - hemp hurds and sustainable limestone - must be framed; possibly with Hemp Wood, which is stronger than spec lumber of the same dimensions.
FWIU, Hemp batting insulation is soaked in sodium to meet code.
Hopefully the production and distribution processes for these carbon sinks keeps net negative carbon in the black.
I want to like hempwood but the price needs to come down. Hopefully it will as production increases.
What are the limits? Input costs, current economy of scale?
Scale, reportedly. https://www.youtube.com/watch?v=3Qy6awPeric
Investors use AI to analyse CEOs’ language patterns and tone
This might be the best NewsArticle headline on HN I've ever seen.
Why, what does it say? Can you log that in a reproducible Notebook with Docs and Test assertions please?
Or are we talking about maybe a ScholarlyArticle CreativeWork with a https://schema.org/funder property or just name and url.
Graph of Keybase commits pre and post Zoom acquisition
Oh boy. Is it too early to say RIP?
The github repo looks ripe for a fork.
The servers are not FOSS and would need reimplementing.
Likely easy enough for a client based on E2E encryption principles; the backend is in many ways a (fancy) dumb pipe. (It could still require complex infrastructure, but at least there'd be relative little "feature" code on the backend to be rewritten.)
FWIU, Cyph does Open Source E2E chat, files, and unlimited length social posts to circles or to public; but doesn't yet do encrypted git repos that can be solved with something like git-crypt. https://github.com/cyph/cyph
It would be wasteful to throw away the Web of Trust (people with handles to keys) that everyone entered into Keybase. Hopefully, Zoom will consider opening up the remaining pieces of Keybase if not just spinning the product back out to a separate entity?
From https://news.ycombinator.com/item?id=19185998 https://westurner.github.io/hnlog/#comment-19185998 :
> There's also "Web Key Directory"; which hosts GPG keys over HTTPS from a .well-known URL for a given user@domain identifier: https://wiki.gnupg.org/WKD
> GPG presumes secure key distribution
> Compared to existing PGP/GPG keyservers [HKP], WKD does rely upon HTTPS.
Blockcerts can be signed when granted to a particular identity entity:
> Here are the open sources of blockchain-certificates/cert-issuer and blockchain-certificates/cert-verifier-js: https://github.com/blockchain-certificates
CT Certificate Transparency logs for key grants and revocations may depend upon a centralized or a decentralized Merkleized datastore: https://en.wikipedia.org/wiki/Certificate_Transparency
How do I specify the correct attributes of my schema.org/Person record (maybe on my JAMstack site) in order to approximate the list of identities that e.g. Keybase lets one register and refer to a cryptographic proof of?
Do I generate a W3C DID and claim my identities by listing them in a JSON-LD document signed with W3C ld-proofs (ld-signatures)? Which of the key directory and Web of Trust features of Keybase are covered by existing W3C spec Use Cases?
From https://news.ycombinator.com/item?id=28701355:
> "Use Cases and Requirements for Decentralized Identifiers" https://www.w3.org/TR/did-use-cases/
>> 2. Use Cases: Online shopper, Vehicle assemblies, Confidential Customer Engagement, Accessing Master Data of Entities, Transferable Skills Credentials, Cross-platform User-driven Sharing, Pseudonymous Work, Pseudonymity within a supply chain, Digital Permanent Resident Card, Importing retro toys, Public authority identity credentials (eIDAS), Correlation-controlled Services
> And then, IIUC W3C Verifiable Credentials / ld-proofs can be signed with W3C DID keys - that can also be generated or registered centrally, like hosted wallets or custody services. There are many Use Cases for Verifiable Credentials: https://www.w3.org/TR/vc-use-cases/ :
>> 3. User Needs: Education, Retail, Finance, Healthcare, Professional Credentials, Legal Identity, Devices
>> 4. User Tasks: Issue Claim, Assert Claim, Verify Claim, Store / Move Claim, Retrieve Claim, Revoke Claim
>> 5. Focal Use Cases: Citizenship by Parentage, Expert Dive Instructor, International Travel with Minor and Upgrade
>> 6. User Sequences: How a Verifiable Credential Might Be Created, How a Verifiable Credential Might Be Used
Is there an ACME-like thing to verify online identity control like Keybase still does?
Hopefully, Zoom will consider opening up the remaining pieces of Keybase if not just spinning the product back out to a separate entity?
> Is there an ACME-like thing to verify online identity control like Keybase still does?
From https://news.ycombinator.com/item?id=28926739 :
> NIST SP 800-63 https://pages.nist.gov/800-63-3/ :
> SP 800-63-3: Digital Identity Guidelines https://doi.org/10.6028/NIST.SP.800-63-3
> SP 800-63A: Enrollment and Identity Proofing https://doi.org/10.6028/NIST.SP.800-63a
FWIU, NIST SP 800-63A Enrollment and Identity Proofing specifies a spec sort of like ACME but for offline identity.
"Key server (cryptographic)" https://en.wikipedia.org/wiki/Key_server_(cryptographic)
> The last IETF draft for HKP also defines a distributed key server network, based on DNS SRV records: to find the key of someone@example.com, one can ask it by requesting example.com's key server.
> Keyserver examples: These are some keyservers that are often used for looking up keys with `gpg --recv-keys`.[6] These can be queried via https:// (HTTPS) or hkps:// (HKP over TLS) respectively: keys.openpgp.org , pgp.mit.edu , keyring.debian.org , keyserver.ubuntu.com ,
"Linked Data Signatures for GPG" https://gpg.jsld.org/
npm i @transmute/lds-gpg2020 -g
gpg2020 sign -u "3BCAC9A882DEFE703FD52079E9CB06E71794A713" $(pwd)/docs/example/doc.json did:btcr:xxcl-lzpq-q83a-0d5#yubikey
> GpgSignature2020: A JSON-LD Document has been signed with GpgSignature2020, when it contains a proof field with type GpgSignature2020. The proof must contain a key signatureValue with value defined by the signing algorithm described here. Example:
{
"@context": [
"https://gpg.jsld.org/contexts/lds-gpg2020-v0.0.jsonld",
{
"schema": "http://schema.org/",
"name": "schema:name",
"homepage": "schema:url",
"image": "schema:image"
}
],
"name": "Manu Sporny",
"homepage": "https://manu.sporny.org/",
"image": "https://manu.sporny.org/images/manu.png",
"proof": {
"type": "GpgSignature2020",
"created": "2020-02-16T18:21:26Z",
"verificationMethod": "did:web:did.or13.io#20a968a458342f6b1a822c5bfddb584bdf141f95",
"proofPurpose": "assertionMethod",
"signatureValue": "-----BEGIN PGP SIGNATURE-----\n\niQEzBAABCAAdFiEEIKlopFg0L2sagixb/dtYS98UH5UFAl5JiCYACgkQ/dtYS98U\nH5U8TQf/WS92hXkdkdBQ0xJcaSkoTsGspshZ+lT98N2Dqu6I1Q01VKm+UMniv5s/\n3z4VX83KuO5xtepFjs4S95S4gLmr227H7veUdlmPrQtkGpvRG0Ks5mX7tPmJo2TN\nDwm1imm+zvJ+MXr3Ld24qaRJA9dI+AoZ5HXqNp96Yncj3oWD+DtVIZmC/ZiUw43a\nLpMYy94Hie7Ad86hEoqsdRxrwq7O6KZ29TAKi5T/taemayyXY7papU28mGjVEcvO\na7M3XNBflMcMEB+g6gjrANsgFNO6tOuvOQ2+4v6yMfpJ0ji4ta7q2d4QKqGi5YhE\nsRUORN+7HJrkmSTaT7gBpFQ+YUnyLA==\n=Uzp1\n-----END PGP SIGNATURE-----\n"
}
}Single sign-on: What we learned during our identity alpha
Reading documents like passports with an NFC reader does indeed work, and does indeed produce verifiable material. Specifically the passport has proof (via a digital signature) that it was issued by a specific authority, and in turn, proof that the contents of the passport (name, date of birth, a picture and so on) are as issued.
But, the problem here is that the issuer is the British government, so, what are you proving? "Here, you issued this passport". "Oh yes, so we did". I presume the British government does own a database of the passports they issued, so this isn't news to them.
A modestly smart device, such as a Yubico device, is capable of providing fresh proof of its identity. My Security Key doesn't prove "The security key that enrolled me with GitHub in fact exists" which is redundant - but "I am still the same security key that you enrolled". However the passport can't do that, your passport is inert, and the fact that Sarah Smith existed isn't the thing you presumably want to prove to a single-sign-on service. You want to prove that you are Sarah Smith, something the passport doesn't really do.
I think the GDS ignores this problem, which is to be fair no worse than lots of other systems, but the result isn't actually what it seems to be, all the digital technology isn't actually proving anybody's identity in this space.
It reminds me of the bad old days of the Web PKI where it was found that the "email validation" being used would accept automated "virus checking" of email. A CA sends the "Are you sure you want to issue a cert for mycorp.example?" message to somebody@mycorp.example and even though Somebody is on vacation in Barbados for two weeks, the automatic "virus" check reads the URL out of the email, follows it, ignores the page saying "Success, your certificate has been issued" and passes it to Somebody's inbox... All the "security" is doing what it was designed to do, but, what it was designed to do isn't what it should have been designed to do, and so it's futile.
Isn't there two things going on here? A single sign on service, and also an identity check.
A yubico device can say "I'm still the same security key that was enrolled" but it doesn't say at enrolment "I am being used by Fred Jones of <address> with passport no 123, NiNo ABC".
GDS verify / government gateway do support 'authenticators' (typically password + totp) to provide a level of assurance that the person logging into the account is the person the account belongs to.
The "document check" is part of the identity bit. That is, how confident are we that the person creating this account / performing this action is who they say they are and can do this thing or see that information. The document check is part of their solution but other parts are supposed to layer on top of it to provide a proportionate level of assurance. e.g. a video of you holding up the passport to check that it's you using it or asking you to provide answers to details you already have like 'how much tax did you pay last year' or checking multiple documents.
> I presume the British government does own a database of the passports they issued, so this isn't news to them.
The passport office (part of the Home Office) own that database. Part of this work is basically a web service to that database for the rest of the government to use.
This is the difference between 'identity assurance' (at enrollment) and 'authentication assurance' in the relevant NIST standard, SP 800-63 https://pages.nist.gov/800-63-3/
A remote passport check might be suitable to claim an identity assurance level of 2, say, while getting to identity assurance level 3 would need an in-person visit with multiple forms of other government identification records.
In that way you might then constrain some actions to be taken remotely only by users who have provided strong assurance of their identity at enrollment (even if they login with strong non-phishable authentication... doesn't matter that the auth is ironclad if the user's identity is muddy).
Thx. NIST SP 800-63* https://pages.nist.gov/800-63-3/ :
> SP 800-63-3: Digital Identity Guidelines https://doi.org/10.6028/NIST.SP.800-63-3
> SP 800-63A: Enrollment and Identity Proofing https://doi.org/10.6028/NIST.SP.800-63a
> SP 800-63B: Authentication and Lifecycle Management https://doi.org/10.6028/NIST.SP.800-63b
> SP 800-63C: Federation and Assertions https://doi.org/10.6028/NIST.SP.800-63c
Five things we still don’t know about water
Also interesting:
https://sciencenordic.com/chemistry-climate-denmark/the-eart...
> The Earth has lost a quarter of its water
> In its early history, the Earth's oceans contained significantly more water than they do today. A new study indicates that hydrogen from split water molecules has escaped into space.
But where did the water come from? Neptune? Europa? Comet(s)? Is it just the distance to our nearest star in our habitable zone here that results in liquid water being likely?
From the article:
> But the exact mechanism for how water evaporates isn’t completely understood. The evaporation rate is traditionally represented in terms of a rate of collision between molecules, multiplied by a fudge factor called the evaporation coefficient, which varies between zero and one. Experimental determination of this coefficient, spanning several decades, has varied over three orders of magnitude.
From https://en.wikipedia.org/wiki/Evaporation :
> Evaporation is a type of vaporization that occurs on the surface of a liquid as it changes into the gas phase.[1] The surrounding gas must not be saturated with the evaporating substance. When the molecules of the liquid collide, they transfer energy to each other based on how they collide with each other. When a molecule near the surface absorbs enough energy to overcome the vapor pressure, it will escape and enter the surrounding air as a gas.[2] When evaporation occurs, the energy removed from the vaporized liquid will reduce the temperature of the liquid, resulting in evaporative cooling.[3]
New Optical Switch Up to 1000x Faster Than Transistors
Does this work at normal temperatures? Or does it need impractical cooling setups?
Show HN: I built a sonar into my surfboard
I wonder if there is some way you could pull data from the sound of the waves breaking instead of an active ping? That might be the only reasonable way to get accurate readings in that zone.
FWIU, EM backscatter can be used for e.g. gesture recognition, heartbeat detection, and metal detection. https://en.wikipedia.org/wiki/Backscatter
Entropy of wave noises may or may not be the issue.
Edit: (NASA spinoff) "Radar Device Detects Heartbeats Trapped under Wreckage" https://spinoff.nasa.gov/Spinoff2018/ps_1.html
> The Edgewood, Maryland-based company is developing a line of such remote sensing devices to aid search and rescue teams, based on advanced radar technologies developed by NASA and refined for this purpose at the Agency’s Jet Propulsion Laboratory (JPL).
> NASA has long analyzed weak radio signals to identify slight physical movements, such as seismic activity seen from low-Earth orbit or minor alterations in a satellite’s path around another planet that might indicate gravity fluctuations, explains Jim Lux, JPL’s task manager for the FINDER project. However, to pick out such faint patterns in the data, these devices must cancel out huge amounts of noise. “The core technology here is measuring a small signal in the context of another larger signal that’s confusing you,” Lux says.
(FWIW, some branches may have helicopters with infrared that they can cost over for disaster relief.)
Cortical Column Networks
Hey, Cortical Columns!
From "Jeff Hawkins Is Finally Ready to Explain His Brain Research" https://news.ycombinator.com/item?id=18214707 https://westurner.github.io/hnlog/#comment-18218504
What does (parallel) spreading activation have to do with Cortical Column Networks maybe and redundancy? https://en.wikipedia.org/wiki/Spreading_activation
I got the impression that anything transformers could be applied to, that would be an advantage. So now I want everything to use them.
Doesn't sound like these CCNs necessarily reuse a lot of information across object categories.
From https://medium.com/syncedreview/google-replaces-bert-self-at... :
> New research from a Google team proposes replacing the self-attention sublayers with simple linear transformations that “mix” input tokens to significantly speed up the transformer encoder with limited accuracy cost. Even more surprisingly, the team discovers that replacing the self-attention sublayer with a standard, unparameterized Fourier Transform achieves 92 percent of the accuracy of BERT on the GLUE benchmark, with training times that are seven times faster on GPUs and twice as fast on TPUs."
Would Transformers (with self-attention) make what things better? Maybe QFT? There are quantum chemical interactions in the brain. Are they necessary or relevant for what fidelity of emulation of a non-discrete brain?
Startup Ideas
Bring back the dumb terminal. Place mobile phone on a docking pad and see a standard PC OS on the screen - open apps, access files or use a browser. When done just grab the phone and go.
IIUC, in 2021, you can dock a PineTab or a PinePhone with a USB-C PD hub that has HDMI, USB, and Ethernet and use any of a number of Linux Desktop operating systems on a larger screen with full size keyboard and mouse.
The PineTab has a backlit keyboard and IIUC the PinePhone has a keyboard & aux battery case that doesn't yet also include the fingerprint sensor or wireless charging. https://www.pine64.org/blog/
It is easier to educate a Do-er than to motivate the educated
~ "Imagine that one could give you a copy of all of their knowledge. If you do not choose to apply and learn on your own, you can never."
This is about regimen, this is about stamina, this is about sticktoitiveness; and if you don't want it, you don't need it, you'll never. And I mean never.
The Grit article on Wikipedia mentions persistence and tenacity and stick-to-it-tiveness as roughly synonymous; and that grit may not be that distinct from other Big Five personality traits, but we're not about to listen to that, we're not going with that, because Grit is predictor of success. https://en.wikipedia.org/wiki/Grit_(personality_trait)
To the original point,
> In psychology, grit is a positive, non-cognitive trait based on an individual's perseverance of effort combined with the passion for a particular long-term goal or end state (a powerful motivation to achieve an objective). This perseverance of effort promotes the overcoming of obstacles or challenges that lie on the path to accomplishment and serves as a driving force in achievement realization. Distinct but commonly associated concepts within the field of psychology include "perseverance", "hardiness", "resilience", "ambition", "need for achievement" and "conscientiousness". These constructs can be conceptualized as individual differences related to the accomplishment of work rather than talent or ability.
Are software engineering “best practices” just developer preferences?
"The parallel he drew was to another friend who’s a Civil Engineer. His friend had to be state certified and build everything to certain codes that stand up to specific stressors and inspections.
I gave him the usual answer about how Software Engineers deal with low stakes and high iterability compared to Civil Engineers, but, honestly, he has a point."
I've argued for awhile now that Software Engineering with a big E should be licensed and regulated the same as any other Engineering discipline. Not all software is low stakes and fast changing. In fact I'd argue the most important software is never that. software for control systems, avionics, cars etc are very high stakes and have no reason to iterate beyond what is needed to interface with changing hardware. I think if software engineers had to be licensed to work on such things then those 737s wouldn't have fallen out of the sky and Tesla wouldn't be allowed to beta test self driving cars on public roads.
To those who ask what do you now call software engineers who don't work on those things, you are programmers. Or if you prefer a less formal term, coders. Engineer is a powerful word and I don't like how the we the IT industry have appropriated it for less critical tasks.
In a way there are standards for software but those standards are not expressed in software terms.
Firstly, most software is harmless. If it goes wrong people may be annoyed, but no one is harmed. But if I write software to control an aircraft, then it would have to abide by aviation standards. If I write financial software then I would have financial regulations to follow. Same for medical devices. So, there are standards for software but they are indirect and expressed in terms of the wider domain in which it runs.
Critical systems: https://en.wikipedia.org/wiki/Critical_system :
> There are four types of critical systems: safety critical, mission critical, business critical and security critical.
Safety-critical systems > "Software engineering for safety-critical systems" https://en.wikipedia.org/wiki/Safety-critical_system#Softwar... :
> By setting a standard for which a system is required to be developed under, it forces the designers to stick to the requirements. The avionics industry has succeeded in producing standard methods for producing life-critical avionics software. Similar standards exist for industry, in general, (IEC 61508) and automotive (ISO 26262), medical (IEC 62304) and nuclear (IEC 61513) industries specifically. The standard approach is to carefully code, inspect, document, test, verify and analyze the system. Another approach is to certify a production system, a compiler, and then generate the system's code from specifications. Another approach uses formal methods to generate proofs that the code meets requirements.[11] All of these approaches improve the software quality in safety-critical systems by testing or eliminating manual steps in the development process, because people make mistakes, and these mistakes are the most common cause of potential life-threatening errors.
awesome-safety-critical lists very many resources for safety critical systems: https://awesome-safety-critical.readthedocs.io/en/latest/
There are many ['Engineering'] certification programs for software and other STEM fields. One test to qualify applicants does not qualify as a sufficient set of controls for safety critical systems that must be resilient, fault-tolerant, and redundant.
A real Engineer knows that there are insufficient process controls from review of very little documentation; it's just process wisdom from experience. An engineer starts with this premise: "There are insufficient controls to do this safely" because [test scenario parameter set n] would result in the system state - the output of probably actually a complex nonlinear dynamic system - being unacceptable: outside of acceptable parameters for safe operation.
Are there [formal] Engineering methods that should be requisite to "Computer Science" degrees? What about "Applied Secure Coding Practices in [Language]"? Is that sufficient to teach theory and formal methods?
From "How We Proved the Eth2 Deposit Contract Is Free of Runtime Errors" https://news.ycombinator.com/item?id=28513922 :
>> From "Discover and Prevent Linux Kernel Zero-Day Exploit Using Formal Verification" https://news.ycombinator.com/item?id=27442273 :
>> [Coq, VST, CompCert]
>> Formal methods: https://en.wikipedia.org/wiki/Formal_methods
>> Formal specification: https://en.wikipedia.org/wiki/Formal_specification
>> Implementation of formal specification: https://en.wikipedia.org/wiki/Anti-pattern#Software_engineer...
>> Formal verification: https://en.wikipedia.org/wiki/Formal_verification
>> From "Why Don't People Use Formal Methods?" https://news.ycombinator.com/item?id=18965964 :
>>> Which universities teach formal methods?
>>> - q=formal+verification https://www.class-central.com/search?q=formal+verification
>>> - q=formal+methods https://www.class-central.com/search?q=formal+methods
>>> Is formal verification a required course or curriculum competency for any Computer Science or Software Engineering / Computer Engineering degree programs? https://news.ycombinator.com/item?id=28513922
From "Ask HN: Is it worth it to learn C in 2020?" https://news.ycombinator.com/item?id=21878372 :
> There are a number of coding guidelines e.g. for safety-critical systems where bounded running time and resource consumption are essential. These coding guidelines and standards are basically only available for C, C++, and Ada.
awesome-safety-critical > Software safety standards: https://awesome-safety-critical.readthedocs.io/en/latest/#so...
awesome-safety-critical > Coding Guidelines: https://awesome-safety-critical.readthedocs.io/en/latest/#co...
Major Quantum Computing Strategy Suffers Serious Setbacks
Inevitably, this article (and implications about quantum computing) is going to be vastly misinterpreted due to the title. It would be like saying “Major Shuttle Strategy Suffers Serious Setbacks” but the strategy is using slingshots to get us into orbit, ignoring the work of, say, NASA and SpaceX.
This quantum computing strategy neither was nor is practiced by the vast majority of quantum institutions—commercial or otherwise. It was attempted by a group at Microsoft (and a small collection of other university groups) and was known from the start that it would be a search for essentially fundamentally new observations.
Other quantum computing players, like Rigetti, Google, HRL Laboratories, IBM, Amazon, Honeywell, and others are doing an approach that is nothing like the article, and have demonstrated significant results.
It should really be "A quantum computing strategy suffers..."
Or: "One idea in quantum computing bears no fruit"
"Quantized Majorana conductance not actually observed within indium antimonide nanowires"
"Quantum qubit substrate found to be apparently insufficient" (Given the given methods and probably available resources)
And then - in an attempt to use terminology from Constructor Theory https://en.m.wikipedia.org/wiki/Constructor_theory :
> In constructor theory, a transformation or change is described as a task. A constructor is a physical entity which is able to carry out a given task repeatedly. A task is only possible if a constructor capable of carrying it out exists, otherwise it is impossible. To work with constructor theory everything is expressed in terms of tasks. The properties of information are then expressed as relationships between possible- and impossible tasks. Counterfactuals are thus fundamental statements and the properties of information may be described by physical laws.[4] If a system has a set of attributes, the set of permutations of these attributes is seen as a set of tasks. A computation medium is a system whose attributes permute to always produce a possible task. The set of permutations, and hence of tasks, is a computation set. If it is possible to copy the attributes in the computation set, the computation medium is also an information medium.
> Information, or a given task, does not rely on a specific constructor. Any suitable constructor will serve. This ability of information to be carried on different physical systems or media is described as interoperability, and arises as the principle that the combination of two information media is also an information medium.[4] Media capable of carrying out quantum computations are called superinformation media, and are characterised by specific properties. Broadly, certain copying tasks on their states are impossible tasks. This is claimed to give rise to all the known differences between quantum and classical information.[4]
"Subsequent attempts to reproduce [Quantized Majorana conductance (topological qubits of arranged electrons) within indium antimonide nanowires] eventually as a (quantum) computation medium for the given tasks failed"
"Quantum computation by Majorana zero-mode (MZM) quasiparticles in indium antimonide nanowires not actually apparently possible"
... "But what about in DDR5?" Which leads us to a more generally interesting: "Rowhammer for qubits", which is already an actual Quantum on Silicon (QoS) thing.
Attempts to scientifically “rationalize” policy may be damaging democracy
First, not having read the article:
#EvidenceBasedPolicy is a worthwhile objective even if only because the alternative is to just blow money without measuring ROI at all [because government expenditures are the actual key to feeding the beast, the economic beast, the...].
What are some examples of policy failures where Systematic review and Meta-analysis could have averted loss, harms, waste, catastrophe, long-term costs? Is that cherry picking? The other times we can just throw a dart and that's better than, ahem, these idiots we afford trying to do science?
Wouldn't it be fair to require that constituent ScholarlyArticles (and other CreativeWorks) be kept on file with e.g. the Library of Congress?
Non-federal governments usually have very similar IT and science policy review needs. Should adapting one system for non-federal governments be more complex than specifying a different String or URL in the token_name field in a transaction?
When experts review ScholarlyArticles on our behalf, they should share their structured and unstructured annotations in such a way that their cryptographically signed reviews - and highlights to identify and extract structured facts like summary statistics like sample size and IRB-reviewed study controls - become part of a team-focused collaborative systematic meta-analysis that is kept on file and regularly reviewed in regards to e.g. retractions, typical cognitive biases, failures in experimental design and implementation, and general insufficiencies that should cause us to re-evaluate our beliefs given all available information which meets our established inclusion criteria.
We have a process for peer review of PDFs - and hopefully datasets with locality for reproducibility and unitarity which purportedly helps us work through something like this sequence:
Data / Information / Knowledge / Experience / Wisdom
We often have gaps in our processes to support such progress in developing wisdom from knowledge that should be predicated upon sound information and data and then experience, bias, creeps in.
Basic principles restricting the powers of the government should prevent the government - us, we - from specifically violating the protected rights of persons; but we have allowed "Science" to cloud our judgement in application of our most basic principles of justice - i.e. Life, Liberty, and the pursuit of Happiness; and Equality and Equitability - and should we chalk the unintended consequences up to ignorance or malice?
More science all around: more Data Literacy - awareness of how many bad statistical claims are made all day around the world everywhere - is good and necessary and essential to Media Literacy, which is how we would be forming our opinions if we didn't have better tools for truth and belief for science.
"What does it mean to know?" etc.
Logic, Inference, Reasoning and Statistics probably predicated upon classical statistical mechanics are supposed to bring us closer to knowing: to bring our beliefs closer to the most widely observed truths.
Which Verifiable Claims do we trust? What studies do we admit into our personal and community meta-analyses according to our shared inclusion criteria?
"Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)" is one standard for meta-analyses, for example. http://www.prisma-statement.org/ . Could the bad guys or the dumb good guys lie with that control in place, too? Can knowing our rights - and upholding oaths to uphold values - protect us from meta-analytical group failure?
Perhaps STEM (Science, Technology, Engineering, art, and Medicine/Math) majors and other interested parties can help develop solutions for #EvidenceBasedPolicy?
This one fell flat. Maybe it was the time of day? The question should be asked every year, at least, eh? "Ask HN: Systems for supporting Evidence-Based Policy?" https://news.ycombinator.com/item?id=22920613
>> What tools and services would you recommend for evidence-based policy tasks like meta-analysis, solution criteria development, and planned evaluations according to the given criteria?
>> Are they open source? Do they work with linked open data?
> I suppose I should clarify that citizens, consumers, voters, and journalists are not acceptable answers
"#LinkedMetaAnalyses", "#StructuredPremises"; Ctrl-F "linkedmeta", "linkedrep", "#LinkedResearch": https://westurner.github.io/hnlog/
Alright, my fair biases disclosed, on to the reading the actual article: /1
Response to 'Call for Review: Decentralized Identifiers (DIDs) v1.0'
Better imho to ask: what user problem is DID solving?
Because: if it's truly decentralized then there's no need to publish it. But publication is a core aspect of DID. That it must be published is a jedi mind trick that allows in "on a blockchain". Now we see the real problem being solved (not a user problem): need something published on a blockchain.
The user problem- for those who see it that way- is that right now, digital stuff related to a person, is tied up primarily with the person's email address, or in some cases with the person's phone number.
(For the purposes of this discussion, call the email address or phone number an "identifier".)
Why is this a problem? Two reasons:
1. People don't "control" those identifiers
Many email addresses used in this context are controlled by employers, and the person's right to use them ends when they leave employment. Or they are controlled through expensive commercial arrangement between the person and the platform, and the person may lose access if they are no longer able to afford the platform. Or, they are free, offered by large data harvesting/advertising platforms, who mine data stored on the platform, and as below, linked to those identifiers from other platforms, to create advertising and propaganda targeting profiles.
2. Those identifiers are used by others to contact the person, and are therefore long-lived, which means they are also a vehicle for correlating an individual's activity across internet platforms who are necessarily presented with those identifiers by the person when they engage with the platform.
DIDs are an attempt to have identifiers that are controlled by people that:
* are inexpensive
* can be short-lived and "rotated"
* can be specific to the relationship between a person and a particular platform
* can support more tailored association of personal data to identifier
* can better support the person's management and correlation of their platform relationships, while minimizing if desired that the correlation of identifiers back to a person by the platforms themselves
* and support other use cases
But the key issue is that right now it is hard to impossible for a normal person to engage with a small or large platform that does not involve a widely used identifier.
[EDIT: whether normal users consider this to be a problem is an open question, and as is whether they would if a solution existed to the problem...]
Somebody introduces a new technology to address these concerns every couple years and it doesn't go anywhere. These aren't actually problems to a lot of users. That's the real problem that needs to be solved - awareness. And that's a lot harder than taking the identity solutions we came up with in the Identity 2.0 days and adding a blockchain.
> Somebody introduces a new technology to address these concerns every couple years and it doesn't go anywhere. These aren't actually problems to a lot of users.
"Use Cases and Requirements for Decentralized Identifiers" https://www.w3.org/TR/did-use-cases/
> 2. Use Cases: Online shopper, Vehicle assemblies, Confidential Customer Engagement, Accessing Master Data of Entities, Transferable Skills Credentials, Cross-platform User-driven Sharing, Pseudonymous Work, Pseudonymity within a supply chain, Digital Permanent Resident Card, Importing retro toys, Public authority identity credentials (eIDAS), Correlation-controlled Services
And then, IIUC W3C Verifiable Credentials / ld-proofs can be signed with W3C DID keys - that can also be generated or registered centrally, like hosted wallets or custody services. There are many Use Cases for Verifiable Credentials: https://www.w3.org/TR/vc-use-cases/ :
> 3. User Needs: Education, Retail, Finance, Healthcare, Professional Credentials, Legal Identity, Devices
> 4. User Tasks: Issue Claim, Assert Claim, Verify Claim, Store / Move Claim, Retrieve Claim, Revoke Claim
> 5. Focal Use Cases: Citizenship by Parentage, Expert Dive Instructor, International Travel with Minor and Upgrade
> 6. User Sequences: How a Verifiable Credential Might Be Created, How a Verifiable Credential Might Be Used
IIRC DHS funded some of the W3C DID and Verified Credentials specification efforts. See also: https://news.ycombinator.com/item?id=26758099
There's probably already a good way to bridge between sub-SKU GS1 schema.org/identifier on barcodes and QR codes and with DIDs. For GS1, you must register a ~namespace prefix and then you can use the rest of the available address space within the barcode or QR code IIUC.
DIDs can replace ORCIDs - which you can also just generate a new one of - for academics seeking to group their ScholarlyArticles by a better identifier than a transient university email address.
The new UUID formats may or may not be optionally useful in conjunction with W3C DID, VC, and Verifiable News, etc. https://news.ycombinator.com/item?id=28088213
When would a DID be a better choice than a UUID?
Apple didn't revolutionize power supplies; new transistors did (2012)
While obviously Jobs’ claim was false, I will say that Apple is the only company I am aware of that manufacturers power supplies which are reliably completely free of perceptible inductor whine. I have very acute high-frequency hearing and I often have to replace non-Apple USB(-C) switching power supplies with Apple ones so I don’t go crazy from the whining. Teardowns of Apple PSUs typically reveal very favorable electronic and industrial design as well.
It’s extremely frustrating, and I was always surprised when I returned mid-range USB chargers because of whine only to receive a replacement with the same problem and hundreds of reviews that failed to mention it. I’ve never had an issue with Apple chargers, and the extra cost is money we’ll spent.
Buy genuine Apple chargers, if not for you, then for your dog.
What does my engineering manager do all day?
I’ve been in those meetings, we’ve all been with different responsibility levels, we know that people are slacking at least 80% of the time and that most of those meetings could be efficiently replaced by emails.
So if your work is 90% meetings I am sorry but you have a bullshit job, nothing more, nothing less.
I kind of agree.
In my last role I ended up in a fantastic place where the teams under me could operate fairly autonomously and were happy. My only recurring meetings were:
- 1-1s fortnightly and only with senior members
- My 1-1 with my boss, again fortnightly
- Senior leadership team, once a week
I kept 1-1s to 30mins unless someone had an important issue, in which case I was completely flexible.
Most of my time went on planning/strategy, customer research, and acting as an ad-hoc coach for the teams beneath me.
The key things I found were:
- many meetings can be replaced by an update email
- decisions often work better through docs + feedback than big meetings
- you don't need frequent contact with the team if the goals and constraints are communicated very clearly
I realise none of this is groundbreaking. But it seems quite rare, and also hard to achieve in practice.
> - many meetings can be replaced by an update email
Highlights from the feed(s); GitLab has the better activity view IMHO but I haven't tried the new GitHub Issues beta yet.
3 questions from 5 minute Stand-Up Meetings (because everyone's actually standing there trying to leave) for Digital Stand Up Meetings: Since, Before, Obstacles:
## 2021-09-28
### @teammembername
#### Since
#### Before
#### Obstacles
You can do cool video backgrounds for any video conferencing app with pipewire.
You can ask team members to prep a .txt with their 3 questions and drop it in the chat such that the team can reply to individual #fragments of your brief status report / continued employment justification argument
> - decisions often work better through docs + feedback than big meetings
SO, ah, asynchronous communication doesn't require transcripting for the "Leader Assistant" that does the Mando quarterly minutes from the team chat logs, at least
6 Patterns of Collaboration: GRCOEB: Generate, Reduce, Clarify, Organize, Evaluate, Build Consensus [Six Patterns]; voting on specific Issues, and ideally Chat - [x] lineitems, and https://schema.org/SocialMediaPosting with emoji reactions
[Six Patterns]: http://wrdrd.github.io/docs/consulting/team-building#six-pat... , Text Templates, Collaboration Checklist: Weighted Criteria, Ranked-choice Voting.
Docs and posts with URLs and in-text pull-quotes do better than another list of citations at the end.
> - you don't need frequent contact with the team if the goals and constraints are communicated very clearly
Metrics: OKRs, KPIs, #GlobalGoals Goals Targets and Indicators
Tools / Methods; Data / Information / Knowledge / Experience / Wisdom:
- Issues: Title, - [ ] Description, Labels, Assignee, - [ ] Comments, Emoji Reactions;
- Pull Requests, - [ ] [Optional] [Formal] Reviews, Labels & "Codelabels", label:SkipPreflight, CI Build Logs, and Signed Deployed Documented Applications; code talks, the tests win again, docs sell
- Find and Choose - with Consensus - a sufficiently mature Component that already testably does: unified Email notifications (with inbound replies,) and notifications on each and every Chat API and the web standard thing finally, thanks: W3C Web Notifications.
- Contribute Tests for [open source] Components.
- [ ] Create a workflow document with URLs and Text Templates
- [ ] Create a daily running document with my 3 questions and headings and indented markdown checkbox lists; possibly also with todotxt/todo.txt / TaskWarrior & BugWarrior -style lineitem markup.
What does an engineering manager do all day?
A polite answer would be, continuously reevaluate the tests of the product and probably also the business model if anyone knew what they were up to in there
Using two keyboards at once for pain relief
> I tried a few of those Kinesis split keyboards. Too squishy for me. Not far enough apart. The CherryMX Kinesis split keyboard was is too clickey for calls and screenshares. Muscle memory made it difficult to switch.
This is a really cool hack, and I’m happy that the author found a solution for their pain that works for them, but this bit confused me.
Kinesis are keyboards with separated key clusters, but not split keyboards. When one says split keyboard I think they are normally talking about things like the Ergodox EZ/Moonlander which have two physically separate bodies, one for each hand. There are many different models of these with various shapes and sizes, and you can separate them as much as you like. The normal advice is to set them up around shoulder width apart so you aren’t rounding your back to bring your arms together.
Most of these kinds of keyboards also support whatever key switches you prefer, and there are plenty of options that are sufficiently quiet for zoom (pretty much anything linear should do the trick)
I have been using a Moonlander for a couple of years now, and an EZ before that. They are expensive at around $400 but I don’t think I can ever go back. Most of these split keyboards also run QMK so you can setup binds, layers, and generally configure them however you like.
It took me years to find a split, ergo-style keyboard with mechanical keys, but I finally did. The freestyles have them, but I don't like the super flat layout.
I am loving the split, angled setup with my preferred Cherry MX Blues.
The MS Natural split keyboards are easy to find but aren't satisfyingly clicky mechanical keys just like olden times.
How long do these last?
Edit: "Ergonomic keyboard" https://en.wikipedia.org/wiki/Ergonomic_keyboard > #Split_keyboard:
> Split keyboards group keys into two or more sections. Ergonomic split keyboards can be fixed, where you cannot change the positions of the sections, or adjustable. Split keyboards typically change the angle of each section, and the distance between them. On an adjustable split keyboard, this can be tailored exactly to the user. People with a broad chest will benefit from an adjustable split keyboard's ability to customize the distance between the two halves of the board. This ensures the elbows are not too close together when typing. [2]
Waydroid – Run Android containers on Ubuntu
Doubts:
- Is it better/faster/more compatible than anbox?
- Runs on ARM?
- Does it allows me to watch DRM streaming services on my linux box?
- Can I install google play on it?
Waydroid has Android use your actual Linux kernel, so on an x86-64 host you’ll run x86-64 Android, and on an ARM host, ARM Android. This means that there will be some apps that won’t run on your Intel/AMD computer. I have no idea at all how common it is for Android apps to be tied to ARM, but I imagine that ARM64 will have helped with architecture-neutrality.
> This means that there will be some apps that won’t run on your Intel/AMD computer.
Should be able to still run them if you use binfmt_misc. Of course it will be slower but it's possible.
> binfmt_misc
https://en.wikipedia.org/wiki/Binfmt_misc
> binfmt_misc can also be combined with QEMU to execute programs for other processor architectures as if they were native binaries.[9]
QEMU supported [ARM guest] machines: https://wiki.qemu.org/Documentation/Platforms/ARM#Supported_...
Edit: from "Running and Building ARM Docker Containers on x86" (which also describes how to get CUDA working) https://www.stereolabs.com/docs/docker/building-arm-containe... :
sudo apt-get install qemu binfmt-support qemu-user-static # Install the qemu packages
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes # Execute the registering scripts
docker run --rm -t arm64v8/ubuntu uname -m # Test the emulation environment
> multiarch/qemu-user-static is to enable an execution of different multi-architecture containers by QEMU [1] and binfmt_misc [2]. Here are examples with Docker [3].
Why the heck isn't there just an official Android container and/or a LineageOS container?
It's not a certified device, so.
There are a number of ways to build "multi-arch docker images" e.g. for both x86 and ARM: OCI, docker build, podman build, buildx, buildah.
Containers are testable.
Here's this re: whether the official OpenWRT container should run /sbin/init in order to run procd, ubusd,: https://github.com/docker-library/official-images/pull/7975#...
AFAIU, from a termux issue thread re: repackaging everything individually, latest Android requires binaries to be installed from APKs to get the SELinux context label necessary to run?
Biologists Rethink the Logic Behind Cells’ Molecular Signals
I think (hope) that the deterministic model known as lock and key has been known to be a flawed view for quite some time. Books published in the early 00s (notably "Ni Dieu ni gène", by Kupiec and Sonigo) were already making this point in a popular science format, and explaining that even the concept of cells exchanging signals was flawed.
A cell has no evolutionary reason to transmit signals. It will however eat molecules it can use, and excrete molecules that are no longer needed, because this allows the cell to survive and reproduce. A white blood cell eating a bacteria doesn't do so with an intention to protect some organ somewhere in the body, it does so like predator eats prey. Leukocytes who eat well, ie face a bacteria they can eat, then multiply, and end up eating all the bacteria, before dying off when there's nothing more to eat. So they protect the body and then remove themselves, not because they have the intention to do it for the greater good, or because they received a signal, but because they have evolved to prey on bacteria within the ecosystem of the body.
Thinking of the body as a well ordered mechanism is a flawed view, there are no locks and keys, and most likely very few signals if any. Thinking of the body as a dynamically balanced ecosystem seems much closer to how cells behave and to the fantastically complex feedback systems that have evolved over eons and are now balancing the populations of cells in our bodies.
We may be ecosystems of individually oblivious and dumb little cells, but isn't it wondrous that from this emerges a complexity that can say "I" and has consciousness of self?
But reproduction is a fundamental aspect of evolution, and (most?) cells in the body don't reproduce on their own, but rather they are manufactured. T cells for example are manufactured by bone marrow and the thymus multiplies them if I'm understanding correctly. So I'm not sure how that fits in with the view you explained in your post.
In other words the T cell doesn't get feedback on its own fitness. The fitness feedback is at the level of the reproductive success of a human being (healthy humans can have more kids than sick ones), not the reproductive success of a T cell (because it has no reproductive capabilities and therefore cannot be subject to selection pressures). Correct me if I'm wrong.
Most cells or matter in the body?
From https://www.nature.com/articles/nature.2016.19136 :
> A 'reference man' (one who is 70 kilograms, 20–30 years old and 1.7 metres tall) contains on average about 30 trillion human cells and 39 trillion bacteria, […] Those numbers are approximate — another person might have half as many or twice as many bacteria, for example — but far from the 10:1 ratio commonly assumed.
Symbiosis https://en.wikipedia.org/wiki/Symbiosis :
> Symbiosis […] is any type of a close and long-term biological interaction between two different biological organisms, be it mutualistic, commensalistic, or parasitic. […]
> Symbiosis can be obligatory, which means that one or more of the symbionts depend on each other for survival, or facultative (optional), when they can generally live independently. […]
> Symbiosis is also classified by physical attachment. When symbionts form a single body it is called conjunctive symbiosis, while all other arrangements are called disjunctive symbiosis.[3] When one organism lives on the surface of another, such as head lice on humans, it is called ectosymbiosis; when one partner lives inside the tissues of another, such as Symbiodinium within coral, it is termed endosymbiosis.
Endosymbiont: https://en.wikipedia.org/wiki/Endosymbiont :
> Two major types of organelle in eukaryotic cells, mitochondria and plastids such as chloroplasts, are considered to be bacterial endosymbionts.[6] This process is commonly referred to as symbiogenesis.
Symbiogenesis: https://en.wikipedia.org/wiki/Symbiogenesis #Secondary_endosymbiosis ... Viral eukaryogenesis: https://en.wikipedia.org/wiki/Viral_eukaryogenesis :
> A number of precepts in the theory are possible. For instance, a helical virus with a bilipid envelope bears a distinct resemblance to a highly simplified cellular nucleus (i.e., a DNA chromosome encapsulated within a lipid membrane). In theory, a large DNA virus could take control of a bacterial or archaeal cell. Instead of replicating and destroying the host cell, it would remain within the cell, thus overcoming the tradeoff dilemma typically faced by viruses. With the virus in control of the host cell's molecular machinery, it would effectively become a functional nucleus. Through the processes of mitosis and cytokinesis, the virus would thus recruit the entire cell as a symbiont—a new way to survive and proliferate.
T-Cell # Activation: https://en.wikipedia.org/wiki/T_cell#Activation
> Both are required for production of an effective immune response; in the absence of co-stimulation, T cell receptor signalling alone results in anergy. […]
> Once a T cell has been appropriately activated (i.e. has received signal one and signal two) it alters its cell surface expression of a variety of proteins.
T-cell receptor § Signaling pathway: https://en.wikipedia.org/wiki/T-cell_receptor#Signaling_path...
Co-stimulation : https://en.wikipedia.org/wiki/Co-stimulation :
> Co-stimulation is a secondary signal which immune cells rely on to activate an immune response in the presence of an antigen-presenting cell.[1] In the case of T cells, two stimuli are required to fully activate their immune response. During the activation of lymphocytes, co-stimulation is often crucial to the development of an effective immune response. Co-stimulation is required in addition to the antigen-specific signal from their antigen receptors.
Anergy: https://en.wikipedia.org/wiki/Clonal_anergy :
> [Clonal] Anergy is a term in immunobiology that describes a lack of reaction by the body's defense mechanisms to foreign substances, and consists of a direct induction of peripheral lymphocyte tolerance. An individual in a state of anergy often indicates that the immune system is unable to mount a normal immune response against a specific antigen, usually a self-antigen. Lymphocytes are said to be anergic when they fail to respond to their specific antigen. Anergy is one of three processes that induce tolerance, modifying the immune system to prevent self-destruction (the others being clonal deletion and immunoregulation ).[1]
Clonal deletion: https://en.wikipedia.org/wiki/Clonal_deletion :
> There are millions of B and T cells inside the body, both created within the bone marrow and the latter matures in the thymus, hence the T. Each of these lymphocytes express specificity to a particular epitope, or the part of an antigen to which B cell and T cell receptors recognize and bind. There is a large diversity of epitopes recognized and, as a result, it is possible for some B and T lymphocytes to develop with the ability to recognize self.[4] B and T cells are presented with self antigen after developing receptors while they are still in the primary lymphoid organs.[3][4] Those cells that demonstrate a high affinity for this self antigen are often subsequently deleted so they cannot create progeny, which helps protect the host against autoimmunity.[2][3] Thus, the host develops a tolerance for this antigen, or a self tolerance.[3]
"DNA threads released by activated CD4+ T lymphocytes provide autocrine costimulation" (2019) https://www.pnas.org/content/116/18/8985
> A growing body of literature has shown that, aside from carrying genetic information, both nuclear and mitochondrial DNA can be released by innate immune cells and promote inflammatory responses. Here we show that when CD4+ T lymphocytes, key orchestrators of adaptive immunity, are activated, they form a complex extracellular architecture composed of oxidized threads of DNA that provide autocrine costimulatory signals to T cells. We named these DNA extrusions “T helper-released extracellular DNA” (THREDs).
FWIU, there's also a gut-brain pathway? Or is that also this "signaling method" for feedback in symbiotic complex dynamic systems?
From https://en.wikipedia.org/wiki/Complex_system :
> Complex systems are systems whose behavior is intrinsically difficult to model due to the dependencies, competitions, relationships, or other types of interactions between their parts or between a given system and its environment. Systems that are "complex" have distinct properties that arise from these relationships, such as nonlinearity, emergence, spontaneous order, adaptation, and feedback loops, among others. Because such systems appear in a wide variety of fields, the commonalities among them have become the topic of their independent area of research. In many cases, it is useful to represent such a system as a network where the nodes represent the components and links to their interactions.
Graph, Hypergraph, Property graph, Linked Data, AtomSpace, RDF* + SPARQL*, ONNX, {...}
> The term complex systems often refers to the study of complex systems, which is an approach to science that investigates how relationships between a system's parts give rise to its collective behaviors and how the system interacts and forms relationships with its environment.[1] The study of complex systems regards collective, or system-wide, behaviors as the fundamental object of study; for this reason, complex systems can be understood as an alternative paradigm to reductionism, which attempts to explain systems in terms of their constituent parts and the individual interactions between them.
A multi-digraph of probably nonlinear relations may not be the best way to describe the fields of even just a few electroweak magnets?
> As an interdisciplinary domain, complex systems draws contributions from many different fields, such as the study of self-organization and critical phenomena from physics, that of spontaneous order from the social sciences, chaos from mathematics, adaptation from biology, and many others. Complex systems is therefore often used as a broad term encompassing a research approach to problems in many diverse disciplines, including statistical physics, information theory, nonlinear dynamics, anthropology, computer science, meteorology, sociology, economics, psychology, and biology.
... Glossary of Systems Theory: https://en.wikipedia.org/wiki/Glossary_of_systems_theory
The Shunting-yard algorithm converts infix notation to RPN
RosettaCode has examples of the Shunting-Yard algorithm for parsing infix notation ((1+2)*3)^4 to an AST or just a stack of data and operators such as RPN: [ ]
Parsing/Shunting-yard algorithm: https://rosettacode.org/wiki/Parsing/Shunting-yard_algorithm
Parsing/RPN to infix conversion: https://rosettacode.org/wiki/Parsing/RPN_to_infix_conversion...
Applications: testing all combinations of operators with and without term grouping; parentheses; such as evolutionary algorithms or universal function approximaters that explore the space.
For example: https://github.com/westurner/notebooks/blob/gh-pages/maths/b... :
> This still isn't the complete set of possible solutions
[deleted]
How should logarithms be taught?
As one shape of a curve; in a notebook that demonstrates multiple methods of curve fitting with and without a logarithmic transform.
Logarithm: https://simple.wikipedia.org/wiki/Logarithm ; https://en.wikipedia.org/wiki/Logarithm :
> In mathematics, the logarithm is the inverse function to exponentiation. That means the logarithm of a given number x is the exponent to which another fixed number, the base b, must be raised, to produce that number x.
List of logarithmic identities: https://en.wikipedia.org/wiki/List_of_logarithmic_identities
List of integrals of logarithmic functions: https://en.wikipedia.org/wiki/List_of_integrals_of_logarithm...
As functions in a math library or a CAS that should implement the correct axioms correctly:
Sympy Docs > Functions > Contents: https://docs.sympy.org/latest/modules/functions/index.html#c...
sympy.functions.elementary.exponential. log(x, base=e) == log(x)/log(e), exp(), LambertW(), exp_polar() https://docs.sympy.org/latest/modules/functions/elementary.h...
"Exponential, Logarithmic and Trigonometric Integrals" sympy.functions.special.error_functions. Ei: exponential integral, li: logarithmic integral, Li: offset logarithmic integral https://docs.sympy.org/latest/modules/functions/special.html...
numpy.log. log() base e, log2(), log10(), log1p(x) == log(1 + x) https://numpy.org/doc/stable/reference/generated/numpy.log.h...
numpy.exp. exp(), expm1(x) == exp(x) - 1, exp2(x) == 2*x https://numpy.org/doc/stable/reference/generated/numpy.exp.h...
Khan Academy > Algebra 2 > Unit: Logarithms: https://www.khanacademy.org/math/algebra2/x2ec2f6f830c9fb89:...
Khan Academy > Algebra (all content) > Unit: Exponential & logarithmic functions https://www.khanacademy.org/math/algebra-home/alg-exp-and-lo...
3blue1brown: "Logarithm Fundamentals | Lockdown math ep. 6", "What makes the natural log "natural"? | Lockdown math ep. 7" https://www.youtube.com/playlist?list=PLZHQObOWTQDP5CVelJJ1b...
Feynmann Lectures 22-6: Algebra > Imaginary Exponents: https://www.feynmanlectures.caltech.edu/I_22.html#Ch22-S6
Power law functions: https://en.wikipedia.org/wiki/Power_law#Power-law_functions
In a two-body problem, of the 4-5 fundamental interactions: Gravity, Electroweak interaction, Strong interaction, Higgs interaction, a fifth force; which have constant exponential terms in their symbolic field descriptions? https://en.wikipedia.org/wiki/Fundamental_interaction#The_in...
Natural logs in natural systems:
Growth curve (biology) > Exponential growth: https://en.wikipedia.org/wiki/Growth_curve_(biology)#Exponen...
Basic reproduction number: https://en.wikipedia.org/wiki/Basic_reproduction_number
(... Growth hacking; awesome-grwoth-hacking: https://github.com/bekatom/awesome-growth-hacking )
Metcalf's law: https://en.wikipedia.org/wiki/Metcalfe%27s_law
Moore's law; doubling time: https://en.wikipedia.org/wiki/Moore's_law
A block reward halving is a doubling of difficulty. What block reward difficulty schedule would be a sufficient inverse of Moore's law?
A few queries:
logarithm cheatsheet https://www.google.com/search?q=logarithm+cheatsheet
logarithm on pinterest https://www.pinterest.com/search/pins/?q=logarithm
logarithm common core worksheet https://www.google.com/search?q=logarithm+common+core+worksh...
logarithm common core autograded exercise (... Khan Academy randomizes from a parametrized (?) test bank for unlimited retakes for Mastery Learning) https://www.google.com/search?q=logarithm+common+core+autogr...
If only I had started my math career with a binder of notebooks or at least 3-hole-punched notes.
- [ ] Create a git repo with an environment.yml that contains e.g. `mamba install -y jupyter-book jupytext jupyter_contrib_extensions jupyterlab-git nbdime jupyter_console pandas matplotlib sympy altair requests-html`, build a container from said repo with repo2docker, and git commit and push changes made from within the JupyterLab instance that repo2docker layers on top of your reproducible software dependency requirement specification ("REES"). {bash/zsh, git, docker, repo2docker, jupyter, [MyST] markdown and $$ mathTeX $$; Google Colab, Kaggle Kernels, ml-workspace, JupyterLite}
"How I'm able to take notes in mathematics lectures using LaTeX and Vim" https://news.ycombinator.com/item?id=19448678
Here's something like MyST Markdown or Rmarkdown for Jupyter-Book and/or jupytext:
## Log functions
Log functions in the {PyData} community
### LaTeX
#### sympy2latex
What e.g. sympy2latex parses that LaTeX into, in terms of symbolic objects in an expression tree:
### numpy
see above
### scipy
### sympy
see above
### sagemath
### statsmodels
### TensorFlow
### PyTorch
## Logarithmic and exponential computational complexity
- Docs: https://www.bigocheatsheet.com/
- [ ] DOC: Rank these with O(1) first: O(n log n), O(log n), O(1), O(n), O(n*2) +growthcurve +exponential
## Combinatorics, log, exp, and Shannon classical entropy and classical Boolean bits
https://www.google.com/search?q=formula+for+entropy :
S=k_{b}\ln\Omega
SI unit for [ ] entropy: joules per kelvin (J*K*-1)
*****
In terms of specifying tasks for myself in order to learn {Logarithms,} I could use e.g. todo.txt markup to specify tasks with [project and concept] labels and contexts; but todo.txt doesn't support nested lists like markdown checkboxes with todo.txt markup and/or codelabels (if it's software math)
- [ ] Read the Logarithms wikipedia page <url> and take +notes +math +logarithms @workstation
- [o] Read
- [x] BLD: mathrepo: generate from cookiecutter or nbdev
- [ ] DOC: mathrepo: logarithm notes
- [ ] DOC,ART: mathrepo: create exponential and logarithmic charts +logarithms @workstation
- [ ] ENH,TST,DOC: mathrepo: logarithms with stdlib math, numpy, sympy (and *pytest* or at least `assert` assertion expressions)
- [ ] ENH,TST,DOC: mathrepo: logarithms and exponents with NN libraries (and *pytest*)
### LaTeX
$$ \log_{b} x = (b^? = x) $$
$$ 2^3 = 8 $$
$$ \log_{2} 8 = 3 $$
$$ \ln e = 1 $$
$$ \log_b(xy)=\log_b(x)+\log_b(y) $$
$ \begin{align}
\textit{(1) } \log_b(xy) & = \log_b(x)+\log_b(y)
\end{align} $
#### sympy2latex
What e.g. sympy2latex parses that LaTeX into, in terms of symbolic objects in an expression tree:
# install
#!python -m pip install antlr4-python3-runtime sympy
#!mamba install -y -q antlr-python-runtime sympy
import sympy
from sympy.parsing.latex import parse_latex
def displaylatexexpr(latex):
expr = parse_latex(latex)
display(str(expr))
display(expr)
return expr
displaylatexexpr('\log_{2} 8'))
# 'log(8, 2)'
displaylatexexpr('\log_{2} 8 = 3'))
# 'Eq(log(8, 2), 3)'
displaylatexexpr('\log_b(xy) = \log_b(x)+\log_b(y)'))
# 'Eq(log(x*y, b), log(x, b) + log(y, b))'
displaylatexexpr('\log_{b} (xy) = \log_{b}(x)+\log_{b}(y)')
# 'Eq(log(x*y, b), log(x, b) + log(y, b))'
displaylatexexpr('\log_{2} (xy) = \log_{2}(x)+\log_{2}(y)')
# 'Eq(log(x*y, 2), log(x, 2) + log(y, 2))'
https://docs.python.org/3/library/operator.html#operator.pow
https://docs.python.org/3/library/math.html#power-and-logari...
math. exp(x), expm1(), log(x, base=e), log1p(x), log2(x), log10(x), pow(x, y) : float, assert sqrt() == pow(x, 1/2)
## scipy
https://docs.scipy.org/doc/scipy/reference/generated/scipy.s... scipy.special. xlog1py()
https://docs.scipy.org/doc/scipy/reference/generated/scipy.s...
### sagemath
https://doc.sagemath.org/html/en/reference/functions/sage/fu...
### statsmodels
### TensorFlow https://www.tensorflow.org/api_docs/python/tf/math tf.math. log(), log1P(), log_sigmoid(), exp(), expm1()
https://keras.io/api/layers/activations/
SmoothReLU ("softplus") adds ln to the ReLU activation function, for example: https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#So...
E.g. Softmax & LogSumExp also include natural logarithms in their definitions: https://en.wikipedia.org/wiki/Softmax_function
### PyTorch
https://pytorch.org/docs/stable/generated/torch.log.html torch. log(), log10(), log1p(), log2(), exp(), exp2(), expm1(); logaddexp() , logaddexp2(), logsumexp(), torch.special.xlog1py()
***
Regarding this learning process and these tools, Now I have a few replies to myself (!) in not-quite-markdown and with various headings: I should consolidate this information into a [MyST] markdown Jupyter Notebook and re-lead the whole thing. If this was decent markdown from the start, I'd have less markup work to do to create a ScholarlyArticle / Notebook.
Automatic cipher suite ordering in Go’s crypto/tls
Summary: The Golang team is deciding what ranked order TLS cipher suites should be used in. You are not able to decide what cipher suites to use, the Golang team sets that in the code and will update it as they see fit.
My take on this is that Filippo is taking a heavy handed approach here. This works for the majority of "dev write code fast ship it over the wall" scenarios. But it also falls apart in a couple scenarios:
* Companies/govt agencies which mandate the use of certain algorithms, such as RSA, and both sides of the connection are running Golang code. Now you need to vendor the TLS library to allow for what the company wants.
* If an issue is discovered with an algorithm it would be really nice to be able to turn off the algorithm in production until the issue can be patched. I'm not sure if this is possible with the current set of changes.
Any government agency that mandates use of specific TLS algorithms (like whitelist) almost certainly requires FIPS cryptography (or classified cryptography), so you won’t be using crypto/TLS anyway.
As a security/cryptography engineer, I love this change: for cryptography, it’s clear that more knobs == more problems.
As a developer, however, I dislike it: more knobs it’s easier to get the code to do what I need it to do.
Personally, I think it’s a good choice by Filippo.
From "Go Crypto and Kubernetes — FIPS 140–2 and FedRAMP Compliance" (2021) https://gokulchandrapr.medium.com/go-crypto-and-kubernetes-f... :
> If a vendor wants to supply cloud-based services to the US Federal Government, then they have to get FedRAMP approval. This certification process covers a whole host of security issues, but is very specific about its requirements on cryptography: usage of FIPS 140–2 validated modules wherever cryptography is needed, these encryption standards protect the cryptographic module from being cracked, altered, or otherwise tampered with. FIPS 140–2 validated encryption is a prerequisite for FedRAMP. [...]
> [...] Go Cryptography and Kubernetes — FIPS 140–2 Kubernetes is a Go project, as are most of the Kubernetes subcomponents and ecosystem. Golang has a crypto standard library, Golang Crypto which fulfills almost all the application crypto needs (TLS stack implementation for HTTPS servers and clients all the way to HMAC or any other primitive that are needed to make signatures to verify hashes, encrypt messages.). Go has made a different choice compared to most languages, which usually come with links or wrappers for OpenSSL or simply don’t provide any cryptography in the standard library (Rust doesn’t have standard library cryptography, JavaScript only has web crypto, Python doesn’t come with a crypto standard library). [...]
> The native go crypto is not FIPS compliant and there are few open proposals to facilitate Go code to meet FIPS requirements. Users can use prominent go compilers/toolsets backed by FIPS validated SSL libraries provided by Google or Redhat which enables Go to bypass the standard library cryptographic routines and instead call into a FIPS 140–2 validated cryptographic library. These toolsets are available as container images, where users can use the same to compile any Go based applications. [...]
> When a RHEL system is booted in FIPS mode, Go will instead call into OpenSSL via a new package that bridges between Go and OpenSSL. This also can be manually enabled by setting `GOLANG_FIPS=1`. The Go Toolset is available as a container image that can be downloaded from Red Hat Container Registry. Red Hat mentions that this as a new feature built on top of existing upstream work (BoringSSL). [...]
> To be FIPS 140–2 compliant, the module must use FIPS 140–2 complaint algorithms, ciphers, key establishment methods, and other protection profiles.
> FIPS-approved algorithms do change at times; not extremely frequently, but more often than they come out with a new version of FIPS 140. [...]
> Some of the fundamental requirements (not limited to) are as follows:
> [...] Support for TLS 1.0 and TLS 1.1 is now deprecated (only allowed in certain cases). TLS 1.3 is the preferred option, while TLS 1.2 is only tolerated.
> [...] DSA/RSA/ECDSA are only approved for key generation/signature.
> [...] The 0-RTT option in TLS 1.3 should be avoided.
Was there lag between the release of TLS 1.3 and an updated release of FIPS 140? @18f @DefenseDigital Can those systems be upgraded as easily?
NIST just moves slowly is all.
Common Criteria (NIAP) still only allows TLSv1.2 and TLSv1.1.
Scikit-Learn Version 1.0
There are scikit-learn (sklearn) API-compatible wrappers for e.g. PyTorch and TensorFlow.
Skorch: https://github.com/skorch-dev/skorch
tf.keras.wrappers.scikit_learn: https://www.tensorflow.org/api_docs/python/tf/keras/wrappers...
AFAIU, there are not Yellowbrick visualizers for PyTorch or TensorFlow; though PyTorch abd TensorFlow work with TensorBoard for visualizing CFG execution.
> Many machine learning libraries implement the scikit-learn `estimator API` to easily integrate alternative optimization or decision methods into a data science workflow. Because of this, it seems like it should be simple to drop in a non-scikit-learn estimator into a Yellowbrick visualizer, and in principle, it is. However, the reality is a bit more complicated.
> Yellowbrick visualizers often utilize more than just the method interface of estimators (e.g. `fit()` and `predict()`), relying on the learned attributes (object properties with a single underscore suffix, e.g. `coef_`). The issue is that when a third-party estimator does not expose these attributes, truly gnarly exceptions and tracebacks occur. Yellowbrick is meant to aid machine learning diagnostics reasoning, therefore instead of just allowing drop-in functionality that may cause confusion, we’ve created a wrapper functionality that is a bit kinder with it’s messaging.
Looks like there are Yellowbrick wrappers for XGBoost, CatBoost, CuML, and Spark MLib; but not for NNs yet. https://www.scikit-yb.org/en/latest/api/contrib/wrapper.html...
From the RAPIDS.ai CuML team: https://docs.rapids.ai/api/cuml/stable/ :
> cuML is a suite of fast, GPU-accelerated machine learning algorithms designed for data science and analytical tasks. Our API mirrors Sklearn’s, and we provide practitioners with the easy fit-predict-transform paradigm without ever having to program on a GPU.
> As data gets larger, algorithms running on a CPU becomes slow and cumbersome. RAPIDS provides users a streamlined approach where data is intially loaded in the GPU, and compute tasks can be performed on it directly.
CuML is not an NN library; but there are likely performance optimizations from CuDF and CuML that would accelerate performance of NNs as well.
Dask ML works with models with sklearn interfaces, XGBoost, LightGBM, PyTorch, and TensorFlow: https://ml.dask.org/ :
> Scikit-Learn API
> In all cases Dask-ML endeavors to provide a single unified interface around the familiar NumPy, Pandas, and Scikit-Learn APIs. Users familiar with Scikit-Learn should feel at home with Dask-ML.
dask-labextension for JupyterLab helps to visualize Dask ML CFGs which call predictors and classifiers with sklearn interfaces: https://github.com/dask/dask-labextension
well, this is realy the reason i asked my question.
It is a pain to move in and out of scikit and write all these wrappers and converters. I would prefer to do more in one framework.
For example, doing hyperparameter optimization in pytorch using scikit can be a bit painful sometimes
Ctrl-F automl https://westurner.github.io/hnlog/
> /? hierarchical automl "sklearn" site:github.com : https://www.google.com/search?q=hierarchical+automl+%22sklea...
https://westurner.github.io/hnlog/#comment-18798244
> Dask-ML works with {scikit-learn, xgboost, tensorflow, TPOT,}. ETL is your responsibility. Loading things into parquet format affords a lot of flexibility in terms of (non-SQL) datastores or just efficiently packed files on disk that need to be paged into/over in RAM. (Edit)
scale-scikit-learn https://examples.dask.org/machine-learning/scale-scikit-lear... -> dask.distributed parallel predication: https://examples.dask.org/machine-learning/parallel-predicti...
"Hyperparameter optimization with Dask" https://examples.dask.org/machine-learning/hyperparam-opt.ht...
> Sklearn.pipeline.Pipeline API: {fit(), transform(), predict(), score(),} https://scikit-learn.org/stable/modules/generated/sklearn.pi... : ```
decision_function(X) # Apply transforms, and decision_function of the final estimator
fit(X[, y]) # Fit the model
fit_predict(X[, y]) # Applies fit_predict of last step in pipeline after transforms.
fit_transform(X[, y]) # Fit the model and transform with the final estimator
get_params([deep]) # Get parameters for this estimator.
predict(X, *predict_params) # Apply transforms to the data, and predict with the final estimator
predict_log_proba(X) # Apply transforms, and predict_log_proba of the final estimator
predict_proba(X) # Apply transforms, and predict_proba of the final estimator
score(X[, y, sample_weight]) # Apply transforms, and score with the final estimator
score_samples(X) # Apply transforms, and score_samples of the final estimator.
set_params(**kwargs) # Set the parameters of this estimator
```
> https://docs.featuretools.com can also minimize ad-hoc boilerplate ETL / feature engineering :
>> Featuretools is a framework to perform automated feature engineering. It excels at transforming temporal and relational datasets into feature matrices for machine learning
From https://featuretools.alteryx.com/en/stable/guides/using_dask... :
> Creating a feature matrix from a very large dataset can be problematic if the underlying pandas dataframes that make up the entities cannot easily fit in memory. To help get around this issue, Featuretools supports creating Entity and EntitySet objects from Dask dataframes. A Dask EntitySet can then be passed to featuretools.dfs or featuretools.calculate_feature_matrix to create a feature matrix, which will be returned as a Dask dataframe. In addition to working on larger than memory datasets, this approach also allows users to take advantage of the parallel and distributed processing capabilities offered by Dask
Signed Exchanges on Google Search
From https://blog.cloudflare.com/automatic-signed-exchanges/ :
> The broader implication of SXGs is that they make content portable: content delivered via an SXG can be easily distributed by third parties while maintaining full assurance and attribution of its origin. Historically, the only way for a site to use a third party to distribute its content while maintaining attribution has been for the site to share its SSL certificates with the distributor. This has security drawbacks. Moreover, it is a far stretch from making content truly portable.
> In the long-term, truly portable content can be used to achieve use cases like fully offline experiences. In the immediate term, the primary use case of SXGs is the delivery of faster user experiences by providing content in an easily cacheable format. Specifically, Google Search will cache and sometimes prefetch SXGs. For sites that receive a large portion of their traffic from Google Search, SXGs can be an important tool for delivering faster page loads to users.
> It’s also possible that all sites could eventually support this standard. Every time a site is loaded, all the linked articles could be pre-loaded. Web speeds across the board would be dramatically increased.
"Signed HTTP Exchanges" draft-yasskin-http-origin-signed-responses https://wicg.github.io/webpackage/draft-yasskin-http-origin-...
"Bundled HTTP Exchanges" draft-yasskin-wpack-bundled-exchanges https://wicg.github.io/webpackage/draft-yasskin-wpack-bundle... :
> Web bundles provide a way to bundle up groups of HTTP responses, with the request URLs and content negotiation that produced them, to transmit or store together. They can include multiple top-level resources with one identified as the default by a primaryUrl metadata, provide random access to their component exchanges, and efficiently store 8-bit resources.
From https://web.dev/web-bundles/ :
> Introducing the Web Bundles API. A Web Bundle is a file format for encapsulating one or more HTTP resources in a single file. It can include one or more HTML files, JavaScript files, images, or stylesheets.
> Web Bundles, more formally known as Bundled HTTP Exchanges, are part of the Web Packaging proposal.
> HTTP resources in a Web Bundle are indexed by request URLs, and can optionally come with signatures that vouch for the resources. Signatures allow browsers to understand and verify where each resource came from, and treats each as coming from its true origin. This is similar to how Signed HTTP Exchanges, a feature for signing a single HTTP resource, are handled.
AlphaGo documentary (2020) [video]
I'm curious to ask what's next for this series of AI at Deepmind? Is there other more challenging problems they are tackling at the moment? I read they have already master StarCraft 2 even.
Is it the case that they have stopped this series of AI and going all in on protein folding at the moment?
This blog post mentions testing on Atari and being applied to chemistry and quantum physics
https://deepmind.com/blog/article/muzero-mastering-go-chess-...
AlphaFold 2 solved the CASP protein folding problem that AFAIU e.g. Folding@home et. al have been churning at for awhile FWIU. From November 2020: https://deepmind.com/blog/article/alphafold-a-solution-to-a-...
https://en.wikipedia.org/wiki/AlphaFold#SARS-CoV-2 :
> AlphaFold has been used to a predict structures of proteins of SARS-CoV-2, the causative agent of COVID-19 [...] The team acknowledged that though these protein structures might not be the subject of ongoing therapeutical research efforts, they will add to the community's understanding of the SARS-CoV-2 virus.[74] Specifically, AlphaFold 2's prediction of the structure of the ORF3a protein was very similar to the structure determined by researchers at University of California, Berkeley using cryo-electron microscopy. This specific protein is believed to assist the virus in breaking out of the host cell once it replicates. This protein is also believed to play a role in triggering the inflammatory response to the infection (... Berkeley ALS and SLAC beamlines ... S309 & Sotrovimab: https://scitechdaily.com/inescapable-covid-19-antibody-disco... )
Is there yet an open implementation of AlphaFold 2? edit: https://github.com/search?q=alphafold ... https://github.com/deepmind/alphafold
How do I reframe this problem in terms of fundamental algorithmic complexity classes (and thus the Quantum Algorithm Zoo thing that might optimize the currently fundamentally algorithmically computationally hard part of the hot loop that is the cost driver in this implementation)?
To cite in full from the MuZero blog post from December 2020: https://deepmind.com/blog/article/muzero-mastering-go-chess-... :
> Researchers have tried to tackle this major challenge in AI by using two main approaches: lookahead search or model-based planning.
> Systems that use lookahead search, such as AlphaZero, have achieved remarkable success in classic games such as checkers, chess and poker, but rely on being given knowledge of their environment’s dynamics, such as the rules of the game or an accurate simulator. This makes it difficult to apply them to messy real world problems, which are typically complex and hard to distill into simple rules.
> Model-based systems aim to address this issue by learning an accurate model of an environment’s dynamics, and then using it to plan. However, the complexity of modelling every aspect of an environment has meant these algorithms are unable to compete in visually rich domains, such as Atari. Until now, the best results on Atari are from model-free systems, such as DQN, R2D2 and Agent57. As the name suggests, model-free algorithms do not use a learned model and instead estimate what is the best action to take next.
> MuZero uses a different approach to overcome the limitations of previous approaches. Instead of trying to model the entire environment, MuZero just models aspects that are important to the agent’s decision-making process. After all, knowing an umbrella will keep you dry is more useful to know than modelling the pattern of raindrops in the air.
> Specifically, MuZero models three elements of the environment that are critical to planning:
> * The value: how good is the current position?
> * The policy: which action is the best to take?
> * The reward: how good was the last action?
> These are all learned using a deep neural network and are all that is needed for MuZero to understand what happens when it takes a certain action and to plan accordingly.
> Illustration of how Monte Carlo Tree Search can be used to plan with the MuZero neural networks. Starting at the current position in the game (schematic Go board at the top of the animation), MuZero uses the representation function (h) to map from the observation to an embedding used by the neural network (s0). Using the dynamics function (g) and the prediction function (f), MuZero can then consider possible future sequences of actions (a), and choose the best action.
> MuZero uses the experience it collects when interacting with the environment to train its neural network. This experience includes both observations and rewards from the environment, as well as the results of searches performed when deciding on the best action.
> During training, the model is unrolled alongside the collected experience, at each step predicting the previously saved information: the value function v predicts the sum of observed rewards (u), the policy estimate (p) predicts the previous search outcome (π), the reward estimate r predicts the last observed reward (u). This approach comes with another major benefit: MuZero can repeatedly use its learned model to improve its planning, rather than collecting new data from the environment. For example, in tests on the Atari suite, this variant - known as MuZero Reanalyze - used the learned model 90% of the time to re-plan what should have been done in past episodes.
FWIU, from what's going on over there:
AlphaGo => AlphaGo {Fan, Lee, Master, Zero} => AlphaGoZero => AlphaZero => MuZero
AlphaGo: https://en.wikipedia.org/wiki/AlphaGo_Zero
AlphaZero: https://en.wikipedia.org/wiki/AlphaZero
MuZero: https://en.wikipedia.org/wiki/MuZero
AlphaFold {1,2}: https://en.wikipedia.org/wiki/AlphaFold
IIRC, there is not an official implementation of e.g. AlphaZero or MuZero with e.g. openai/gym (and openai/retro) for comparing reinforcement learning algorithms? https://github.com/openai/gym
What are the benchmarks for Applied RL?
From https://news.ycombinator.com/item?id=28499001 :
> AFAIU, while there are DLTs that cost CPU, RAM, and Data storage between points in spacetime, none yet incentivize energy efficiency by varying costs depending upon whether the instructions execute on a FPGA, ASIC, CPU, GPU, TPU, or QPU? [...]
> To be 200% green - to put a 200% green footer with search-discoverable RDFa on your site - I think you need PPAs and all directly sourced clean energy.
> (Energy efficiency is very relevant to ML/AI/AGI, because while it may be the case that the dumb universal function approximator will eventually find a better solution, "just leave it on all night/month/K12+postdoc" in parallel is a very expensive proposition with no apparent oracle; and then to ethically filter solutions still costs at least one human)
I'm seeing a lot of cool stuff that people start to build on AlphaFold lately:
- ChimeraX: https://www.youtube.com/watch?v=le7NatFo8vI
Libraries.io indexes software dependencies; but no Dependent packages or Dependent repositories are yet listed for the pypi:alphafold package: https://libraries.io/pypi/alphafold
The GitHub network/dependents view currently lists one repo that depends upon deepmind/alphafold: https://github.com/deepmind/alphafold/network/dependents
(Linked citations for science: How to cite a schema:SoftwareApplication in a schema:ScholarlyArticle , How to cite a software dependency in a dependency specification parsed by e.g. Libraries.io and/or GitHub. e.g. FigShare and Zenodo offer DOIs for tags of git repos, that work with BinderHub and repo2docker and hopefully someday repo2jupyterlite. https://westurner.github.io/hnlog/#comment-24513808 )
/?gscholar alphafold: https://scholar.google.com/scholar?q=alphafold
On a Google Scholar search result page, you can click "Cited by [ ]" to check which documents contain textual and/or URL citations gscholar has parsed and identified as indicating a relation to a given ScholarlyArticle.
/?sscholar alphafold: https://www.semanticscholar.org/search?q=alphafold
On a Semantic Scholar search result page, you can click the "“" to check which documents contain textual and/or URL citations Semantic Scholar has parsed and identified as indicating a relation to a given ScholarlyArticle.
/?smeta alphafold: https://www.meta.org/search?q=t---alphafold
On a Meta.org search result page, you can click the article title and scroll down to "Citations" to check which documents contain textual and/or URL citations Meta has parsed and identified as indicating a relation to a given ScholarlyArticle.
Do any of these use structured data like https://schema.org/ScholarlyArticle ? (... https://westurner.github.io/hnlog/#comment-28495597 )
Interpretable Model-Based Hierarchical RL Using Inductive Logic Programming
I don't work in the field, but I sort of passively follow it.
A year ago I made this comment, in another ML thread:
https://news.ycombinator.com/item?id=23315739
"I often wonder about whether neural networks might need to meet at a crossroads with other techniques."
"Inductive Logic/Answer Set Programming or Constraints Programming seems like it could be a good match for this field. Because from my ignorant understanding, you have a more "concrete" representation of a model/problem in the form of symbolic logic or constraints and an entirely abstract "black box" solver with neural networks. I have no real clue, but it seems like they could be synergistic?"
I've been thinking along the same lines, it seems like logic + ML would complement each other well. Acquiring trustworthy labeled data is "THE" problem in ML, and figuring out which predicates to string together is "THE" problem in logic programming, seems like a perfect match.
A logic program can produce a practically infinite number of perfectly consistent test cases for the ML model to learn from, and the ML model can predict which problem should be solved. I'd like to see a conversational interface that combines these two systems, ML generates logic statements and observes the results, repeat. That might help to keep it from going off the rails like a long GPT-3 session tends to do.
AutoML is RL? The entire exercise of publishing and peer review is an exercise in cybernetics?
https://en.wikipedia.org/wiki/Probabilistic_logic_network :
> The basic goal of PLN is to provide reasonably accurate probabilistic inference in a way that is compatible with both term logic and predicate logic, and scales up to operate in real time on large dynamic knowledge bases.
> The goal underlying the theoretical development of PLN has been the creation of practical software systems carrying out complex, useful inferences based on uncertain knowledge and drawing uncertain conclusions. PLN has been designed to allow basic probabilistic inference to interact with other kinds of inference such as intensional inference, fuzzy inference, and higher-order inference using quantifiers, variables, and combinators, and be a more convenient approach than Bayesian networks (or other conventional approaches) for the purpose of interfacing basic probabilistic inference with these other sorts of inference. In addition, the inference rules are formulated in such a way as to avoid the paradoxes of Dempster–Shafer theory.
Has anybody already taught / reinforced an OpenCog [PLN, MOSES] AtomSpace hypergraph agent to do Linked Data prep and also convex optimization with AutoML and better than grid search so gradients?
Perhaps teaching users to bias analyses with e.g. Yellowbrick and the sklearn APIs would be a good curriculum traversal?
opening/baselines "Logging and vizualizing learning curves and other training metrics" https://github.com/openai/baselines#logging-and-vizualizing-...
https://en.wikipedia.org/wiki/AlphaZero
There's probably an awesome-automl by now? Again, the sklearn interfaces.
TIL that SymPy supports NumPy, PyTorch, and TensorFlow [Quantum; TFQ?]; and with a Computer Algebra System something for mutating the AST may not be necessary for symbolic expression trees without human-readable comments or symbol names? Lean mathlib: https://github.com/leanprover-community/mathlib , and then reasoning about concurrent / distributed systems (with side channels in actual physical component space) with e.g. TLA+.
There are new UUID formats that are timestamp-sortable; for when blockchain cryptographic hashes aren't enough entropy. "New UUID Formats – IETF Draft" https://news.ycombinator.com/item?id=28088213
... You can host online ML algos through SingularityNet, which also does PayPal now for the RL.
Our visual / auditory biological neural networks do appear to be hierarchical and relatively highly plastic as well.
If you're planning to mutate, crossover, and select expression trees, you'll need a survival function (~cost function) in order to reinforce; RL.
Blockchains cost immutable data storage with data integrity protections by the byte.
Smart contracts cost CPU usage with costed opcodes. eWASM (Ethereum WebAssembly) has costed opcodes for redundantly-executed smart contracts (that execute on n nodes of a shard) https://ewasm.readthedocs.io/en/mkdocs/determining_wasm_gas_...
AFAIU, while there are DLTs that cost CPU, RAM, and Data storage between points in spacetime, none yet incentivize energy efficiency by varying costs depending upon whether the instructions execute on a FPGA, ASIC, CPU, GPU, TPU, or QPU?
To be 200% green - to put a 200% green footer with search-discoverable RDFa on your site - I think you need PPAs and all directly sourced clean energy.
(Energy efficiency is very relevant to ML/AI/AGI, because while it may be the case that the dumb universal function approximator will eventually find a better solution, "just leave it on all night/month/K12+postdoc" in parallel is a very expensive proposition with no apparent oracle; and then to ethically filter solutions still costs at least one human)
> Perhaps teaching users to bias analyses with e.g. Yellowbrick and the sklearn APIs would be a good curriculum traversal?
Yellowbrick > Third Party Estimaters: (yellowbrick.contrib.wrapper: https://www.scikit-yb.org/en/latest/api/contrib/wrapper.html
From https://www.scikit-yb.org/en/latest/quickstart.html#using-ye... :
> The Yellowbrick API is specifically designed to play nicely with scikit-learn. The primary interface is therefore a Visualizer – an object that learns from data to produce a visualization. Visualizers are scikit-learn Estimator objects and have a similar interface along with methods for drawing. In order to use visualizers, you simply use the same workflow as with a scikit-learn model, import the visualizer, instantiate it, call the visualizer’s fit() method, then in order to render the visualization, call the visualizer’s show() method.
> For example, there are several visualizers that act as transformers, used to perform feature analysis prior to fitting a model. The following example visualizes a high-dimensional data set with parallel coordinates:
from yellowbrick.features import ParallelCoordinates
visualizer = ParallelCoordinates()
visualizer.fit_transform(X, y)
visualizer.show()
IIRC, some automl tools - which test various combinations of, stacks of, ensembles of e.g. Estimators - do test hierarchical ensembles? Are those 'piecewise' and ultimately not the unified theory we were looking for here either (but often a good enough, fast enough, sufficient approximate solution with a sufficiently low error term)?
/? hierarchical automl "sklearn" site:github.com : https://www.google.com/search?q=hierarchical+automl+%22sklea...
Ship / Show / Ask: A modern branching strategy
> The reason you’re reliant on a lot of “Asking” might be that you have trust issue. “All changes must be approved” or “Every pull request needs 2 reviewers” are common policies, but they show a lack of trust in the development team.
I'm not following. We do reviews because a second pair of eyes can spot things the author missed.
I've been labelled controversial (amongst other things..) in the past for stating this but hey ho, I'll take another leap:
Aside from the "small changes, fast fixes" type of mantra, and "pairing is better than reviewing" that I suspect Martin Fowler is leaning on (and that I both support strongly):
There is a difference between having reviews, and enforcing reviews.
In my multi-decade career I'm yet to have a signle instance of the thought "thank god we prohibit changes that haven't been reviewed by two other people"
But I have lost count the number of times I've had the thought "I need to bypass this policy because shit's on fire and I've got a fix"
Then comes the question of quality of review when they are mandated.
I'll interject with: Code reviews are no inherently bad. Feedback is good. Collaboration is good. There are better methods of providing feedback than code reviews (I strongly object to the post-facto nature of code reviews)
Mandated peer reviews less so. Feedback is hurried, if not entirely absent, as it becomes another task that people _have_ to do. The time spent reviewing is rarely accounted for. There's a lot of knowledge transfer required for any non-trivial review to be effective, further increasing the demand on the participants.
Code reviews are another tool in the box, but they should not be used for every job.
Where I currently work, we have "skip review" and "skip preflight" labels for this. The mergers have the power to merge anything anyway, the labels are only to make it an official request.
> Where I currently work, we have "skip review" and "skip preflight" labels for this. The mergers have the power to merge anything anyway, the labels are only to make it an official request.
From the OP:
> Changes are categorized as either Ship (merge into mainline without review), Show (open a pull request for review, but merge into mainline immediately), or Ask (open a pull request for discussion before merging).
Right, its like saying checklists are a sign of distrust. No, its an acknowledgement that good intentions dont fix human nature. Have an agreed upon process and stick to it.
Checklists are often a good thing; and an opportunity to optimize processes with team feedback!
"Post-surgical deaths in Scotland drop by a third, attributed to a checklist" https://news.ycombinator.com/item?id=19684376 https://westurner.github.io/hnlog/#comment-19684376
Show HN: TweeView – A Tree Visualisation of Twitter Conversations
It would be more useful if one could either follow a link to the replies, or read the full tweet text of the corresponding node by clicking on it.
Also the node div should have a background color, for when an image doesn't load.
As an aside, even though @paulgb says he's not bothered with the use of his library that way, it's so similar that I think common OSS etiquette would've been to acknowledge the original project more prominently.
Cheers. It is all open source on https://github.com/edent/TweeView - where I also credit Paul.
The main 2D view is based on his work - but the interactive view (and 3D views) are based on other projects.
Wireless Charging Power Side-Channel Attacks
The severity and number of side-channel attacks is slowly becoming cripplingly ridiculous. At what point do we just give up with side-channel attacks on consumer devices and assume the mentality that all consumer devices connected to the internet should be treated as insecure by default.
> assume the mentality that all consumer devices connected to the internet should be treated as insecure by default.
"Zero trust security model" https://en.wikipedia.org/wiki/Zero_trust_security_model :
> The main concept behind zero trust is that devices should not be trusted by default, even if they are connected to a managed corporate network such as the corporate LAN and even if they were previously verified.
Checked the power draw a couple years ago on reddit's new design, which was a huge increase from the old design. Should have written a paper with lots of clever words in it, could have been my first scientific contribution!
I didn't because this research has been done long ago and with rsa key operations, which are both more severe if you can capture it and much more difficult to measure. That you can see the difference between multi-megabyte JavaScript pages is a given at that point.
From https://planetfriendlyweb.org/mental-model :
> When you think about how a digital product or website creates an environmental impact, you can think of it creating it in three main ways - through the Packets of data it sends to users, the Platform the product runs on, and the Process used to make the product itself.
From https://sustainableux.com/talks/2018/how-to-build-a-planet-f... :
> SustainableUX: design vs. climate change. Online, Worldwide, Free. The online event for UX, front-end, and product people who want to make a positive impact—on climate-change, social equality, and inclusion
How We Proved the Eth2 Deposit Contract Is Free of Runtime Errors
You know, I'm not really into cryptocurrency, but it does seem like it's contributing to a resurgence of interest in formal methods. So - thank you, cryptocurrency community!
From "Discover and Prevent Linux Kernel Zero-Day Exploit Using Formal Verification" https://news.ycombinator.com/item?id=27442273 :
> [Coq, VST, CompCert]
> Formal methods: https://en.wikipedia.org/wiki/Formal_methods
> Formal specification: https://en.wikipedia.org/wiki/Formal_specification
> Implementation of formal specification: https://en.wikipedia.org/wiki/Anti-pattern#Software_engineer...
> Formal verification: https://en.wikipedia.org/wiki/Formal_verification
> From "Why Don't People Use Formal Methods?" https://news.ycombinator.com/item?id=18965964 :
>> Which universities teach formal methods?
>> - q=formal+verification https://www.class-central.com/search?q=formal+verification
>> - q=formal+methods https://www.class-central.com/search?q=formal+methods
>> Is formal verification a required course or curriculum competency for any Computer Science or Software Engineering / Computer Engineering degree programs?
To clarify, they implemented the algorithm in Dafny, and then proved that version correct. They did not verify code that will actually run in production.
From the paper:
> Dafny is a practical option for the verification of mission-critical smart contracts, and a possible avenue for adoption could be to extend the Dafny code generator engine to support Solidity … or to automatically translate Solidity into Dafny. We are currently evaluating these options
https://github.com/dafny-lang/dafny
Dafny Cheat Sheet: https://docs.google.com/document/d/1kz5_yqzhrEyXII96eCF1YoHZ...
Looks like there's a Haskell-to-Dafny converter.
Physics-Based Deep Learning Book
"Physics-based" Deep Learning seems like a misnomer. From the abstract "Deep Learning Applications for Physics" sounds more apt. There definitely is value in transferring standard terminology and methods from physics to deep learning. But from the preview it's unclear if that is the focus.
Indeed, “Deep Learning Based Physics” seems a little more correct.
The common terminology is Physics Informed Neural Networks (PINNs)
"Physics-informed neural networks" https://en.wikipedia.org/wiki/Physics-informed_neural_networ...
But what about statistical thermodynamics and information theory? What about thin film?
What are some applications for PINNs and for {DL, RL,} in physics?
Ask HN: Books that teach you programming languages via systems projects?
Looking for a book/textbook that teaches you a programming language through systems (or vice versa). For example, a book that teaches modern C++ by showing you how to program a compiler; a book that teaches operating systems and the language of choice in the book is Rust; a book that teaches database internals through Golang; etc. Basically, looking for a fun project-based book that I can walk through and spend my free time working through.
Any recommendations?
From "Ask HN: What are some books where the reader learns by building projects?" https://news.ycombinator.com/item?id=26042447 :
> "Agile Web Development with Rails [6]" (2020) teaches TDD and agile in conjunction with a DRY, CoC, RAD web application framework: https://g.co/kgs/GNqnWV
And:
> "ugit – Learn Git Internals by Building Git in Python" https://www.leshenko.net/p/ugit/
How you can track your personal finances using Python
> We take the output of the previous step, pipe everything over to our .beancount file, and "balance" transactions.
> Recall that the flow of money in double-entry accounting is represented using transactions involving at least two accounts. When you download CSVs from your bank, each line in that CSV represents money that's either incoming or outgoing. That's only one leg of a transaction (credit or debit). It's up to us to provide the other leg.
> This act is called "balancing".
Balance (accounting) https://en.wikipedia.org/wiki/Balance_(accounting)
Are unique record IDs necessary for this [financial] application? FWICS, https://plaintextaccounting.org/ just throws away the (probably per-institution) transaction IDs; like a non-reflexive logic that eschews Law of identity? Just grep and wc?
> What does the ledger look like?
> I wrote earlier that one of the main things that Beancount provides is a language specification for defining financial transactions in a plain-text format.
> What does this format look like? Here's a quick example:
option "title" "Alice"
option "operating_currency" "EUR"
; Accounts
2021-01-01 open Assets:MyBank:Checking
2021-01-01 open Expenses:Rent
2021-01-01 * "Landlord" "Thanks for the rent"
Assets:MyBank:Checking -1000.00 EUR
Expenses:Rent 1000.00 EUR
The star is just an 1-digit field for "flag". I don't think there are any defined semantics for the field but by convention 'star' means something 'no special flags on this transaction'.
People use other flags to indicate, for instance, whether a transaction has been reconciled, or cleared their bank, or whatnot.
a REST API such as: https://plaid.com/
Plaid has serious privacy and security concerns, so I would be careful:
From https://news.ycombinator.com/item?id=28203393 :
> No, your personal data is not sold or rented or given away or bartered to parties that are not Plaid, your bank, or the connected app. We talk about all of this in our privacy policy, including ways that data could be used — for example, with data processors/service providers (like AWS which hosts our services) for the purposes of running Plaid’s services or for a user’s connected app to provide their services.
>> I saw that. Thank you for your patience and persistence in responding to so many pointed questions.
>> For any interested, here is a link to relevant section of the referenced privacy policy: https://plaid.com/legal/#consumers
>> I am also impressed by the Legal Changelog on the same page that clearly lays out a log of changes made to privacy & other published legal documents.
The comments in those threads were more negative than positive, and the fact that Plaid paid a $58 million settlement for allegedly sharing personal banking data with third parties without consent is telling enough. I am not going to give my banking usernames and passwords to Plaid in plaintext, when their employee is arguing on HN over what the word "sold" means:
Are you making claims without evidence? Settling is not admission of guilt.
Banks should implement read-only OAuth APIs, so that users are not required to store their u/p/sqa answers.
From "Canada calls screen scraping ‘unsecure,’ sets Open Banking target for 2023" https://news.ycombinator.com/item?id=28229957 :
> AFAIU, there are still zero (0) consumer banking APIs with Read-Only e.g. OAuth APIs in the US as well?
Looks like there may be less than 3 so far.
> Banks could save themselves CPU, RAM, bandwidth, and liability by implementing read-only API tokens and methods that need only return JSON - instead of HTML or worse, monthly PDF tables for a fee - possibly similar to the Plaid API: https://plaid.com/docs/api/
> There is competition in consumer/retail banking, but still the only way to do e.g. budget and fraud analysis with third party apps is to give away all authentication factors: u/p/sqa; and TBH that's unacceptable.
> Traditional and distributed ledger service providers might also consider W3C ILP: Interledger Protocol (in starting their move to quantum-resistant ledgers by 2022 in order to have a 5 year refresh cycle before QC is a real risk by 2027, optimistically, for science) when reviewing the entropy of username+password_hash+security_question_answer strings in comparison to the entropy of cryptoasset account public key hash strings: https://interledger.org/developer-tools/get-started/overview...
> Are you making claims without evidence?
No. Plaid did agree to pay the $58 million, and the lawsuit was for alleged data sharing with third parties without user consent. I don't care if they admit guilt or not. They agreed to pay $58 million to end the lawsuit, and that does not engender trust. Shifting the blame to banks doesn't make Plaid any more reputable.
Providing usernames and passwords of sensitive accounts to a third party is a privacy and security risk, and Plaid has not earned enough trust from me to justify the risk I would need to assume to use their services.
How did their policies change before and after said settlement?
From https://my.plaid.com/help/360043065354-does-plaid-have-acces... :
> Does Plaid have access to my credentials?
> The type of connection Plaid has to your financial institution determines whether or not we have access to the login credentials for your financial account: your username and password.
> In many cases, when you link a financial institution to an app via Plaid, you provide your login credentials to us and we securely store them. We use those credentials to access and obtain information from your financial institution in order to provide that information, at your direction, to the apps and services you want to use. For more information on how we use your data, please refer to our End User Privacy Policy.
> In other cases, after you request that we link your financial institution to an app or service you want to use, you will be prompted to provide your login credentials directly to your financial institution––not to Plaid––and, upon successful authentication, your financial institution will then return your data to Plaid. In these cases, Plaid does not access or store your account credentials. Instead, your financial institution provides Plaid with a type of security identifier, which permits Plaid to securely reconnect to your financial institution at regularly scheduled intervals to keep your apps and services up-to-date.
> Regardless of which type of connection is made, we do not share your credentials with the apps or services you’ve linked to your financial institution via Plaid. You can read more about how Plaid handles data here.
What do you think this should say instead?
Do you think they use the same key to securely store all accounts, like ACH? Or no key, like the bank ledger that you're downloading a window of as CSV through hopefully a read-only SQL account, hopefully with data encrypted at rest and in motion.
When you download a CSV or a OFX to a local file, is the data then still encrypted at rest?
Again, US Banks can eliminate the need for {Plaid, Mint, } as the account data access middlemen by providing a read-only OAuth API. Because banks do not have a way to allow users to grant read-only access to their account ledgers, the only solution is to securely store the u/p/sqa. If you write a script to fetch your data and call it from cron, how can you decrypt the account credentials after an unattended reboot? When must a human enter key material to decrypt the stored u/p/sqa?
Here, we realize that banks should really have people that do infosec - that comprehend symmetric and assymetric cryptography - audits to point out these sorts of vulnerabilities and risks. And if they had kept current with the times, we would have a very different banking and finance information system architecture with fewer single points of failure.
I'm not interested in what Plaid puts in a help page, since Plaid's $58 million settlement is for alleged data sharing with third parties without consent, meaning that Plaid is accused of not properly communicating the alleged data sharing to its users or obtaining permission.
And Plaid's terms of service (https://plaid.com/legal/#how-we-use-your-information) contains vague catch-alls such as:
> We share your End User Information for a number of business purposes:
> With our data processors and other service providers, partners, or contractors in connection with the services they perform for us or developers
Sure, it would be great if banks offered different authentication systems, but that has nothing to do with my lack of trust for Plaid. A different authentication system wouldn't eliminate the data sharing concerns I have with Plaid.
CISA Lays Out Security Rules for Zero Trust Clouds
"Cloud Security Technical Reference Architecture (TRA)" (2021) https://cisa.gov/publication/cloud-security-technical-refere...
> The Cloud Security TRA provides agencies with guidance on the shared risk model for cloud service adoption (authored by FedRAMP), how to build a cloud environment (authored by USDS), and how to monitor such an environment through robust cloud security posture management (authored by CISA).
> Public Comment Period - NOW OPEN! CISA is releasing the Cloud Security TRA for public comment to collect critical feedback from agencies, industry, and academia to ensure the guidance fully addresses considerations for secure cloud migration. The public comment period begins Tuesday, September 7, 2021 and concludes on Friday, October 1, 2021. CISA is interested in gathering feedback focused on the following key questions: […]
"Zero Trust Maturity Model" (2021) https://cisa.gov/publication/zero-trust-maturity-model
> CISA’s Zero Trust Maturity Model is one of many roadmaps for agencies to reference as they transition towards a zero trust architecture. The goal of the maturity model is to assist agencies in the development of their zero trust strategies and implementation plans and present ways in which various CISA services can support zero trust solutions across agencies.
> The maturity model, which include five pillars and three cross-cutting capabilities, is based on the foundations of zero trust. Within each pillar, the maturity model provides agencies with specific examples of a traditional, advanced, and optimal zero trust architecture.
> Public Comment Period – NOW OPEN! CISA drafted the Zero Trust Maturity Model in June to assist agencies in complying with the Executive Order. While the distribution was originally limited to agencies, CISA is excited to release the maturity model for public comment.
> CISA is releasing the Zero Trust Maturity Model for public comment beginning Tuesday, September 7, 2021 and concludes on Friday, October 1, 2021. CISA is interested in gathering feedback focused on the following key questions: […]
"Zero trust security model": https://en.wikipedia.org/wiki/Zero_trust_security_model
Show HN: Heroku Alternative for Python/Django apps
I have been following this space alot. Heroku like deployment spaces have exploded recently partly because people miss that experience, partly because buildpack is opensource[1], partly because we have gone through a techshift where it is pretty easy to emulate that. There are two broad categories - 1. Dokku's of the world, one VM (very recently tried adding multi node) and then more recently getporter.dev[2] and okteto[3] of the world that is trying to replicate heroku on kubernetes.
One of things, I noticed is that, It looks really cool at first look but every organisation has their own complexity in deploying stuff. It is an integration hell, you have to provide white glove treatment to all customers and eventually you have a hard time scaling. We tried it too, and what we noticed if you have to go an enterprise and sell, you start doing leaky abstraction. It works really well for early stage startups but once they have the money you prefer something more customizable and robust solution.
This project looks cool. I do not want to discourage anyone but again, it is a red ocean out there.
The problem is the layering of the abstractions. Having PaaS on top of Kubernetes in a way that you move down to a more powerful primitive is going to be the most scalable.
What happened in the public cloud space was a disparate set of services which made the choices really costly. Azure went in heavy on PaaS while AWS was focused on VMs. The easy adoption and migration was on AWS. K8s seems like a reasonable abstraction that reduces the cost of these decisions because people seem to agree that K8s is a reasonable base.
Being able to slide on a compatible PaaS layer shouldn't negate that K8s investment, and make it easier to adopt for some organizations, or parts of those organizations. But, I wouldn't argue that we are the point where that layer is mature.
dokku-scheduler-kubernetes https://github.com/dokku/dokku-scheduler-kubernetes#function...
> The following functionality has been implemented: Deployment and Service annotations, Domain proxy support via the Nginx Ingress Controller, Environment variables, Letsencrypt SSL Certificate integration via CertManager, Pod Disruption Budgets, Resource limits and reservations (reservations == kubernetes requests), Zero-downtime deploys via Deployment healthchecks, Traffic to non-web containers (via a configurable list)
SPDX Becomes Internationally Recognized Standard for Software Bill of Materials
From OP:
> Between eighty and ninety percent (80%-90%) of a modern application is assembled from open source software components. An SBOM accounts for the software components contained in an application — open source, proprietary, or third-party — and details their provenance, license, and security attributes. SBOMs are used as a part of a foundational practice to track and trace components across software supply chains. SBOMs also help to proactively identify software issues and risks and establish a starting point for their remediation.
> SPDX results from ten years of collaboration from representatives across industries, including the leading Software Composition Analysis (SCA) vendors – making it the most robust, mature, and adopted SBOM standard.
https://en.wikipedia.org/wiki/Software_Package_Data_Exchange
Show HN: Arxiv.org on IPFS
Ah cool… I also took a stab at something similar several years ago: https://github.com/ecausarano/heron
Also at the time I was considering IPFS.
But I guess the real trick is implementing a WOT to implement peer review and filter out the inevitable junk that will be published
"Help compare Comment and Annotation services: moderation, spam, notifications, configurability" executablebooks/meta#102 https://github.com/executablebooks/meta/discussions/102 :
> jupyter-comment supports a number of commenting services [...]. In helping users decide which commenting and annotation services to include on their pages and commit to maintaining, could we discuss criteria for assessment and current features of services?
> Possible features for comparison:
> * Content author can delete / hide
> * Content author can report / block
> * Comments / annotations are screened by spam-fighting service
> * Content / author can label as e.g. toxic
> * Content author receives notification of new comments
> * Content author can require approval before user-contributed content is publicly-visible
> * Content author may allow comments for a limited amount of time (probably more relevant to BlogPostings)
> * Content author may simultaneously denounce censorship in all it's forms while allowing previously-published works to languish
#ForScience
FWIW, archiving repo2docker-compatible git repos with a DOI attached to a git tag, is possible with JupyterLite:
> JupyterLite is a JupyterLab distribution that runs entirely in the browser built from the ground-up using JupyterLab components and extensions
With JupyterLite, you can build a static archive of a repo2docker-like environment so that the ScholarlyArticle notebook or computer modern latex css, its SoftwareRelease dependencies, and possibly also the Datasets can be run in a browser tab with WASM. HTML + JS + WASM
New Texas Abortion Law Likely to Unleash a Torrent of Lawsuits Against Education
All this talk about Austin becoming the new SV - but then you see regressive laws like this getting pushed through. No thanks.
Migration of people from SV to Austin will change the political landscape and stop stuff like this from being passed.
You don't have to pay prisoners for work in Texas.
Unfortunately, I find it necessary to help elucidate some moral issues that recent legal developments in one state of the Union have brought to light.
Such a God is correctly assessed as hostile to the patrons of Earth.
Such a creation myth involving seven (7) days to create Heaven, Hell Lite, and Hell (Heaven, Earth, and Hell) is what we must interpret.
At issue is whether the singular, supernatural God in Heaven is Benevolent (Good) and Omnipotent (All-knowing). Why would a loving God create Heaven without suffering on the first day, and then create Earth etc. with suffering on the succeeding days? Why not create Heaven for all? Why send infants into a life of suffering on Earth (Hell Lite) when they could be sent directly to Heaven with no suffering?
Ergo, God's plan is to force babies to suffer on Earth for awhile; but if they die sinless, then they go to Heaven where parents aren't needed. Ergo, Heaven is full of parentless dead babies.
You've made asses of yourselfs in trying to manipulate others into your golden candlestick tax districts, where the whole congregation does not have healthcare.
God created a world in need of healthcare, and isn't getting healthcare to it.
Ergo, because God created a world of suffering for us - and babies - god is not Friendly. And if god is not friendly, god is a very poor basis for our morality, and for our legal system.
Case in point: did god ask for Mary's (or even Joseph's) consent?
A benevolent god would design in required consent.
In the Christian bible, furthermore, god tells them to kill even the enemies' women and unborn children. Ergo, god does not value the sanctity of infants' lives over other concerns.
For citations of scripture, see e.g. Freedom from Religion Foundation > What Does the Bible Really Say About Abortion? https://ffrf.org/component/k2/item/25602-abortion-rights
A very poor basis for morality and law, indeed.
What is "Original Sin" if not a suggestion that we should choose very wisely?
And when they rolled that [silver] rock from the tomb, behold what did they find? No dead baby; for the dead baby of god was in heaven, gentlemen. Sinless dead babies all go to heaven, gentlemen.
I think we can all agree that preventing unintended pregnancy is a worthwhile objective; though we have very differing views on whether abstinence-based education works. States with higher rates of religiosity (by church attendance) have higher rates of teen pregnancy: there is a somewhat strong positive correlation.
A standing prayer request: Pray that we might get healthcare to our congregation.
IDK, what do we say here? We're going to start needing to be making some changes?
Roman society context on this one:
Vestal virgins: https://en.wikipedia.org/wiki/Vestal_Virgin
Baiae: https://en.wikipedia.org/wiki/Baiae
https://pbsinternational.org/programs/underwater-pompeii/ :
> Baiae: an ancient Roman city lost to the same volcanoes that entombed Pompeii. But unlike Pompeii, Baiae sits under water, in the Bay of Naples. Nearly 2,000 years ago, the city was an escape for Rome’s rich and powerful elite, a place where they were free of the social restrictions of Roman society. But then the city sank into the ocean, to be forgotten in the annals of history. Now, a team of archaeologists is mapping the underwater ruins and piecing together what life was like in this playground for the rich. What made Baiae such a special place? And what happened to it?
Woe! Woe unto the obviously promiscuous.
DARPA grant to work on sensing and stimulating the brain noninvasively [video]
Won't this mean that you cannot think of anything else while operating the equipment?
Do you need to not think of anything else to move your arms?
Ex-BCI research student here.
The nerves that control your arm are wired directly into your central nervous system and can active your muscles in direct response to action potentials fanned out from a single neuron.
These noninvasive technologies pick up aggregate signals from hundreds of thousands of neurons all firing pseudo-randomly, filtered spatially by the dura (thick layer of membrane that protects the brain), skull, flesh, skin and hair.
A more "HN" analogy would be like comparing a directly wired Ethernet connection (nerves connected to the CNS) to a side channel attack on an air-gapped system (noninvasive EEG). While randomly running background processes (normally) won't affect the data transmitted to the network, they will absolutely foil most side channel attacks.
What about with realtime NIRS with an (inverse?) scattering matrix? From https://www.openwater.cc/technology :
> Below are examples of the image quality we have achieved with our breakthrough scanning systems that use just red and near-infrared light and ultrasound pings.
https://en.wikipedia.org/wiki/Near-infrared_spectroscopy
Another question: is it possible to do ah molecular identification similar to idk quantum crystallography with photons of any wavelength, such as NIRS? Could that count things in samples?
https://twitter.com/westurner/status/1239012387367387138 :
> ... quantum crystallography: https://en.wikipedia.org/wiki/Quantum_crystallography There's probably some limit to infrared crystallography that anyone who knows anything about particles and lattices would know about ?
> What about with realtime NIRS with an (inverse?) scattering matrix?
I've purged a lot of the knowledge from that time from my brain, but from what I recall fNIRS takes a long time (on the order of multiple seconds) to take a single reading.
It also only really shows which regions of the brain are receiving more blood supply. While a huge improvement in spacial precision over EEG, it's still not anywhere near the same level of precision that a directly-wired nerve has.
That doesn't mean it's useless technology, just probably not viable for the low-latency, high-accuracy control you'd want for a military drone.
>Another question: is it possible to do ah molecular identification similar to idk quantum crystallography with photons of any wavelength, such as NIRS? Could that count things in samples?
I have absolutely no idea about molecular identification or quantum crystallography, so probably can't give you a good answer on that one.
The newer systems can sample much faster (10-100 Hz), but they're still limited by the fact that they're measuring a hemodynamic signal that lags behind neural activity.
Which other strong and weak forces could [photonic,] sensors detect?
IIUC, they're shooting for realtime MRI resolution with NIRS; to be used during surgery to assist surgery in realtime.
edit: https://en.wikipedia.org/wiki/Neural_oscillation#Overview says brainwaves are 1-150 Hz? IIRC compassion is acheivable on a bass guitar.
A couple of studies linked in Wikipedia for this use (one retracted):
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4049706/
Retracted: https://journals.plos.org/plosbiology/article?id=10.1371/jou...
Table with resolution differences between different techniques:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8116487/table/T...
Most NIRS systems use continuous-wave (CW) systems detecting blood oxygenation/chromophore concentration. I say this because, in the time domain, you're looking at 5-7 seconds to see changes in cerebral oxygenation.
It's also noteworthy that you're dealing with huge amounts of noise, very low resolution, and are limited to the cortical surface. This limits the applications in the BCI-domain.
I'm currently researching the feasability of fast optical imaging. This has to be done in the frequency domain, but may yield temporal resolution in the milliseconds. The downside is finding an incoherent light source that's able to be modulated fast enough to detect the scattering changes.
You mentioned "time-domain", and I recalled "time-polarization".
From https://twitter.com/westurner/status/1049860034899927040 :
https://web.archive.org/web/20171003175149/https://www.omnis...
"Mind Control and EM Wave Polarization Transductions" (1999)
> To engineer the mind and its operations directly, one must perform electrodynamic engineering in the time * domain, not in the 3-space EM energy density domain.*
Could be something there.
Topological Axion antiferromagnet https://phys.org/news/2021-07-layer-hall-effect-2d-topologic... :
> Researchers believe that when it is fully understood, TAI can be used to make semiconductors with potential applications in electronic devices, Ma said. The highly unusual properties of Axions will support a new electromagnetic response called the topological magneto-electric effect, paving the way for realizing ultra-sensitive, ultrafast, and dissipationless sensors, detectors and memory devices.
Optical topological antennas https://engineering.berkeley.edu/news/2021/02/light-unbound-... :
> The new work, reported in a paper published Feb. 25 in the journal Nature Physics, throws wide open the amount of information that can be multiplexed, or simultaneously transmitted, by a coherent light source. A common example of multiplexing is the transmission of multiple telephone calls over a single wire, but there had been fundamental limits to the number of coherent twisted light waves that could be directly multiplexed.
Rydberg sensor https://phys.org/news/2021-02-quantum-entire-radio-frequency... :
> Army researchers built the quantum sensor, which can sample the radio-frequency spectrum—from zero frequency up to 20 GHz—and detect AM and FM radio, Bluetooth, Wi-Fi and other communication signals.
> The Rydberg sensor uses laser beams to create highly-excited Rydberg atoms directly above a microwave circuit, to boost and hone in on the portion of the spectrum being measured. The Rydberg atoms are sensitive to the circuit's voltage, enabling the device to be used as a sensitive probe for the wide range of signals in the RF spectrum.
> "All previous demonstrations of Rydberg atomic sensors have only been able to sense small and specific regions of the RF spectrum, but our sensor now operates continuously over a wide frequency range for the first time,"
Sometimes people make posters or presentations for new tech, in medicine.
The xMed Exponential Medicine conference / program is in November this year: https://twitter.com/ExponentialMed
Space medicine also presents unique constraints that more rigorously select from possible solutions: https://en.wikipedia.org/wiki/Space_medicine
There is no progress in medicine without volunteers for clinical research trials. https://en.wikipedia.org/wiki/Phases_of_clinical_research
New Ways to Be Told That Your Python Code Is Bad
> Python programmers in general have an irrational aversion to if-expressions, a.k.a. the “ternary” operator.
Because the Python ternary operator is fucking backwards! Why on earth would you put the condition in the middle!?
x = 4 if condition() else 5
condition() ? 4 : 5
(setq x (if (condition) 4 5))
As I recall, object? and object?? are and work in IPython because the Python mailing list said that the ternary operator was not reserved. (IIRC there was yet no formal grammar or collections.abc or maybe even datetime or json yet at the time).
Ternary expressions on one line require branch coverage to be enabled in your e.g. pytest; otherwise it'll look like the whole line is covered by tests when each branch on said line hasn't actually been tested.
.get() -> Union[None, T]Web-based editor
It looks like this is actually just the editor from Visual Studio Online unless I've missed something.
It's great, but if it were really Visual Studio Code that would be awesome and I'd be pleasantly surprised.
(The difference being that if it's just the editor it misses out on all the compiler/analysis integrations. If Github were providing Linux containers in the cloud for working on projects - essentially the SSH feature from Visual Studio Code - it would be absolutely brilliant.)
I think what you might want is GitHub Codespaces?: https://github.com/features/codespaces. Was on HN recently too.
Hmm. $0.18 per hour for 4GB of RAM is a rather steep. A c6g.large on AWS would run you less than a half of that, for the same number of CPUs and memory. A quarter if you're happy to use spot instances. Might be neat to have an OSS tool that spins up an EC2 (or Azure or GCP) instance running Code-Server[1] (VS Code running natively on the server, but presenting the UI in the browser), with a given git repo and credentials.
Admittedly, the storage costs would be higher.
The ml-workspace docker image includes Git, Jupyter, VS Code, SSH, and "many popular data science libraries & tools" https://github.com/ml-tooling/ml-workspace
docker run -p 8080:8080 -v "${PWD}:/workspace" mltooling/ml-workspace
docker run --name=cocalc -d -v ~/cocalc:/projects -p 443:443 sagemathinc/cocalcGitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment
For comparison, what percentage of human-generated code is secure?
> For comparison, what percentage of human-generated code is secure?
Yeah how did they measure? Did static and dynamic analysis find design bugs too?
Maybe - as part of a Copilot-assisted DevSecOps workflow involving static and dynamic analysis run by GitHub Actions CI - create Issues with CWE "Common Weakness Enumeration" URLs from e.g. the CWE Top 25 in order to train the team, and Pull Requests to fix each issue?: https://cwe.mitre.org/top25/
Which bots send PRs?
AAS Journals Will Switch to Open Access
Publication charges for authors and students will probably rise for a factor of 2 or more... open access journals in technical fields are approaching $500 per page.
This will basically lock out people in the developing world from publishing in these journals.
> JOSS (Journal of Open Source Software) has managed to get articles indexed by Google Scholar [rescience_gscholar]. They publish their costs [joss_costs]: $275 Crossref membership, DOIs: $1/paper:
>> Assuming a publication rate of 200 papers per year this works out at ~$4.75 per paper
> [joss_costs]: https://joss.theoj.org/about#costs
^^ from https://news.ycombinator.com/item?id=24517711 & this log of my non- markdown non- W3C Web Annotation threaded comments with URIs: https://westurner.github.io/hnlog/#comment-24517711
Does anyone here have thoughts on JOSS? I've reviewed for them once, had the impression the editors take their job seriously, and think I'll review again in the future. The review approach that the journal facilitates has a strong focus on engineering aspects, i.e. it addresses a weakness of other venues, where it often does not matter how messy, unstable, and poorly documented the code is (or even if it compiles). On the other hand, the JOSS reviewers are typically not experts on the problem that the software is solving.
I'm currently reviewing for JOSS, and have done so before. In many ways they're a very strange journal: the paper is nearly an afterthought, and the review is focused on the code. But I like them. As you say, the editors take their role seriously. And it seems to have two valuable contributions.
Firstly, encouraging and structuring code review in academia. My own code is almost entirely solo (and messy), so a venue for structured review and an incentive to robustify public code is good. Secondly, the culture in some disciplines is that code is not citable, only papers - and JOSS is an end-run around this. I hope this second situation is changing, but we're not there yet so JOSS has a valuable role for the moment in simply being a 'journal' assigning DOIs basically for code packages.
[Scholarly] Code review tools; criteria and implementations?
Does JOSS specify e.g. ReviewBoard, GitHub Pull Request reviews, or Gerrit for code reviews?
The reviews for JOSS happen on github[0] but the journal's not prescriptive about how you develop your package as long as the code is public. The criteria for the JOSS review are very clear[1].
I don't want to oversell the depth of the code review possible; not all of the reviewers will be fully expert in whatever tiny cutting-edge area the package is for (making correctness checks difficult beyond the test suite), and most of us are academics-who-code rather than research software engineers. But the fact it's happening at all is a great step forward.
[0]: https://github.com/openjournals/joss-reviews/issues [1]: https://joss.readthedocs.io/en/latest/review_criteria.html
Thanks for the citations. Looks like Wikipedia has "software review" and "software peer review":
https://en.wikipedia.org/wiki/Software_review
https://en.wikipedia.org/wiki/Software_peer_review
I'd add "Antipatterns" > "Software" https://en.wikipedia.org/wiki/Anti-pattern#Software_design
and "Code smells" > "Common code smells" https://en.wikipedia.org/wiki/Code_smell#Common_code_smells
and "Design smells" for advanced reviewers: https://en.wikipedia.org/wiki/Design_smell
and the CWE "Common Weakness Enumeration" numbers and thus URLs for Issues from the CWE Top 25 and beyond: https://cwe.mitre.org/top25/
FWIW, many or most scientists are not even trying to be software engineers: they just write slow code without reusing already-tested components and expect someone else to review Pull Requests after their PDF is considered impactful. They know enough coding to push the bar for their domain a bit higher each time.
Are there points for at least in-writing planing for the complete lifecycle and governance of an ongoing thesis defense of open source software for science; after we publish, what becomes of this code?
From https://joss.theoj.org/about#costs :
> Income: JOSS has an experimental collaboration with AAS publishing where authors submitting to one of the AAS journals can also publish a companion software paper in JOSS, thereby receiving a review of their software. For this service, JOSS receives a small donation from AAS publishing. In 2019, JOSS received $200 as a result of this collaboration.
A low cost of $1/page just shows how much of a scam even prestigious journals have become for all involved parties.
Referees work for free. Submitting authors pay through the nose for presumably typesetting and proofs whose true cost is closer to $1/page rather than hundreds. Editors are overworked.
Sci-Hub was supposed to disrupt that industry, but it seems all it's done is shift burden to authors and created pay-for publishing.
Moderation costs money, too.
Additional ScholarlaryArticle "Journal" costs: moderation, BinderHub / JupyterLite white label SaaS?, hosting data and archived reproducible container images on IPFS and academictorrents and Git LFS, hosting {SQL, SPARQL, GraphQL,} queries and/or a SOLID HTTPS REST API and/or RSS feeds with dynamic content but static feed item URIs and/or ActivityStreams and/or https://schema.org/Action & InteractAction & https://schema.org/ReviewAction & ClaimReview fact check reviews, W3C Web Notifications, CRM + emailing list, keeping a legit cohort of impactful peer reviewers,
#LinkedData for #LinkedResearch: Dokieli, parsing https://schema.org/ScholarlyArticle citation styles,
> keeping a legit cohort of impactful peer reviewers, [who are time-constrained and unpaid, as well]
"Ask HN: How are online communities established?" https://news.ycombinator.com/item?id=24443965 re: building community, MCOS Marginal Cost of Service, CLV Customer Lifetime Value, etc
White House Launches US Digital Corps
I've worked with state government as a volunteer advisor. They're still developing everything with waterfall. Only contracting out to big firms, even if it's a small project. Lawmakers and aides sit in a room and write down what is to be done.
That's changing at the federal level. They know they've got a problem. Why shouldn't federal software be as easy to use as the best web software? If you've ever tried to use it you will quickly learn that isn't the case.
Some sites will only work with IE and no other browser. Developers in two years can make a huge difference for making the government be more agile and operate better.
I always suggest joining a local Code For America brigade. Work on a local project and see if it is for you. If you find yourself drawn to it then consider applying for a two year stint with the federal government. You can really make a difference!
> I've worked with state government as a volunteer advisor. They're still developing everything with waterfall. Only contracting out to big firms, even if it's a small project. Lawmakers and aides sit in a room and write down what is to be done.
The US Digital Services Playbook likely needs few modifications for use at state and local levels? https://github.com/usds/playbook#readme
"PLAY 1: Understand what people need" https://playbook.cio.gov/#play1
"PLAY 4: Build the service using agile and iterative practices" https://playbook.cio.gov/#play4
Do [lawmakers and aides] make good "Product Owners", stakeholders, [incentivized, gamified] app feedback capability utilizers? GitLab has Service Desk: you can email into the service desk email without having an account as necessary to create and follow up on [software] issues in GitHub/BitBucket/GitLab/Gitea project management sytems.
> That's changing at the federal level. They know they've got a problem. Why shouldn't federal software be as easy to use as the best web software? If you've ever tried to use it you will quickly learn that isn't the case.
"PLAY 3: Make it simple and intuitive" https://playbook.cio.gov/#play3
> Some sites will only work with IE and no other browser. Developers in two years can make a huge difference for making the government be more agile and operate better.
US Web Design Standards https://designsystem.digital.gov/
From https://github.com/uswds/uswds#browser-support :
>> We’ve designed the design system to support older and newer browsers through progressive enhancement. The current major version of the design system (2.0) follows the 2% rule: we officially support any browser above 2% usage as observed by analytics.usa.gov. Currently, this means that the design system version 2.0 supports the newest versions of Chrome, Firefox, Safari, and Internet Explorer 11 and up.
> I always suggest joining a local Code For America brigade. Work on a local project and see if it is for you. If you find yourself drawn to it then consider applying for a two year stint with the federal government. You can really make a difference!
From https://en.wikipedia.org/wiki/Code_for_America :
>> [...] described Code for America as "the technology world's equivalent of the Peace Corps or Teach for America". The article goes on to say, "They bring fresh blood to the solution process, deliver agile coding and software development skills, and frequently offer new perspectives on the latest technology—something that is often sorely lacking from municipal government IT programs. This is a win-win for cities that need help and for technologists that want to give back and contribute to lower government costs and the delivery of improved government service."
Launch HN: Litnerd (YC S21) – Teaching kids to read with the help of live actors
Hi HN, my name is Anisa and I am the founder of Litnerd (https://litnerd.com/), an online reading program designed to teach elementary school students in America how to read.
There are 37M elementary school students in America. Schools spend $20B on reading and supplemental education programs. Yet 42% of 4th grade students are reading at a 1st or 2nd grade proficiency level! The #1 reason students aren’t reading? They say it’s boring. We change that by bringing books to life. Think your favorite book turned into a tv-show style episode-by-episode reenactment, coupled with a complete curriculum and lesson plans.
1 in 8 Americans is functionally illiterate. Like any skill, reading is a habit. If you grew up in a household where you did not see your parents reading, you likely do not develop the habit. This correlates to the socio-economic divide. Two thirds of American students who lack reading skills by the end of fourth grade will rely on welfare as adults. To impact this, research suggests that we need to start at the earliest years.
I am passionate about the research in support of art and theatre as well as story-telling to improve childhood learning. Litnerd is the marriage of these interests. The inspiration comes from Sesame Street and Hamilton The Musical. In the late 60s, Joan Cooney decided to produce a children’s TV show that would influence children across America to learn to read—it became Sesame Street. Cooney researched her idea extensively, consulting with sociologists and scientists, and found that TV’s stickiness can be an important tool for education. Lin-Manuel Miranda took the story of Alexander Hamilton and brought it to life as a musical. Kids have learned more about Hamilton’s history thanks to Hamilton the Musical than any of their textbooks. In fact, this was the case so much that a program called EduHam is used to teach history in middle schools across the nation. When I heard that, the lightbulb went off and I decided to go all in on starting Litnerd.
We hire art and theatre professionals to recreate scenes directly from books in episode style format to bring the book to life, in a similar fashion to watching your favorite TV shows. We literally lead 'read out loud' in the classroom while the teacher/actor is acting out the main character in the book. We have a weekly designated Litnerd period in the schools/classes we serve and we live-stream in our teachers/actors for an interactive session (the students participate and read live with the actor as well as complete written lesson plans, phonetic exercises etc). We are currently serving 14,000 students in this manner.
The format of our program is such that if you don't complete the assigned reading and worksheets, you will feel like you are missing out on what is happening in later episodes. In this way, reading is layered in as a fundamental core to the program. Our program is part of scheduled classroom time.
A big part of our business involves curating content and materials that capture the interest and coolness-factor for elementary school students. We’ve found that students love choose-your-own-adventure style stories, especially ones involving mythical creatures—something about being able to have autonomy on the outcomes. So far, it seems to be working. We've even received fan mail from students! But we are obsessed with staying cool/relevant in our content.
Teachers like our product because it eases the burden placed on them. US teachers typically spend 4 to 10 hours a week (unpaid) planning their curriculum and $400-800 of their own money for classroom supplies. That's outrageous! When designing Litnerd, we wanted to ensure our product was not adding more work to their plate. Our programs are led by our own Resident Teaching Artists, who are live streamed into the classroom and remain in character to the episode as they teach the Litnerd curriculum built on top of the books. Our programs come with lesson plans, activity packets, curriculum correlations, educator resources, and complete ebooks.
Traditional K-12 education has extremely long sale cycles and is hard to break into. It can take years to become a contracted vendor, especially with large districts like NYC Department of Education. Because of my experience with my first YC backed startup that sold to government and nonprofits, coupled with my experience working at a large edtech company that built content for Higher Ed, I understand this sector and how to navigate the budget line item process.
Since launching in January, we have become contracted vendors with the New York City Department of Education (the largest education district in America). As a result, we’ve been growing at 60% MoM, are currently used by over 14k students in their classrooms and hit $110K in ARR. Our program is part of scheduled classroom time for elementary schools—not homework, and not extracurricular. Here’s a walkthrough video from a teacher’s perspective: https://www.loom.com/share/9ffc59f0d7ed4a66964003703bba7b94.
I am so grateful for the opportunity to share our story and mission with you. If you loved or struggled with reading as a kid, what factors do you think contributed? Also, if you have experience teachIng Elementary School or if you are a parent, I would love to hear your thoughts and ideas on how you foster reading amongst your students/children! I am excited to hear your feedback and ideas to help us inspire the next generation of readers.
This looks like a really thoughtful resource, and I can tell you are hitting all the right marks when it comes to getting this into classrooms. As a first grade teacher, however, I all curious how you see Litnerd fitting into a balanced literacy program. In browsing your programs, I see language comprehension and social emotional learning but no phonics or phonemic awareness. Do you have plans to support these crucial reading skills as well?
I'm thrilled to have our first elementary school teacher comment!! Yes! We have a team of educators and curriculum writers (broken up by Grades PreK-K, 1-2 and 3-5) that follow Common Core standards and build custom ELA curriculum on top of each book and Litnerd program. We provide 2 lesson plans per week and a total of 8 lesson plans per program. We also create SEL lessons (again, based on the book) that follow NYSED competencies and 'Leader in Me' program language (since that is what most of our schools currently use for SEL). We hope to continue to improve overtime and I would definitely any feedback for us as we grow intros area!
TIL a new acronym word symbol lexeme: SEL: Social and Emotional Learning
> Social Emotional Learning (SEL) is an education practice that integrates social emotional skills into school curriculum. SEL is otherwise referred to as "socio-emotional learning" or "social-emotional literacy." When in practice, social emotional learning has equal emphasis on social and emotional skills to other subjects such as math, science, and reading.[1] The five main components of social emotional learning are self-awareness, self management, social awareness, responsible decision making, and relationship skills.
https://en.wikipedia.org/wiki/Social_and_Emotional_Learning
For good measure, Common Core English Language Arts standards: https://en.wikipedia.org/wiki/Common_Core_State_Standards_In...
Khan Academy has 2nd-9th Grade ELA exercises: English & Language Arts: https://www.khanacademy.org/ela
Unfortunately AFAIU there's not a good way to explore the Khan Academy Kids curriculum graph; which definitely does include reading: https://learn.khanacademy.org/khan-academy-kids/
> The app engages kids in core subjects like early literacy, reading, writing, language, and math, while encouraging creativity and building social-emotional skills
In terms of Phonemic awareness and Phonological awareness, is there a good a survey of US and World reading programs and their evidence-based basis, if any??
From https://en.wikipedia.org/wiki/Phonemic_awareness :
> Phonemic awareness is a subset of phonological awareness in which listeners are able to hear, identify and manipulate phonemes, the smallest mental units of sound that help to differentiate units of meaning (morphemes). Separating the spoken word "cat" into three distinct phonemes, /k/, /æ/, and /t/, requires phonemic awareness. The National Reading Panel has found that phonemic awareness improves children's word reading and reading comprehension and helps children learn to spell.[1] Phonemic awareness is the basis for learning phonics.[2]
> Phonemic awareness and phonological awareness are often confused since they are interdependent. Phonemic awareness is the ability to hear and manipulate individual phonemes. *Phonological awareness includes this ability, but it also includes the ability to hear and manipulate larger units of sound, such as onsets and rimes and syllables.*
What are some of the more evidence-based (?) (early literacy,) reading curricula? OTOH: LETRS, Heggerty, PAL: https://www.google.com/search?q=site%3Aen.wikipedia.org+%22l...
Looks like Cambium acquired e.g. Kurzweil Education in 2005?
More context:
Reading readiness in the United States: https://en.wikipedia.org/wiki/Reading_readiness_in_the_Unite...
Emergent literacies: https://en.wikipedia.org/wiki/Emergent_literacies
An interactive IPA chart with videos and readings linked with RDF (e.g. ~WordNet RDF) would be great. From "Duolingo's language notes all on one page" https://westurner.github.io/hnlog/#comment-26430146 :
> An IPA (International Phonetic Alphabet) reference would be helpful, too. After taking linguistics in college, I found these Sozo videos of US english IPA consonants and vowels that simultaneously show {the ipa symbol, example words, someone visually and auditorily producing the phoneme from 2 angles, and the spectrogram of the waveform} but a few or a configurable number of [spaced] repetitions would be helpful: https://youtu.be/Sw36F_UcIn8
> IDK how cartoonish or 3d of an "articulatory phonetic" model would reach the widest audience. https://en.wikipedia.org/wiki/Articulatory_phonetics
> IPA chart: https://en.wikipedia.org/wiki/International_Phonetic_Alphabe...
> IPA chart with audio: https://en.wikipedia.org/wiki/IPA_vowel_chart_with_audio
> All of the IPA consonant chart played as a video: "International Phonetic Alphabet Consonant sounds (Pulmonic)- From Wikipedia.org" https://youtu.be/yFAITaBr6Tw
> I'll have to find the link of the site where they playback youtube videos with multiple languages' subtitles highlighted side-by-side along with the video.
>> [...] Found it: https://www.captionpop.com/
>> It looks like there are a few browser extensions for displaying multiple subtitles as well; e.g. "YouTube Dual Subtitles", "Two Captions for YouTube and Netflix"
Phonics programs really could reference IPA from the start: there are different sounds for the same letters; IPA is the most standard way to indicate how to pronounce words: it's in the old school dictionary, and now it's in the Google "define:" or just "define word" dictionary.
UN Sustainable Development Goal 4: Quality Education: https://www.globalgoals.org/4-quality-education
> Target 4.6: Universal Literacy and Numeracy
> By 2030, ensure that all youth and a substantial proportion of adults, both men and women, achieve literacy and numeracy.
https://sdgs.un.org/goals/goal4 :
> Indicator 4.6.1: Percentage of population in a given age group achieving at least a fixed level of proficiency in functional (a) literacy and (b) numeracy skills, by sex
... Goals, Targets, and Indicators.
Which traversals of a curriculum graph are optimal or sufficient?
You can add https://schema.org/about and https://schema.org/educationalAlignment Linked Data to your [#OER] curriculum resources to increase discoverability, reusability.
Arne-Thompson-Uther Index code URN URIs could be helpful: https://en.wikipedia.org/wiki/Aarne%E2%80%93Thompson%E2%80%9...
> The Aarne–Thompson–Uther Index (ATU Index) is a catalogue of folktale types used in folklore studies.
Are there competencies linked to maybe a nested outline that we typically traverse in depth-first order? https://github.com/todotxt/todo.txt : Todo.txt format has +succinct @context labels. Some way to record and score our own paths objectively would be great.
> What are some of the more evidence-based (?) (early literacy,) reading curricula? OTOH: LETRS, Heggerty, PAL
Looks like there are only 21 search results for: "LETRS" "Fundation" "Heggerty": https://www.google.com/search?q="LETRS"+"fundation"+"heggert...
What is the name for this category of curricula?
Perhaps the US Department of Education or similar could compare early reading programs in a wiki[pedia] page, according to criteria to include measures of evidence-basedness? Just like https://collegescorecard.ed.gov/data/ has "aggregate data for each institution [&] Includes information on institutional characteristics, enrollment, student aid, costs, and student outcomes."
From YouTube, it looks like there are cool hand motions for Heggerty.
Nimforum: Lightweight alternative to Discourse written in Nim
awesome-selfhosted > "Communication - Social Networks and Forums" doesn't yet link to Nimforum: https://github.com/awesome-selfhosted/awesome-selfhosted#com...
An Opinionated Guide to Xargs
Wanting verbose logging from xargs, years ago I wrote a script called `el` (edit lines) that basically does `xargs -0` with logging. https://github.com/westurner/dotfiles/blob/develop/scripts/e...
It turns out that e.g. -print0 and -0 are the only safe way: line endings aren't escaped:
find . -type f -print0 | el -0 --each -x echo
(author here) Hm I don't see either of these points because:
GNU xargs has --verbose which logs every command. Does that not do what you want? (Maybe I should mention its existence in the post)
xargs -P can do everything GNU parallel do, which I mention in the post. Any counterexamples? GNU parallel is a very ugly DSL IMO, and I don't see what it adds.
--
edit: Logging can also be done with by recursively invoking shell functions that log with the $0 Dispatch Pattern, explained in the post. I don't see a need for another tool; this is the Unix philosophy and compositionality of shell at work :)
Parallel's killer feature is how it spools subprocess output, ensuring that it doesn't get jumbled together. xargs can't do that. I use parallel for things like shelling out to 10000 hosts and getting some statistics. If I use xargs the output stomps all over itself.
Ah OK thanks, I responded to this here: https://news.ycombinator.com/item?id=28259473
As far as I'm aware, xargs still has the problem of multiple jobs being able to write to stdout at the same time, potentially causing their output streams to be intermingled. Compare this with parallels --group.
Also parallels can run some of those threads on remote machines. I don't believe xargs has an equivalent job management function.
In your examples you fail to put 'xargs -P' in the middle of a pipeline: You only put it at the end.
In other words:
some command | xargs -P other command | third command
UNIX is great in that you can pipe commands together, but due to the interleaving issue 'xargs -P' fails here. It does not live up to the UNIX philosophy. Which is probably why you unconsciously only use it at the end of a pipeline.
You can find a different counterexample on https://unix.stackexchange.com/questions/405552/using-xargs-... I will be impressed if you can implement that using xargs. Especially if you can make it more clean than the paralel version.
[deleted]
Yeah but xargs doesn't refuse to run until I have agreed to a EULA stating I will cite it in my next academic paper.
parallel doesn't either, it just nags. I agree about how silly and annoying it is. Imagine if every time the parallel author opened Firefox he got a message reminding him to personally thank me if he uses his web browser for research, or if every time his research program calls malloc he has to acknowledge and cite Ulrich Drepper. Very very silly.
Parallel is the better tool but the nagware impairs its reputation.
Enhanced Support for Citations on GitHub
> CITATION.cff files are plain text files with human- and machine-readable citation information. When we detect a CITATION.cff file in a repository, we use this information to create convenient APA or BibTeX style citation links that can be referenced by others.
https://schema.org/ScholarlyArticle RDFa and JSON-LD can be parsed with a standard Linked Data parser. Looks like YAML-LD requires quoting e.g. "@context": and "@id":
From https://docs.github.com/en/github/creating-cloning-and-archi... ; in your repo's /CITATION.cff:
cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
- family-names: "Lisa"
given-names: "Mona"
orcid: "https://orcid.org/0000-0000-0000-0000"
- family-names: "Bot"
given-names: "Hew"
orcid: "https://orcid.org/0000-0000-0000-0000"
title: "My Research Software"
version: 2.0.4
doi: 10.5281/zenodo.1234
date-released: 2017-12-18
url: "https://github.com/github/linguist"
Canada calls screen scraping ‘unsecure,’ sets Open Banking target for 2023
It's about time. When I learned that applications like YNAB (You Need A Budget) use services like Plaid to connect to my bank account, and that these services literally take my username and password and impersonate me to get my banking data, I was a little sketched out. I use YNAB every day, and having it connected to my bank account is incredibly useful, but if something goes wrong and Plaid loses my money somehow, is there any recourse?
Hopefully individuals will be able to use the Open Banking APIs to access their own data directly, but it looks like accreditation will be required, so probably not.
Here's the full text of the report: https://www.canada.ca/en/department-finance/programs/consult...
Plaid is only one security breach away from being utterly destroyed. And they will take out the financial lives of all their customers with them.
It’s utterly irresponsible and I have no idea how Plaid hasn’t been shut down. You have no recourse if they are breached. The TOS of your online banking probably says that if you disclose your username and password to any third party then you have no liability protections.
This is FUD. Lots of Plaid-based connections only allow reads. This is a regulated industry, and the fallout reputationally might be tough, but consumers are well-protected.
How is that enforced? What is the technical basis that enforces read-only access using user/password auth? Especially since that user/password auth is used by an end user to do "write"-type actions?
Read-only access is not possible. By handing over the credentials you are handing over write access. You are correct.
A couple of my banking institutions let me generate a read-only set of credentials for this sort of purpose.
Citi and Capital One have OAuth flows that Plaid supports, too, which tends to make me angrier at the banks than Plaid; the need for this stuff has been clear for a decade now, but only a few have added OAuth or similar.
AFAIU, there are still zero (0) consumer banking APIs with Read-Only e.g. OAuth APIs in the US as well?
Banks could save themselves CPU, RAM, bandwidth, and liability by implementing read-only API tokens and methods that need only return JSON - instead of HTML or worse, monthly PDF tables for a fee - possibly similar to the Plaid API: https://plaid.com/docs/api/
There is competition in consumer/retail banking, but still the only way to do e.g. budget and fraud analysis with third party apps is to give away all authentication factors: u/p/sqa; and TBH that's unacceptable.
Traditional and distributed ledger service providers might also consider W3C ILP: Interledger Protocol (in starting their move to quantum-resistant ledgers by 2022 in order to have a 5 year refresh cycle before QC is a real risk by 2027, optimistically, for science) when reviewing the entropy of username+password_hash+security_question_answer strings in comparison to the entropy of cryptoasset account public key hash strings: https://interledger.org/developer-tools/get-started/overview...
> Sender – Initiates a value transfer.
> Router (Connector) – Applies currency exchange and forwards packets of value. This is an intermediary node between the sender and the receiver. {MSB: KYC, AML, 10k reporting requirement, etc}
> Receiver – Receives the value
Multifactor authentication: Something you have, something you know, something you are
Multisig: n-of-m keys required to approve a transaction
Edit: from "Fed announces details of new interbank service to support instant payments" https://news.ycombinator.com/item?id=24109576 :
> For purposes of Interledger, we call all settlement systems ledgers. These can include banks, blockchains, peer-to-peer payment schemes, automated clearing house (ACH), mobile money institutions, central-bank operated real-time gross settlement (RTGS) systems, and even more. […]
> You can envision the Interledger as a graph where the points are individual nodes and the edges are accounts between two parties. Parties with only one account can send or receive through the party on the other side of that account. Parties with two or more accounts are connectors, who can facilitate payments to or from anyone they're connected to.
> Connectors [AKA routers] provide a service of forwarding packets and relaying money, and they take on some risk when they do so. In exchange, connectors can charge fees and derive a profit from these services. In the open network of the Interledger, connectors are expected to compete among one another to offer the best balance of speed, reliability, coverage, and cost.
W3C ILP: Interledger Protocol > Peering, Clearing and Settling: https://interledger.org/rfcs/0032-peering-clearing-settlemen...
> Hopefully individuals will be able to use the Open Banking APIs to access their own data directly, but it looks like accreditation will be required, so probably not.
When you loan your money to a bank by depositing ledger dollars or cash - and they, since GLBA in 1999, invest it and offer less than a 1% checking interest rate - and they won't even give you the record of all of your transactions as CSV/OFX `SELECT * FROM transactions WHERE account_id=?`, you have to pay $20/mo per autogenerated PDF containing a table of transactions to scrape with e.g. PDFminer (because they don't keep all account history data online)?
Seemingly OT, but not. APIs for comparison here:
FinTS / HBCI: Home Banking Computer Information protocol https://en.wikipedia.org/wiki/FinTS
E.g. GNUcash (open source double-entry accounting software) supports HBCI (and QIF (Quicken format), and OFX (Open Financial Exchange)). https://www.gnucash.org/features.phtml
HBCI/FinTS has been around in Germany for quite awhile but nowhere else has comparable banking standards? I.e. Plaid may (unfortunately, due to lack of read-only tokens across the entire US consumer banking industry) be the most viable option for implementing HBCI-like support in GNUcash
OpenBanking API Specifications: https://standards.openbanking.org.uk/api-specifications/
Web3 (Ethereum,) APIs: https://web3py.readthedocs.io/en/stable/web3.main.html#rpc-a...
ISO20022 is "A single standardisation approach (methodology, process, repository) to be used by all financial standards initiatives" https://www.iso20022.org/
Brazil's PIX is one of the first real implementers of ISO20022. A note regarding such challenges: https://news.ycombinator.com/item?id=24104351
What data format does the FTC CAT Consolidated Audit Trail expect to receive mandatory financial reporting information in? Could ILP simplify banking and financial reporting at all?
FWIU, RippleNet (?) is the only network that supports attachments of e.g. line-item invoices (that we'd all like to see in the interest of transparency and accountability in government spending).
W3C ILP: Interledger Protocol. See links above.
Of the specs in this loose category, only cryptoledgers do not depend upon (DNS or) TLS/SSL - at the protocol layer, at least - and every CA in the kept-up-to-date trusted CA cert bundle (that could be built from a CT Certificate Transparency log of cert issuance and revocation events kept in a blockchain or e.g. centralized google/trillian, which they have the trusted sole root and backup responsibilities for).
Though, the DNS dependency has probably crept back into e.g. the bitcoind software by now (which used to bootstrap its list of peer nodes (~UNL) from an IRC IP address instead of a DNS domain).
FWIU, each trusted ACH (US 'Direct Deposit') party has a (one) GPG key that they use to sign transaction documents sent over now (S)FTP on scout's honor - on behalf of all of their customers' accounts.
Interactive Linear Algebra (2019)
For those interested in these kinds of interractive math experiences, I have been keeping track of them for a while. Here is my list so far:
• https://www.intmath.com/ - Interactive Mathematics Learn math while you play with it
• http://worrydream.com/LadderOfAbstraction/ - up and down the ladder of abstraction
• https://betterexplained.com/ - Intuitive guides to various things in math
• https://www.math3ma.com/blog/matrices-probability-graphs - Viewing Matrices & Probability as Graphs
• http://immersivemath.com/ila/index.html - immersive linear alg
https://github.com/topics/linear-algebra?l=jupyter+notebook lists "Computational Linear Algebra for Coders" https://github.com/fastai/numerical-linear-algebra
"site:GitHub.com inurl:awesome linear algebra jupyter" lists a few awesome lists with interactive linear algebra resources: https://www.google.com/search?q=site%3Agithub.com+inurl%3Aaw...
3blue1brown's "Essence of linear algebra" playlist has some excellent tutorials with intuition-building visualizations built with manim: https://youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFit...
Git password authentication is shutting down
I'm fine with this change for my usage, I don't think I've used password auth for myself or any automated service I've setup for years now. However, this will introduce more confusion for newcomers who already have to figure out what Git, GitHub, etc are. I just spent some time last weeekend teaching someone the basics of how to create a new project. Such a simple idea required introducing the terminal, basic terminal commands, GitHub, git and its most common commands. It took about 3 hours for us to get through just the most basic pieces. Adding on ssh-keygen, key management adds even more friction.
It's certainly a difficult problem. How can we offer a more gentle learning curve for budding developers while still requiring "real" projects to use best practices for security and development?
I also have found teaching someone how to be even marginally capable of contributing to a Github project from scratch to be a very time consuming and frustrating thing. Think, having your graphics designer able to make commits, or having someone who only wants to update docs.
The worst part is the "easier" solutions are actually just footguns in disguise, as soon as they accidentally click the wrong thing and end up with a detached HEAD, a few commits ahead and behind the REMOTE, and halfway through a botched merge, you have to figure out how to bail them out of that using a GUI you've never actually used. Knowing all this, you either teach them git (high short term pain, high chance of them just giving up immediately) or you tell them to download the first result for "windows foss git gui" and pray that history won't repeat itself.
The solution that everyone actually uses until they learn the unnecessary details is "take a backup of relevant files and blow away & redownload the repo".
wait there is another way?
git stash
git reset --hard master
git pull
git stash popUh oh, you've now merged remote master into your local master which was two commits ahead and 3 behind!
My git config has fast-forward only for pulls and I habitually use --ff-only for any pull :)
The command you want is "git pull --rebase". There are configuration settings to make "git pull" rebase by default, and I'd recommend always turning that on that default (which may be by why the person above omitted it from his pull command).
I actually thought "git pull" did "git pull --rebase" by default (this may be what you get from running `git --configure` without modifications?), but maybe I've just been configuring it that way. You can achieve this in your global Git configuration by setting "pull.rebase" to "true".
I don't think it's sane behavior for "git pull" to do anything else besides rebase the local change onto the upstream branch, so I'm surprised it's not the command's default behavior. Has the project not changed the CLI for compatibility reasons or something?
When do you ever want a "git pull" that's not a rebase? That generates a merge commit saying "Merge master into origin/master" (or something similar) which is stupid. If you really want to use actual branches for some reason, that's fine, but "merge master into master" commits are an anti-pattern that if I ever see in a Git repository I'm working on or responsible for, results in me having a conversation with the author about how to use Git correctly.
I actually don't want to rebase. Rebase can rewrite your local history and make it impossible for you to push without doing a force push (or some other more complicated manuevering). The only time pull with rebase is okay (on a shared branch such as master) is when you know that your local branch is strictly behind remote. That's exactly what --ff-only does. It rebases only if you are behind remote, and declines to do so if you are ahead so you can sort your shit out without rewriting history.
If "git pull --rebase" succeeds without reporting a merge conflict, then it's tantamount to having executed the command with "--ff-only".
I believe you are incorrect however. "git pulll --rebase" will not ever rewrite history from the origin repository. It will only ever modify your local unpublished commits to account for new commits from the origin branch. If your change and the new commits from origin aren't touching the same lines or files, then the update is seamless [1].
If there is a conflict because your change and a new commit from origin both changed the same line in the same file, this can result in a merge commit that you need to resolve. But you resolve this locally, by updating your (unpushed, local) commit(s) to account for the new history. When you complete the merge, it will not show up as a merge commit in the repository -- you will simply have simply amended your local unpublished commit(s) only to account for the changes in upstream, and you will have seen every conflict and resolved each one yourself. When the process is complete, and you've resolved the merge, you'll have a nice linear branch where your commit(s) are on the tip of the origin branch.
The flag "--ff-only" basically just means "refuse to start a local merge while performing this operation, and instead fail".
Because of the potential for these merge conflicts, it's a best practice to "git pull" frequently, so that if there are conflicts you can deal with them incrementally (and possibly discuss the software design with coworkers/project partners if the design is beginning to diverge -- and so people can coordinate what they're working on and try to stay out of each other's way), instead of working through a massive pile of conflicts at the end of your "local 'branch'" (i.e. the code in your personal Git repo that constitutes unpushed commits to the upstream branch).
Additionally, all of the central Git repositories I've worked in in a professional context were also configured to disallow "git push --force" (a command that can rewrite history), for all "real" branches. These systems gave users their own private Git repo sandbox (just like you can fork a repo on GitHub) where they can force push if they want to (but is backed up centrally like GitHub to avoid the loss of work). This personal repo was very useful for saving my work. I was in the habit of committing and pushing to the private repo about once every 30m-1h (to eliminate the chance of major work loss due to hardware failure). Almost always I'd squash all of these commits into one before rebasing onto master, so that the change comes in as a single commit, unless it would be too large to review.
In the occasional circumstances where I've legitimately needed to rewrite history for some reason -- say credentials got into the repository, or someone generated one of these "merge master into master" commits -- then I would change HEAD from master to another branch, delete master, and then recreate it with the desired state. (And even that operation would show up in the system's logs, so in the case of something like credentials you'd additionally contact the security or source control team to make sure the commit objects containing the credentials were actually deleted out of the repo history completely, including stuff you can find only via the reflog.) Then contact the team working on the repo to let them know that you had to rewrite history to correct a problem.
I would recommend disabling "git push --force" for all collaborative projects. If you're operating the repository, you can do this by invoking "git config --system receive" and setting "denyNonFastForwards true". In GitHub there's probably a repository setting switch somewhere.
Once professional software engineers start working with Git all day long, they quickly get past the beginner stage and the need to do this kind of stuff is very rare.
[1] It doesn't mean the software will work though, even if both changes would have worked in isolation. You still need to inspect and test the results of "git pull (--ff-only)", since even if there are no conflicts like commits that modify the same lines as yours, or there are conflicts that Git can resolve automatically, it's possible for the resultant software logic to be defective, since Git has no semantic understanding of the code.
`git pull --rebase` usually is what I need to do. To save local changes and rebase to the git remote's branch:
# git branch -av;
# git remote -v;
# git reflog; git help reflog; man git-reflog
# git show HEAD@{0}
# git log -n5 --graph;
git add -A; git status;
git stash; git stash list;
git pull --rebase;
#git pull --rebase origin develop
# git fetch origin develop
# git rebase origin/develop
git stash pop;
git stash list;
git status;
# git commit
# git rebase -i HEAD~5 # squash
# git push
There's a way to default to --rebase for pulls: is there a reason not to set that in a global gitconfig? Edit: From https://stackoverflow.com/questions/13846300/how-to-make-git... :
> There are now 3 different levels of configuration for default pull behaviour. From most general to most fine grained they are: […]
git config --global pull.rebase trueMaybe start here: http://learngitbranching.js.org/
GitHub Learning Lab: https://lab.github.com/
https://learnxinyminutes.com/docs/git/ #Further_resources
A future for SQL on the web
What's kind of bonkers here is that IndexedDB uses sqlite as its backend. So, this is sqlite (WASM) -> IndexedDB -> sqlite (native).
The Internet is a wild place...
I'm going to build a business that offers SQLite as a web service. It will be backed by a P2P network of browser instances storing data in IndexedDB. Taking investment now.
TIL, about Graph "Protocol for building decentralized applications quickly on Ethereum" https://github.com/graphprotocol
https://thegraph.com/docs/indexing
> Indexers are node operators in The Graph Network that stake Graph Tokens (GRT) in order to provide indexing and query processing services. Indexers earn query fees and indexing rewards for their services. They also earn from a Rebate Pool that is shared with all network contributors proportional to their work, following the Cobbs-Douglas Rebate Function.
> GRT that is staked in the protocol is subject to a thawing period and can be slashed if Indexers are malicious and serve incorrect data to applications or if they index incorrectly. Indexers can also be delegated stake from Delegators, to contribute to the network.
> Indexers select subgraphs to index based on the subgraph’s curation signal, where Curators stake GRT in order to indicate which subgraphs are high-quality and should be prioritized. Consumers (eg. applications) can also set parameters for which Indexers process queries for their subgraphs and set preferences for query fee pricing.
It's Ethereum though, so it's LevelDB, not SQLite on IndexedDB on SQLite.
Show HN: Python Source Code Refactoring Toolkit via AST
A similar type project. Though I haven't seen much activity recently:
I haven't used bowler, but from what I can see it is using lib2to3 (through fissix), which can't parse newer Python code (parenthesized context managers, new pattern matching statement etc.) due to it being using a LL(1) parser. The regular ast on the other hand is always compatible with the current grammar, since that is what CPython internally consumes. It is also more handy to work with since majority of the linters already using it, so it would be very easy to port rules.
(Author of Bowler/maintainer of fissix)
The main concern with using the ast module from stdlib is that you lose all formatting/whitespace/comments in your syntax tree, so writing any changes back to the original sources requires doing a lot more extra work to preserve the original formatting, put back comments, etc. This is entire point of lib2to3/fissix and LibCST, allowing large scale CST manipulation while preserving all of those comments and formatting. We do recognize the limitations of lib2to3/fissix, though, so there have been some backburner plans to move Bowler onto LibCST, as well as building a PEG-based parser for LibCST specifically to enable support for 3.10 and future syntax/grammar changes. But of course, this is very difficult to give any ETA or target for release.
Did you consider PyCQA/RedBaron (which is based upon PyCQA/baron, an AST implementation which preserves comments and whitespace)? https://redbaron.readthedocs.io/en/latest/
It was considered, but the initial goals of Bowler was to a) build on top of lib2to3 in an attempt to get broader upstream support for maintaining it as an "official" cst module, which fizzled out, and b) to use some of the more complex matching semantics that lib2to3 enabled, which baron and other cst alternatives don't really attempt to cover.
Things like "find any function that uses kwargs" or "find any class that defines a method named `bar`" can be easily expressed with lib2to3's matching grammar, and no other CST that I'm aware of (that isn't itself based on lib2to3) has equivalent functionality. This is something we wanted to add to LibCST, but haven't had the time to focus on given other priorities. Meanwhile, we used LibCST to write a safer alternative to isort: https://usort.readthedocs.io
Rog. I think CodeQL (GitHub acquired Semmle and QL in 2019) supports those types of queries; probably atop lib2to3 as well. https://codeql.github.com/docs/writing-codeql-queries/introd...
From https://news.ycombinator.com/item?id=24511280 :
> Additional lists of static analysis, dynamic analysis, SAST, DAST, and other source code analysis tools […]
Emacs' org-mode gets citation support
FWIW, Jupyter-book handles Citations and bibliographies with sphinxcontrib-bibtex: https://jupyterbook.org/content/citations.html
Some notes about Zotero and Schema.org RDFa for publishing [CSL with citeproc] citations: references of Linked Data resources in a graph, with URIs all: https://wrdrd.github.io/docs/tools/index#zotero-and-schema-o...
Compared to trying to parse beautifully typeset bibliographies in PDFs built from LaTeX with a Computer Modern font, search engines can more easily index e.g. https://schema.org/ScholarlyArticle linked data as RDFa, Microdata, or JSON-LD.
Scholarly search engines: Google Scholar, Semantic Scholar, Meta.org,
NSA Kubernetes Hardening Guidance [pdf]
- Scan containers and Pods for vulnerabilities or misconfigurations.
- Run containers and Pods with the least privileges possible.
- Use network separation to control the amount of damage a compromise can cause.
- Use firewalls to limit unneeded network connectivity and encryption to protect confidentiality.
- Use strong authentication and authorization to limit user and administrator access as well as to limit the attack surface.
- Use log auditing so that administrators can monitor activity and be alerted to potential malicious activity.
- Periodically review all Kubernetes settings and use vulnerability scans to help ensure risks are appropriately accounted for and security patches are applied.
> and encryption to protect confidentiality
Probably the hardest part about this. Private networks with private domains. Who runs the private CA, updates DNS records, issues certs, revokes, keeps the keys secure, embeds the keychain in every container's cert stack, and enforces validation?
That is a shit-ton of stuff to set up (and potentially screw up) which will take a small team probably months to complete. How many teams are actually going to do this, versus just terminating at the load balancer and everything in the cluster running plaintext?
> That is a shit-ton of stuff to set up (and potentially screw up) which will take a small team probably months to complete.
Agree! This is why that "Kubernetes Hardening Guidance" is for NSA, not for startups.
Resource needs aside, keeping basic AppSec/InfoSec hygiene is a strong recommendation. Also there are tons of startups that are trying to provide solutions/services to solve that also. A lot of times, it's worth the money.
This guidance is provided by the NSA, not for the NSA.
From the doc:
>It includes hardening strategies to avoid common misconfigurations and guide system administrators and developers of National Security Systems on how to deploy Kubernetes...
Also:
> Purpose > NSA and CISA developed this document in furtherance of their respective cybersecurity missions, including their responsibilities to develop and issue cybersecurity specifications and mitigations. This information may be shared broadly to reach all appropriate stakeholders.
NSA has multiple mandates and many stakeholders.
Looks like there's actually a "summary of the key recommendations from each section" on page 2.
> Works cited:
> [1] Center for Internet Security, "Kubernetes," 2021. [Online]. Available: https://cisecurity.org/resources/?type=benchmark&search=kube... .
> [2] DISA, "Kubernetes STIG," 2021. [Online]. Available: https://dl.dod.cyber.mil.wp- content/uploads/stigs/zip/U_Kubernetes_V1R1_STIG.zip. [Accessed 8 July 2021]
> [3] The Linux Foundation, "Kubernetes Documentation," 2021. [Online]. Available: https://kubernetes.io/docs/home/ . [Accessed 8 July 2021].
> [4] The Linux Foundation, "11 Ways (Not) to Get Hacked," 18 07 2018. [Online]. Available: https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hac... . [Accessed 8 July 2021].
> [5] MITRE, "Unsecured Credentials: Cloud Instance Metadata API." MITRE ATT&CK, 2021. [Online]. Available: https://attack.mitre.org/techniques/T1552/005/. [Accessed 8 July 2021].
> [6] CISA, "Analysis Report (AR21-013A): Strengthening Security Configurations to Defend Against Attackers Targeting Cloud Services." Cybersecurity and Infrastructure Security Agency, 14 January 2021. [Online]. Available:https://us- cert.cisa.gov/ncas/analysis-reports/ar21-013a [Accessed 8 July 2021].
How can k8s and zero-trust cooccur?
> CISA encourages administrators and organizations review NSA’s guidance on Embracing a Zero Trust Security Model to help secure sensitive data, systems, and services.
"Embracing a Zero Trust Security Model" (2021, as well) https://media.defense.gov/2021/Feb/25/2002588479/-1/-1/0/CSI...
In addition to "zero [trust]", I also looked for the term "SBOM". From p.32//39:
> As updates are deployed, administrators should also keep up with removing any old components that are no longer needed from the environment. Using a managed Kubernetes service can help to automate upgrades and patches for Kubernetes, operating systems, and networking protocols. *However, administrators must still patch and upgrade their containerized applications.*
"Existing artifact vuln scanners, databases, and specs?" https://github.com/google/osv/issues/55
Hosting SQLite Databases on GitHub Pages
That's one lovely trick.
If I may suggest one thing... instead of range requests on a single huge file how about splitting the file in 1-page fragments in separate files and fetching them individually? This buys you caching (e.g. on CDNs) and compression (that you could also perform ahead of time), both things that are somewhat tricky with a single giant file and range requests.
With the reduction in size you get from compression, you can also use larger pages almost for free, potentially further decreasing the number of roundtrips.
There's also a bunch of other things that could be tried later, like using a custom dictionary for compressing the individual pages.
I think that's the core innovation here, smart HTTP block storage.
I wonder if there has been any research into optimizing all http range requests at the client level in a similar way. i.e. considering the history of requests on a particular url and doing the same predictive exponential requests, or grabbing the full file asynchronously at a certain point.
> Methods for remotely accessing/paging data in from a client when a complete download of the dataset is unnecessary:
> - Query e.g. parquet on e.g. GitHub with DuckDB: duckdb/test_parquet_remote.test https://github.com/duckdb/duckdb/blob/6c7c9805fdf1604039ebed...
> - Query sqlite on e.g. GitHub with SQLite: [Hosting SQLite databases on Github Pages - (or any static file hoster) - phiresky's blog](...)
>> The above query should do 10-20 GET requests, fetching a total of 130 - 270KiB, depending on if you ran the above demos as well. Note that it only has to do 20 requests and not 270 (as would be expected when fetching 270 KiB with 1 KiB at a time). That’s because I implemented a pre-fetching system that tries to detect access patterns through three separate virtual read heads and exponentially increases the request size for sequential reads. This means that index scans or table scans reading more than a few KiB of data will only cause a number of requests that is logarithmic in the total byte length of the scan. You can see the effect of this by looking at the “Access pattern” column in the page read log above.
> - bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent
>> Sqltorrent is a custom VFS for sqlite which allows applications to query an sqlite database contained within a torrent. Queries can be processed immediately after the database has been opened, even though the database file is still being downloaded. Pieces of the file which are required to complete a query are prioritized so that queries complete reasonably quickly even if only a small fraction of the whole database has been downloaded.
>> […] Creating torrents: Sqltorrent currently only supports torrents containing a single sqlite database file. For efficiency the piece size of the torrent should be kept fairly small, around 32KB. It is also recommended to set the page size equal to the piece size when creating the sqlite database
Would BitTorrent be faster over HTTP/3 (UDP) or is that already a thing for web seeding?
> - https://web.dev/file-system-access/
> The File System Access API: simplifying access to local files: The File System Access API allows web apps to read or save changes directly to files and folders on the user’s device
Hadn't seen wilsonzlin/edgesearch, thx:
> Serverless full-text search with Cloudflare Workers, WebAssembly, and Roaring Bitmaps https://github.com/wilsonzlin/edgesearch
>> How it works: Edgesearch builds a reverse index by mapping terms to a compressed bit set (using Roaring Bitmaps) of IDs of documents containing the term, and creates a custom worker script and data to upload to Cloudflare Workers
> Would BitTorrent be faster over HTTP/3 (UDP) or is that already a thing for web seeding?
The BT protocol itself runs on both TCP and UDP, but it has preferred the UDP variant for many years already.
Thanks. There likely are relative advantages to HTTP/3 QUIC. Here's this from Wikipedia:
> Both HTTP/1.1 and HTTP/2 use TCP as their transport. HTTP/3 uses QUIC, a transport layer network protocol which uses user space congestion control over the User Datagram Protocol (UDP). The switch to QUIC aims to fix a major problem of HTTP/2 called "head-of-line blocking": because the parallel nature of HTTP/2's multiplexing is not visible to TCP's loss recovery mechanisms, a lost or reordered packet causes all active transactions to experience a stall regardless of whether that transaction was impacted by the lost packet. Because QUIC provides native multiplexing, lost packets only impact the streams where data has been lost.
And HTTP Pipelining / Multiplexing isn't specified by just UDP or QUIC:
> HTTP/1.1 specification requires servers to respond to pipelined requests correctly, sending back non-pipelined but valid responses even if server does not support HTTP pipelining. Despite this requirement, many legacy HTTP/1.1 servers do not support pipelining correctly, forcing most HTTP clients to not use HTTP pipelining in practice.
> Time diagram of non-pipelined vs. pipelined connection The technique was superseded by multiplexing via HTTP/2,[2] which is supported by most modern browsers.[3]
> In HTTP/3, the multiplexing is accomplished through the new underlying QUIC transport protocol, which replaces TCP. This further reduces loading time, as there is no head-of-line blocking anymore https://en.wikipedia.org/wiki/HTTP_pipelining
Ask HN: Any good resources on how to be a great technical advisor to startups?
Bumping up https://news.ycombinator.com/item?id=27600539
## Codelabels: Component: title
### ENH,UBY: HN: linkify URIs in descriptions
## User Stories
Users {__, __, } can ___ in order to ___.
Given-When-Then
~ Who-What-Wow
~ {Marketing, Training, Support, Service} Curriculum Competencies
### Users can click on links in descriptions in order to review referenced off-site resources.
Costs/Benefits: Linkspam?
The URL from this {item,} description: https://news.ycombinator.com/item?id=27600539
Teaching other teachers how to teach CS better
git and HTML and Linked Data should be requisite: https://learngitbranching.js.org/
Pedagogy#Modern_pedagogy: https://en.wikipedia.org/wiki/Pedagogy#Modern_pedagogy
Evidence-based_education: https://en.wikipedia.org/wiki/Evidence-based_education
Computational_thinking#Characteristics: https://en.wikipedia.org/wiki/Computational_thinking#Charact... (Abstraction, Automation, Analysis)
Learning: https://en.wikipedia.org/wiki/Learning
Autodidacticism: https://en.wikipedia.org/wiki/Autodidacticism
Design of Experiments; Hypotheses, troubleshooting, debugging, automated testing, Formal Methods, actual Root Cause Analysis: https://en.wikipedia.org/wiki/Design_of_experiments
Critical Thinking; definitions, Logic and Rationality, Logical Reasoning: Deduction, Abduction and Induction: https://en.wikipedia.org/wiki/Critical_thinking#Logic_and_ra...
Doesn't this all derive from [Quantum] Information Theory? It's actually fascinating to start at Information Theory; who knows what that curriculum would look like without reinforcement and [3D] videos: https://en.wikipedia.org/wiki/Information_theory
Stone, James V. "Information theory: a tutorial introduction." (2015). https://scholar.google.com/scholar?q=%22Information+Theory:+...
It used to be that we had to start engines with a turn of a crank: that initial energy to overcome inertia was enough for the system to feed-forward without additional reinforcement. Effective CS instruction may motivate the unmotivated to care about learning the way folks who are receiving reinforcement do: intrinsically.
Ask HN: Best online speech / public speaking course?
Hi HN - Has anyone taken an online course to help them with public speaking, speech and voice skills that they’d highly recommend? Thanks!
"TED Talks: The Official TED Guide to Public Speaking" https://smile.amazon.com/TED-Talks-Official-Public-Speaking-...
TED Masterclass: https://masterclass.ted.com/
"Power Talk: Using Language to Build Authority and Influence" https://smile.amazon.com/Power-Talk-Language-Authority-Influ...
Re: Clean Language and Symbolic Modeling; listening to metaphors and asking clean questions may be a more effective way to facilitate change: https://westurner.github.io/hnlog/#comment-15471868
/? greatest speeches: https://m.youtube.com/results?sp=mAEA&search_query=Greatest+...
"Lend Me Your Ears: Great Speeches in History" by William Safire. https://a.co/8svyoUw
E.g. "The Prosperity Bible: The Greatest Writings of All Time on the Secrets to Wealth and Prosperity" (Napoleon Hill, PT Barnum, Dale Carnegie, Gibran, Benjamin Franklin; 5000+ pages). https://a.co/b8Ej6o7
Talking points: Peaceful coexistence, #GlobalGoals 1-17 (UN SDGs), "Limits to Growth: The 30-Year Update" by Donella H. Meadows. https://a.co/7MgO0bv
Google sunsets the APK format for new Android apps
I was just trying to explain this the other day. Not sure whether to be disappointed in is this a regression? No, bros, you may not just `repack it` and re-sign the package for me. That's not how it should work unless I trust their build server to sign for me; and I don't and we shouldn't. I'll just CC this here from https://westurner.github.io/hnlog/#comment-27410978 :
```
> Unfortunately all packages aren't necessarily signed either; "Which package managers require packages to be cryptographically signed?" is similar to "Which DNS clients can operate DNS resolvers that require DNSSEC signatures on DNS records to validate against the distributed trust anchors?".
> FWIW, `delv pkg.mirror.server.org` is how you can check DNSSEC:
man systemd-resolved # nmcli
man delv
man dnssec-trust-anchors.d
delv pkg.mirror.server.org
> The TUF Overview explains some of the risks of asset signature systems; key compromise, there's one key for everything that we all share and can't log the revocation of in a CT (Certificate Transparency) log distributed like a DLT, https://theupdateframework.io/overview/
> Certificate Transparency: https://en.wikipedia.org/wiki/Certificate_Transparency
> Yeah, there's a channel to secure there at that layer of the software supply chain as well.
> "PEP 480 -- Surviving a Compromise of PyPI: End-to-end signing of packages" (2014-) https://www.python.org/dev/peps/pep-0480/
>> Proposed is an extension to PEP 458 that adds support for end-to-end signing and the maximum security model. End-to-end signing allows both PyPI and developers to sign for the distributions that are downloaded by clients. The minimum security model proposed by PEP 458 supports continuous delivery of distributions (because they are signed by online keys), but that model does not protect distributions in the event that PyPI is compromised. In the minimum security model, attackers who have compromised the signing keys stored on PyPI Infrastructure may sign for malicious distributions. The maximum security model, described in this PEP, retains the benefits of PEP 458 (e.g., immediate availability of distributions that are uploaded to PyPI), but additionally ensures that end-users are not at risk of installing forged software if PyPI is compromised.
> One W3C Linked Data way to handle https://schema.org/SoftwareApplication ( https://codemeta.github.io/user-guide/ ) cryptographic signatures of a JSON-LD manifest with per-file and whole package hashes would be with e.g. W3C ld-signatures/ld-proofs and W3C DID (Decentralized Identifiers) or x.509 certs in a CT log.
```
FWIU, the Fuschia team is building package signing on top of TUF.
A from-scratch tour of Bitcoin in Python
This is reminds me of Ken Shirriff's 2014 "Bitcoins the Hard Way" blog post that also used Python to build a Bitcoin transaction from scratch: http://www.righto.com/2014/02/bitcoins-hard-way-using-raw-bi...
(The subtitle of the blog is "Computer history, restoring vintage computers, IC reverse engineering, and whatever" and it is full of fascinating articles, several of which have been featured here on HN)
> The 'dumbcoin' jupyter notebook is also a good reference: "Dumbcoin - An educational python implementation of a bitcoin-like blockchain" https://nbviewer.jupyter.org/github/julienr/ipynb_playground...
https://github.com/yjjnls/awesome-blockchain#implementation-... and https://github.com/openblockchains/awesome-blockchains#pytho... list a few more ~"blockchain from scratch" [in Python] examples.
... FWIU, Ethereum has the better Python story. There was a reference implementation of Ethereum in Python? https://ethereum.org/en/developers/docs/programming-language...
An Omega-3 that’s poison for cancer tumors
If you are interested in adding Omega-3 to your stack, make sure it is molecularly distilled fish oil. Otherwise there is the risk of excess lead and mercury.
You can also eat fish that are low on the food chain like sardines.
Fish don't synthesize Omega PUFAs, they eat algae (which unfortunately and inopportunely stains teeth)
From "Warning: Combination of Omega-3s in Popular Supplements May Blunt Heart Benefits" https://scitechdaily.com/warning-combination-of-omega-3s-in-... :
> Now, new research from the Intermountain Healthcare Heart Institute in Salt Lake City finds that higher EPA blood levels alone lowered the risk of major cardiac events and death in patients, while DHA blunted the cardiovascular benefits of EPA. Higher DHA levels at any level of EPA, worsened health outcomes.
> Results of the Intermountain study, which examined nearly 1,000 patients over a 10-year-period,
> “Based on these and other findings, we can still tell our patients to eat Omega-3 rich foods, but we should not be recommending them in pill form as supplements or even as combined (EPA + DHA) prescription products,” he said. “Our data adds further strength to the findings of the recent REDUCE-IT (2018) study that EPA-only prescription products reduce heart disease events.”
Now they're sayin'; so I go look for an EPA-only supplement, and TIL about re-esterified triglyceride and it says it's molecularly distilled anchovies in blister packages. Which early land mammals probably ate, so.
I found this comment too late... Already ordered a "kids" supplement that has a 2:1 ratio of DHA:EPA.
I do not think that you should be much concerned about this, because other studies show that more DHA than EPA is preferable when using other criteria.
So the conclusion is that nobody knows for sure if you should eat more DHA than EPA, to have a better brain, or less DHA than EPA, to have a better heart.
What is certain is that you need some minimum quantity of both DHA and EPA.
Especially for children, who do not have yet to worry about cardio-vascular problems, a supplement with more DHA than EPA seems actually a good choice.
Discover and Prevent Linux Kernel Zero-Day Exploit Using Formal Verification
[Coq, VST, CompCert]
Formal methods: https://en.wikipedia.org/wiki/Formal_methods
Formal specification: https://en.wikipedia.org/wiki/Formal_specification
Implementation of formal specification: https://en.wikipedia.org/wiki/Anti-pattern#Software_engineer...
Formal verification: https://en.wikipedia.org/wiki/Formal_verification
From "Why Don't People Use Formal Methods?" https://news.ycombinator.com/item?id=18965964 :
> Which universities teach formal methods?
> - q=formal+verification https://www.class-central.com/search?q=formal+verification
> - q=formal+methods https://www.class-central.com/search?q=formal+methods
> Is formal verification a required course or curriculum competency for any Computer Science or Software Engineering / Computer Engineering degree programs?
Can there still be side channel attacks in formally verified systems? Can e.g. TLA+ help with that at all?
Formal methods could be used to prevent side channel attacks. However preventing each type of attack requires specifying properties which needs to be proven to assure it is is not possible. I am not very famililiar with TLA+, but I think in general it is not guaranteed to provide side channel attacks, such as based on timing, CPU state, etc.
Anatomy of a Linux DNS Lookup
This predates systemd's decision to get involved via systemd-resolved. So now it's got another step :)
Edit: Well, and also browsers doing DNS over https. It's pretty confusing these days to know which path your app took to resolve a name.
Which is super annoying when you actually want to run a DNS server and have to try and convince systemd-resolved to stay in its own lane.
Is there a good example of a Linux package that does this correctly?
I suspect the big part is putting "DNSStubListener=no" into the [Resolve] section of /etc/systemd/resolved.conf if you want to have something else listening on port 53.
Or you can just mask the service (better than disabling):
systemctl mask systemd-resolved.service
systemctl mask systemd-hostnamed.service
systemctl mask systemd-timesyncd.service
systemctl mask systemd-networkd.service
systemctl mask systemd-homed.serviceYeah, but if you regress to 'legacy DNS' by removing systemd-resolved then there's no good way to do per-interface DNS (~client-split DNS), or (optionally) validate DNSSEC, or do DoH/DoT; and then nothing respawns and logs consistently-timestamped process events of substitute network service processes.
FWIU, per-user DNS configs are still elusive. Per-user DNS would make it easier to use family-safe DNS (that redirects to family-safe e.g. SafeSearch domains) by default; some forums are essential for system administration.
Since nothing in the real world is or ever will be DNSSEC signed anyways, it's not that much of a loss. Meanwhile, the most important DoH user --- your browser --- has DoH baked in and doesn't need systemd-resolved's help.
Your system may also depend upon one or more package managers that do all depend upon DNS (and hopefully e.g. DNSSEC and DoH/DoT)
Package managers don't rely on DNSSEC for package security (it would be deeply problematic if they did). But you can also just go look and see that, for instance, Ubuntu.com isn't signed, nor are most of the Pacman mirrors for Arch, nor is alpinelinux.org, &c. DNSSEC is simply not a factor in package management security.
People have weird ideas about how this stuff works.
Unfortunately all packages aren't necessarily signed either; "Which package managers require packages to be cryptographically signed?" is similar to "Which DNS clients can operate DNS resolvers that require DNSSEC signatures on DNS records to validate against the distributed trust anchors?".
FWIW, `delv pkg.mirror.server.org` is how you can check DNSSEC:
man systemd-resolved # nmcli
man delv
man dnssec-trust-anchors.d
delv pkg.mirror.server.org
The TUF Overview explains some of the risks of asset signature systems; key compromise, there's one key for everything that we all share and can't log the revocation of in a CT (Certificate Transparency) log distributed like a DLT, https://theupdateframework.io/overview/
Certificate Transparency: https://en.wikipedia.org/wiki/Certificate_Transparency
Yeah, there's a channel to secure there at that layer of the software supply chain as well.
"PEP 480 -- Surviving a Compromise of PyPI: End-to-end signing of packages" (2014-) https://www.python.org/dev/peps/pep-0480/
> Proposed is an extension to PEP 458 that adds support for end-to-end signing and the maximum security model. End-to-end signing allows both PyPI and developers to sign for the distributions that are downloaded by clients. The minimum security model proposed by PEP 458 supports continuous delivery of distributions (because they are signed by online keys), but that model does not protect distributions in the event that PyPI is compromised. In the minimum security model, attackers who have compromised the signing keys stored on PyPI Infrastructure may sign for malicious distributions. The maximum security model, described in this PEP, retains the benefits of PEP 458 (e.g., immediate availability of distributions that are uploaded to PyPI), but additionally ensures that end-users are not at risk of installing forged software if PyPI is compromised.
One W3C Linked Data way to handle https://schema.org/SoftwareApplication ( https://codemeta.github.io/user-guide/ ) cryptographic signatures of a JSON-LD manifest with per-file and whole package hashes would be with e.g. W3C ld-signatures/ld-proofs and W3C DID (Decentralized Identifiers) or x.509 certs in a CT log.
JupyterLite – WASM-powered Jupyter running in the browser
Despite Pyolite has a miserable performance (20MB of downloads), the overall project direction is correct.
I said this already 10 years ago: We don't need more cloud computing but need to empower users end devices again. Jupyter is typically operated on powerful notebooks and not on mobile devices.
Isn't that not just Java all over again, but this time with JavaScript?
There are crucial differences between Java applets and JS.
- Applets tried to render their own GUI, Wasm doesn't and defers to the browser.
- applets needed a big, slow to start and resource hungry VM. Wasm is running in the same thread your JS is also running in, it's light, and loads faster than JS
- Java and flash were plugins, which needed to be installed and kept up to date separately. Wasm is baked into your browser's JS engine
- Wasm code is very fast and can achieve near native execution speeds. It can make use of advanced optimisations. SIMD has shipped in Chrome, and will soon in Firefox
- The wasm spec is very, very good, and really quite small. This means that implementing it is comparatively cheap, and this should make it easy to see it implemented by different vendors.
- Java was just Java. Wasm can serve as a platform for any language. See my earlier point about the spec
So it's apples and oranges. The need to have something besides JS hasn't gone away, so their use cases might be similar. The two technologies couldn't be more distinct, though.
You must view the browser with JS and WASM as a unit.
The browser renders it's own GUI too, it's not OS native
The browser uses lots of resources too.
The browser is kind of a plugin to the OS and must be updated separately.
Java nowadays is pretty fast too.
Java VM serves a platform for multiple languages like Scala, Kotlin, Clojure.
Let's face it, the browser is the new JVM and a soon it gets the same permissions like the JVM to access the file system and such, we get the same problems.
From https://news.ycombinator.com/item?id=24052393 re: Starboard:
> https://developer.mozilla.org/en-US/docs/Web/Security/Subres... : "Subresource Integrity (SRI) is a security feature that enables browsers to verify that resources they fetch (for example, from a CDN) are delivered without unexpected manipulation. It works by allowing you to provide a cryptographic hash that a fetched resource must match."
> There's a new Native Filesystem API: "The new Native File System API allows web apps to read or save changes directly to files and folders on the user's device." https://web.dev/native-file-system/
> We'll need a way to grant specific URLs specific, limited amounts of storage.
[...]
> https://github.com/deathbeds/jyve/issues/46 :
> Would [Micromamba] and conda-forge build a WASM architecture target?
Accenture, GitHub, Microsoft and ThoughtWorks Launch the GSF
> With data centers around the world accounting for 1% of global electricity demand, and projections to consume 3-8% in the next decade, it’s imperative we address this as an industry.
> To help in that endeavor, we’re excited to announce the formation of The Green Software Foundation – a nonprofit founded by Accenture, GitHub, Microsoft and ThoughtWorks established with the Linux Foundation and the Joint Development Foundation Projects LLC to build a trusted ecosystem of people, standards, tooling and leading practices for building green software. The Green Software Foundation was born out of a mutual desire and need to collaborate across the software industry. Organizations with a shared commitment to sustainability and an interest in green software development principles are encouraged to join the foundation to help grow the field of green software engineering, contribute to standards for the industry, and work together to reduce the carbon emissions of software. The foundation aims to help the software industry contribute to the information and communications technology sector’s broader targets for reducing greenhouse gas emissions by 45% by 2030, in line with the Paris Climate Agreement.
Here's to now hand-optimized efficient EC, SHA-256, SHA-3, and Scrypt routines due to incentives. See also The Crypto Climate Accord, which is also inspired by the Paris Agreement: https://cryptoclimate.org/
... "Thermodynamics of Computation Wiki" https://news.ycombinator.com/item?id=18146854
Is 100% offset by PPAs always 200% Green?
From "Ask HN: What jobs can a software engineer take to tackle climate change?" https://news.ycombinator.com/item?id=20015801 :
> [ ] We should create some sort of a badge and structured data (JSONLD, RDFa, Microdata) for site headers and/or footers that lets consumers know that we're working toward '200% green' so that we can vote with our money.
>inspired by the Paris Agreement
So we’re just going to let China do the polluting? Maybe you didn’t know but the Paris Agreement exempted China from all standards.
No, under the Paris Agreement, countries set voluntary targets for themselves and regularly reassess.
And when China says it’s doing nothing everybody cheered.
TBF, the glut of [Chinese,] solar panels has significantly helped lower the cost of renewables; which is in everyone's interest.
They’re cheap because the secret ingredient is slavery and Uighur blood.
Rocky Linux releases its first release candidate
One of our clients pushed us toward AlmaLinux instead of CentOS 7. At the time, we vaguely suggested Rocky Linux, but noted we were kind of still waiting for the future (hence our suggestion of going with CentOS 7 and migrating later to the "true successor" of CentOS).
What is the HN community's perception of this? Is AlmaLinux a good choice? Do you believe Rocky Linux will succeed?
I’d say “neither”. CentOS would have fail had Redhat not taken over. My guess is that supporting all the companies who do not want to pay Redhat, for free, is going to push developers way pretty quickly.
Even if I’m wrong I’d still want to see at least two or three solid release before using either in production.
Just use Redhat or switch to Ubuntu. Then you can re-evaluate in five tears time.
> I’d say “neither”. CentOS would have fail had Redhat not taken over.
While it's possible that CentOS would have failed (I'm not sure slow initial releases actually indicate that), it also wasn't the only popular clone of RHEL in popular use. Scientific Linux (developed by FermiLab) was also popular and in fairly wide use. I doubt Red Hat would have made the change to CentOS that spurred all this if Scientific Linux hadn't decided not to continue and just use CentOS instead in 2019, because instead of a "crap, we have to start a new distro to provide what CentOS did" movement there would have been a much quicker and easier mass exodus to Scientific Linux.
The Scientific Linux team decided to not make a Scientific Linux 8, and instead switch over to CentOS 8[0]. Note this was announced prior to the announcement about CentOS 8 going away.
CERN et al are still deciding what to do[1].
[0]: https://listserv.fnal.gov/scripts/wa.exe?A2=SCIENTIFIC-LINUX...
[1]: https://linux.web.cern.ch/#update-on-centos-linux-strategy
USB-C is about to go from 100W to 240W, enough to power beefier laptops
What are the costs to add a USB PD module to an electronic device? https://hackaday.com/2021/04/21/easy-usb‑c-power-for-all-you...
- [ ] Create an industry standard interface for charging and using [power tool,] battery packs; and adapters
Half-Double: New hammering technique for DRAM Rowhammer bug
From "Rowhammer for qubits: is it possible?" https://amp.reddit.com/r/quantum/comments/7osud4/rowhammer_f... :
> Sometimes bits just flip due to "cosmic rays"; or, logically, also due to e.g. neutron beams and magnetic fields.
> With rowhammer, there are read/write (?) access patterns which cause predictable-enough information "leakage" to be useful for data exfiltration and privilege escalation.
> With the objective of modeling qubit interactions using quantum-mechanical properties of fields of electrons in e.g. DRAM, Is there a way to use DRAM electron "soft errors" to model quantum interactions; to build a quantum computer from what we currently see as errors in DRAM?
> If not with current DRAM, could one apply a magnetic field to DRAM in order to exploit quantum properties of electrons moving in a magnetic field?
https://en.wikipedia.org/wiki/DRAM
https://en.wikipedia.org/wiki/Row_hammer
https://en.wikipedia.org/wiki/Soft_error
https://en.wikipedia.org/wiki/Crosstalk
> [...] are there DRAM read/write patterns which cause errors due to interference which approximate quantum logic gates? Probably not, but maybe; especially with an applied magnetic field (which then isn't the DRAM sitting on our desks, it's then DRAM + a constant or variable field).
> I suppose to test this longshot theory, one would need to fuzz low-level RAM loads and search for outputs that look like quantum gate outputs. Or, monitor normal workloads which result in RAM faults which approximate quantum logic gate outputs and train a network to recognize the features.
> I am reminded of a recent approach to in-RAM computing that's not memristors.
> Soft errors caused by cosmic rays are obviously more frequent at higher altitudes (and outside of the Van Allen radiation belt).
Thought I'd ask this here as well.
Quantum tunneling was the perceived barrier at like DDR5 and higher densities FWIU? Barring new non-electron-based tech, how can we prevent adjacent electrons from just flipping at that gate grid gap size?
Other Quantum-on-Silicon approaches have coherence issues, too
Setting up a Raspberry Pi with 2 Network Interfaces as a simple router
Old but relevant, I have used an espressobin as a router for years and the performance I am getting is similar to much more expensive semi-professional routers. https://blog.tjll.net/building-my-perfect-router/
For those who wish to undergo lesser trouble to get similar performance/security there are number of Single Board Computers(SBCs) with OpenWrt support[1] sometimes with official support like those from Friendly Elec.
[1]https://openwrt.org/toh/views/toh_single-board-computers
[2]https://www.friendlyarm.com/index.php?route=product/category...
I'm constantly looking for good priced SBC with cellular connectivity - that chart shows only 2/3 with modems and all appear to be via external modules.
Anyone know of any?
> This page shows devices which have a LTE modem built in and are supported by OpenWrt.
https://openwrt.org/toh/views/toh_lte_modem_supported
It looks like this table is neither current nor complete though. And there's a different table of OpenWRT compatible devices that have a battery as well.
> [The Amarok (GL-X1200) Industrial IoT Gateway has] 2x SIM card slots for 2x 4G LTE modems (probably miniPCI-E so maybe upgradeable to 5G later), external antenna connectors for the LTE modems, MicroSD, #OpenWRT: https://store.gl-inet.com/collections/4g-smart-router/produc...
The Turris Omnia also has 4G LTE SIM card support (and LXC in their OpenWRT build). https://openwrt.org/toh/turris/turris_omnia
There's also a [Dockerized] x86 build of OpenWRT that probably also supports Mini PCI-E modules for 4G LTE, LoRa, and 5G. Route metrics determine which [gateway] route is tried first.
From "How much total throughput can your wi-fi router really provide?" https://news.ycombinator.com/item?id=26596395 :
> In 2021, most routers - even with OpenWRT and hardware-offloading - cannot actually push 1 Gigabit over wired Ethernet, though the port spec does say 1000 Mbps
What to do about GPU packages on PyPI?
“ Our current CDN “costs” are ~$1.5M/month and not getting smaller. This is generously supported by our CDN provider but is a liability for PyPI’s long-term existence.”
Wow
Bear in mind this isn't just end-users installing on their machines, it also includes continuous integration scripts that run quite frequently.
[Huge GPU] packages can be cached locally: persist ~/.cache/pip between builds with e.g. Docker, run a PyPI caching proxy,
"[Discussions on Python.org] [Packaging] Draft PEP: PyPI cost solutions: CI, mirrors, containers, and caching to scale" https://discuss.python.org/t/draft-pep-pypi-cost-solutions-c...
> Continuous Integration automated build and testing services can help reduce the costs of hosting PyPI by running local mirrors and advising clients in regards to how to efficiently re-build software hundreds or thousands of times a month without re-downloading everything from PyPI every time.
[...]
> Request from and advisory for CI Services and CI Implementors:
> Dear CI Service,
> - Please consider running local package mirrors and enabling use of local package mirrors by default for clients’ CI builds.
> - Please advise clients regarding more efficient containerized software build and test strategies.
> Running local package mirrors will save PyPI (the Python Package Index, a service maintained by PyPA, a group within the non-profit Python Software Foundation) generously donated resources. (At present (March 2020), PyPI costs ~ $800,000 USD a month to operate; even with generously donated resources).
Looks like the current figure is significantly higher than $800K/mo for science.
How to persist ~/.cache/pip between builds with e.g. Docker in order to minimize unnecessary GPU package re-downloads:
RUN --mount=type=cache,target=/root/.cache/pip
RUN --mount=type=cache,target=/home/appuser/.cache/pipWow, that RUN trick is exactly what I've been looking for! I've spent hours and hours in Docker documentation and hadn't seen that functionality.
Looks like it might be buildkit-specific?
Markdown Notes VS Code extension: Navigate notes with [[wiki-links]]
> Syntax highlighting for #tags.
What's the best way to search for #tags with VS Code? Are #tags indexed into an e.g. ctags file within a project or a directory?
> @bibtex-citations: Use pandoc-style citations in your notes (eg @author_title_year) to get syntax highlighting, autocompletion and go to definition, if you setup a global BibTeX file with your references.
Open VS Code search across files (Press Ctrl+Shift+F on windows) and type the tag that you are searching for: #tag
Simple as that https://code.visualstudio.com/docs/editor/codebasics#_search...
Ask HN: Choosing a language to learn for the heck of it
I'm a technical manager, which means I do a lot of administrative stuff and a little coding. The coding has become a nice distraction when I need to take a break.
For "real work" I write mostly Python, a lot of SQL, a little bit of Go, and some shell scripting to glue it together. I'd like to learn something I have no need of for work. If it becomes useful later, that is OK, but not a goal. The goal is in creating something just for fun. That something is undefined, so general purpose languages are the population.
I have become curious lately in Nim, Crystal, and Zig. Small, modern, high performance languages. Curiousity comes from the cases when they are mentioned here, sometime for similar reasons I list above.
Nim is on top of the list: Sort of Python like, supported on Windows (I use Win/Mac/Linux), appears to have libraries for the things I do: Process text for insights, play projects would use interesting data instead of business data.
Crystal does not support Windows (yet), but appears to closer to Ruby. Its performance may be a bit better.
Zig came on my radar recently, I know less about it, compared to the little I know of the others.
Suggestions on choosing one as a hobby language?
> Suggestions on choosing one as a hobby language?
IDK how much of a hobby it'd remain, but: Rust compiles to WASM, C++ now has auto and coroutines (and real live memory management)
"Ask HN: Is it worth it to learn C in 2020?" https://news.ycombinator.com/item?id=21878664
Show HN: Django SQL Dashboard
This looks great! In a similar vein - does anyone know of a project that will allow for a Django shell in the browser?
I know Jupyter exists - but a solution like this with the permissions would be valuable.
This launches the web-based Werkzeug debugger on Exception:
pip install django-extensions
python manage.py runserver_plus
This should run IPython Notebook with database models already imported :
python manage.py shell_plus --notebook
Interactive IPA Chart
Is there a [Linked Data] resource with the information in this interactive IPA chart (which is from Wikipedia FWICS) in addition to?:
- phoneme, ns:"US English letter combinations", []
- phoneme, ns:"schema.org/CreativeWorks which feature said phoneme", []
AFAIU, WordNet RDF doesn't have links to any IPA RDFS/OWL vocabulary/ontology yet.
Google Dataset Search
Information on how to annotate datasets: https://developers.google.com/search/docs/data-types/dataset
> We can understand structured data in Web pages about datasets, using either schema.org Dataset markup, or equivalent structures represented in W3C's Data Catalog Vocabulary (DCAT) format. We also are exploring experimental support for structured data based on W3C CSVW, and expect to evolve and adapt our approach as best practices for dataset description emerge. For more information about our approach to dataset discovery, see Making it easier to discover datasets.
For more info on those:
- W3C's Data Catalog Vocabulary: https://www.w3.org/TR/vocab-dcat-3/
- Schema.org dataset: https://schema.org/Dataset
- CSVW Namespace Vocabulary Terms: https://www.w3.org/ns/csvw
- Generating RDF from Tabular Data on the Web (examples on how to use CSVW): https://www.w3.org/TR/csv2rdf/
Use cases for such [LD: Linked Data] metadata:
1. #StructuredPremises:
> (How do I indicate that this is a https://schema.org/ScholarlyArticle predicated upon premises including this Dataset and these logical propositions?)
2. #LinkedMetaAnalyses; #LinkedResearch "#StudyGraph"
3. [CSVW (Tabular Data Model),] schema.org/Dataset(s) with per column (per-feature) physical quantity and unit URIs with e.g. QUDT and/or https://schema.org/StructuredValue metadata for maximum data reusability.
4. JupyterLab notebooks:
4a. JupyterLab Metadata Service extension: https://github.com/jupyterlab/jupyterlab-metadata-service :
> - displays linked data about the resources you are interacting with in JuyterLab.
> - enables other extensions to register as linked data providers to expose JSON LD about an entity given the entity's URL.
> - exposes linked data to the user as a Linked Data viewer in the Data Browser pane.
4b. JupyterLab Data Explorer: https://github.com/jupyterlab/jupyterlab-data-explorer :
> - Data changing on you? Use RxJS observables to represent data over time.
> - Have a new way to look at your data? Create React or lumino components to view a certain type.
> - Built-in data explorer UI to find and use available datasets.
Ask HN: Cap Table Service Recommendations
Recent founders, do you have any recommendations for services for managing a cap table? Or do you do it yourself? Any suggestions for how to choose?
Here are the "409a valuation" reviews on FounderKit: https://founderkit.com/legal/409a-valuation/reviews
Hosting SQLite databases on GitHub Pages or any static file hoster
The innovation here is getting sql.js to use http and range requests for file access rather than all being in memory.
I wonder when people using next.js will start using this for faster builds for larger static sites?
See also https://github.com/bittorrent/sqltorrent, same trick but using BitTorrent
Yeah, that was one of the inspirations for this. That one does not work in the browser though, would be a good project to do that same thing but with sqlite in wasm and integrated with WebTorrent instead of a native torrent program.
I actually did also implement a similar thing fetching data on demand from WebTorrent (and in turn helping to host the data yourself by being on the website): https://phiresky.github.io/tv-show-ratings/ That uses a protobufs split into a hashmap instead of SQLite though.
This looks pretty efficient. Some chains can be interacted with without e.g. web3.js? LevelDB indexes aren't SQLite.
Datasette is one application for views of read-only SQLite dbs with out-of-band replication. https://github.com/simonw/datasette
There are a bunch of *-to-sqlite utilities in corresponding dogsheep project.
Arrow JS for 'paged' browser client access to DuckDB might be possible and faster but without full SQLite SQL compatibility and the SQLite test suite. https://arrow.apache.org/docs/js/
> Direct Parquet & CSV querying
In-browser notebooks like Pyodide and Jyve have local filesystem access with the new "Filesystem Access API", but downloading/copying all data to the browser for every run of a browser-hosted notebook may not be necessary. https://web.dev/file-system-access/
DuckDB can directly & selectively query Parquet files over HTTP/S3 as well. See here for examples: https://github.com/duckdb/duckdb/blob/6c7c9805fdf1604039ebed...
Wasm3 compiles itself (using LLVM/Clang compiled to WASM)
Self-hosting (compilers) https://en.wikipedia.org/wiki/Self-hosting_(compilers) :
> In computer programming, self-hosting is the use of a program as part of the toolchain or operating system that produces new versions of that same program—for example, a compiler that can compile its own source code
How big of a milestone is self-hosting for a language?
Semgrep: Semantic grep for code
Is there a more complete example of how to call semgrep from pre-commit (which gets called before every git commit) in order to prevent e.g. Python print calls (print(), print \\n(), etc.) from being checked in?
https://semgrep.dev/docs/extensions/ describes how to do pre-commit.
Nvm, here's semgrep's own .pre-commit-config.yml for semgrep itself: https://github.com/returntocorp/semgrep/blob/develop/.pre-co...
I've never used the `pre-commit` framework, but it's really simple to wire up arbitrary shell scripts; check out the
`.git/hooks` directory in your repo for samples, e.g. `.git/hooks/pre-commit.sample`.
You can run any old shell script there, without having to install a python tool.
Yeah but that githook will only be installed on that one repo on that one machine. And they may have no or a different version of bash installed (on e.g. MacOS or Windows). IMHO, POSIX-compatible portable shell scripts are more trouble than portable Python scripts.
Pre-commit requires Python and pre-commit to be installed (and then it downloads every hook function).
This fetches the latest version of every hook defined in the .pre-commit-config.yml:
pre-commit autoupdate
A person could easily `ln -s repo/.hooks/hook*.sh repo/.git/hooks/` after every git clone.
Out of curiosity, Is there value in doing this over (say) running a GitHub Action post commit and failing the build if it finds something nasty?
If you can catch it before the commit is even made then why do/wait for a build?
Fair enough. Guess IDE plugins work even better for that
IDE plugins are not at all consistent from one IDE to another. Pre-commit is great for teams with different IDEs because all everyone needs to do is:
[pip,] install pre-commit
pre-commit install
# git commit
# pre-commit run --all-files
# pre-commit autoupdate
Ask HN: What to use instead of Bash / Sh for scripting?
I'm at the point where I feel a certain fatigue writing Bash scripts, but I am just not sure of what the alternative is for medium sized (say, ~150-500 LOC) scripts.
The common refrain of "use Python" hasn't really worked fantastically: I don't know what version of Python I'm going to have on the system, installing dependencies is not fun, shelling out when needed is not pleasant, and the size of program always seemingly doubles.
I'm willing to accept something that's not on the system as long as it's one smallish binary that's available in multiple architectures. Right now, I've settled on (ab)using jq, using it whenever tasks get too complex, but I'm wondering if anyone else has found a better way that should also hopefully not be completely a black box to my colleagues?
A configuration management system may have you write e.g. YAML with Jinja2 so that you don't reinvent the idempotent wheel.
It's really easy to write dangerous shell scripts ("${@}" vs ${@} for example) and also easy to write dangerous Python scripts (cmd="{}; {}").
Sarge is one way to use subprocess in Python. https://sarge.readthedocs.io/en/latest/
If you're doing installation and configuration, the most team-maintainable thing is to avoid custom code and work with a configuration management system test runner.
When you "A shell script will be fine, all I have to do is [...]" and then you realize that you need a portable POSIX shell script and to be merged it must have actual automated tests of things that are supposed to run as root - now in a fresh vm/container for testing - and manual verification of `set +xev` output isn't an automated assertion.
> avoid custom code and work with a configuration management system test runner
ansible-molecule is a test runner for Ansible playbooks that can create VMs or containers on local or remote resources.
You can definitely just call shell scripts from Ansible, but the (parallel) script output is only logged after the script returns a return code unless you pipe the script output somewhere and tail that .
> manual verification of `set +xev` output isn't an automated assertion.
From "Bash Error Handling" https://news.ycombinator.com/item?id=24745833 : you can display the line number in `set -x` output by setting $PS4:
export PS4='+(${BASH_SOURCE}:${LINENO}) '
set -x
Estonian Electronic Identity Card and Its Security Challenges [pdf]
Proud e-resident here :)
US citizen here trying to champion such a system in the US. Would you be willing to share pros and cons from your experience using the system day to day?
Are you working with a specific org or initiative? I'm also an American interested in this (and an e-resident who formerly worked for the Estonian government).
Mostly activist citizen efforts from the outside, as legislation is going to be required to appropriate funding and direction from Congress, and the legislators I interface with are busy with arguably more pressing work (unfortunate but entirely understandable, such are the times).
I intend to apply at the USDS for the Login.gov team in some capacity to help on the tech side if the necessary legislation can be put in place to support such an initiative. Their system already supports the DOD CAC (common access card), which is a short walk away from a citizen digital ID card (would be a different org and PKI root to administer and govern citizen cards, to grossly simplify).
Login.gov recently expanded to support city and local gov IAM needs (when they have ties to federal programs) [1] [2], so there is roadmap momentum and executive branch will. "Digital identity is a big deal. [3]" They're already serving 30 million users, and 500k DAUs, really just a matter of scaling up.
[1] https://www.gsa.gov/blog/2021/02/18/logingov-to-provide-auth...
[2] https://www.govloop.com/login-gov-expands-use-to-cities-stat...
I get the spirit and a m familiar with the CAC and general benefits, and using that for all .gov stuff is attractive (taxes, bills, FASFA, etc)
I think you’re glossing over two major implementation factors that need to be part of the discussion from get-go:
* how much CAC use is dependent on fairly specific govt tech infra to be widely deployed, and how will that work for everyone (ever tried to setup a CAC on a civ computer)
* key control, either extremely decentralized like the iPhone (lock yourself out of your passport?), or extremely centralized and the newest honeypot OPM holds (root cert for all e-citizen PKI). US currently doesn’t have the internal cybersec chops to run that at all (closest equivalent is CISA).
You're right to point this out, but my counter argument is that these are solvable pain points for such an implementation (either done today in competent zero trust security architectures in progressive orgs or at nation state levels such as Estonia). You won't need a CAC reader on computers, for the most part, if you have mobile apps that can perform the identity proofing (such that's already done with examples like Apple's biometrics systems, FIDO2/WebAuthn, Apply Pay, etc). You'd still have the card for interfacing at endpoints (banks, postal service, IRS/SSA offices, other trust anchors and government services endpoints). I already login to my US CBP Global Entry account with Login.gov and 2FA, why can I not today use the same IAM system to login to my Social Security account? Or my IRS tax account? Or to attest to my citizenship or other attributes that I'd normally need a certified document for (yuk!).
I'm not arguing for such a system without robust support and reasonable downgrades for failure scenarios (identity reproofing if you lose your digital ID, for example). I'm arguing for, admittedly challenging, digital ID modernization without disenfranchisement. I genuinely appreciate you pointing out the challenges, as they must be addressed.
US DHS CISA is absolutely a resource that needs to be leaned on heavily to implement what I describe, and to ensure a strong security posture throughout the federal government's infrastructure.
FWIU, DHS has funded [1] development of e.g W3C DID Decentralized Identifiers [2] and W3C Verifiable Credentials [3]:
[1] https://www.google.com/search?q=site%3Aw3.org+%22funded+by+t...
[2] https://www.w3.org/TR/did-core/
[3] https://www.w3.org/TR/vc-data-model/
Additional notes regarding credentials (certificates, badges, degrees, honorarial degrees, then-evaluated competencies) and capabilities models: https://news.ycombinator.com/item?id=19813340
westurner/blockchain-credential-resources.md: https://gist.github.com/westurner/4345987bb29fca700f52163c33...
Value storage and transmission networks have developed standards and implementations for identity, authentication, and authorization. ILP (Interledger Protocol) RFC 15 specifies "ILP addresses" for [crypto] ledger account IDs: https://interledger.org/rfcs/0015-ilp-addresses/
From "Verifiable Credentials Use Cases" https://w3c.github.io/vc-use-cases/ :
> A verifiable claim is a qualification, achievement, quality, or piece of information about an entity's background such as a name, government ID, payment provider, home address, or university degree. Such a claim describes a quality or qualities, property or properties of an entity which establish its existence and uniqueness. The use cases outlined here are provided in order to make progress toward possible future standardization and interoperability of both low- and high-stakes claims with the goals of storing, transmitting, and receiving digitally verifiable proof of attributes such as qualifications and achievements. The use cases in this document focus on concrete scenarios that the technology defined by the group should address.
FWIU, the US Department of Education is studying or already working with https://blockcerts.org/ for educational credentials.
Here are the open sources of blockchain-certificates/cert-issuer and blockchain-certificates/cert-verifier-js: https://github.com/blockchain-certificates
Might a natural-born resident get a government ID card for passing a recycling and environmental sustainability quiz.
Systemd makes life miserable, again, this time by breaking DNS
So, I made the mistake of updating my laptop from Fedora 31 to Fedora 33 last night. Normally this is fairly painless, as my laptop is one of the last machines I perform distribution upgrades. Today while doing some pole survey work out in the field, I tethered my laptop to my phone as has been done hundreds of times before. To my surprise, DNS doesn't work anymore, but only in web browsers. Both Firefox and Chrome can't resolve names anymore. Command line tools like ping and host work normally. WTF?
Why are distributions continuing to allow systemd to extend its tentacles deeper and deeper into more parts of Linux userland with poorly tested subsystem replacements for parts of Linux that have been stable for decades? Does nobody else consider this repeating pattern of rewrite-replace-introduce-new-bugs a problem? Newer is not all that better if you break what is a pretty bog standard and common use-case.
As well, Firefox now defaults to DoH (DNS over HTTPS), which may be bypassing systemd-resolved by doing DNS resolution in the app instead of calling `gethostbyname()` (`man gethostbyname`) and/or `getaddrinfo()`.
`man systemd-resolved` describes why there is new DNS functionality: security; "caching and validating DNS/DNSSEC stub resolver, as well as an LLMR and MulticastDNS resolver and responder".
From `man systemd-resolved` https://man7.org/linux/man-pages/man8/systemd-resolved.servi... :
> To improve compatibility, /etc/resolv.conf is read in order to discover configured system DNS servers, but only if it is not a symlink to /run/systemd/resolve/stub-resolv.conf, /usr/lib/systemd/resolv.conf or /run/systemd/resolve/resolv.conf
> [...] Note that the selected mode of operation for this file is detected fully automatically, depending on whether /etc/resolv.conf is a symlink to /run/systemd/resolve/resolv.conf or lists 127.0.0.53 as DNS server.
Is /etc/resolv.conf read on reload and or restart of the systemd-resolved service (`servicectl restart systemd-named`)?
Some examples of validating DNSSEC in `man delv` would be helpful.
NetworkManager (now with systemd-resolved) is one system for doing DNS configuration for zero or more transient interfaces:
man nmcli
nmcli connection help
nmcli c help
nmcli c h
nmcli c show ssid_or_nm_profile | grep -i dns
nmcli c modify help
man systemd-resolved
man delv
man dnssec-trust-anchors.dit's probably dnssec & your time is probably off.
jorunalctl -xe should be showing errors if so. probably check 'joirnalctl -xeu systemd-resolved'
use 'date' to check the time. if it's off, I use ntpdate to update my clock. but since dns isn't resolving I use 'dig pool.ntp.org @9.9.9.9' to resolve a ntp pool server ip address. then ntpdate [that up].
one of those things that hits me a bunch that I haven't automated quite yet. supposedly newer systemd can detect dnssec being bad in some cases & disable it, after a time, after a bunch of failures, but it either hasn't tripped in a number of cases for me or something else was odd in my envs. manually syncing the clock via ntp usually gets my dns working again.
> manually syncing the clock via ntp usually gets my dns working again.
Why is this necessary?
normally systemd-timedated does a good job keeping things in sync!
but some of my systems seem to have not great real-time clock batteries, and the system will forget the time. i think there might be some other circumstance that sometimes causes my system clock to be way out of whack, but i'm not sure what.
so that's why my clock doesn't work and needs sync.
the dnssec internet protocols are designed to guarantee the user that they have up to date, accurate, trustable records. this depends on your system knowing what time it is now. if your system is way ahead or way behind the actual time, the dnssec records it gets dont appear as valid. And systemd-resolved will reject them, if it is set up to respect DNSSEC.
Systemd-timedated: https://www.freedesktop.org/software/systemd/man/systemd-tim...
timedatectl set-ntp false && \
timedatectl set-ntp true
Ask HN: How bad is proof-of-work blockchain energy consumption?
I'm not a blockchain/crypto expert by any means, but I've been hearing about how much energy the proof-of-work blockchains (Bitcoin, Ethereum, NFTs) consume. Unless I'm mistaken their whole design relies on cranking through more and more CPU cycles. Should we be more concerned about this? Are the concerns overblown? Are there ways to improve it without certain crypto currencies imploding?
A rational market would be choosing an asset that offers value storage and transmission (between points in spacetime) according to criteria: "security" (security theater, infosec, cryptologic competency assessment, software assurances), "future stability" (future switching costs), and "cost".
The externalities of energy production are what must be overcome if we are to be able to withstand wasteful overconsumption of electricity. Eventually, we could all have free clean energy and no lightsabers, right?
So, we do need to minimize wasteful overconsumption. Define wasteful in terms of USD/kWHr (irregardless of industry)? In terms of behavioral economics, why are they behaving that way when there are alternatives that cost <$0.01/tx and a fairly-aggregated comprehensive n kWhr of electricity?
TIL about these guys, who are deciding to somewhat-responsibly self-regulate in the interest of long-term environmental sustainability for all of the land: "Crypto Climate Accord". https://cryptoclimate.org/
"Crypto Climate Accord Launches to Decarbonize Cryptocurrency Industry Brings together the likes of CoinShares, ConsenSys, Ripple, and the UNFCCC Climate Champions to lead sustainability in blockchain and crypto" (2021) https://bit.ly/CryptoClimateAccord
> What are the objectives of the Crypto Climate Accord? The Accord’s overall objective is to decarbonize the global crypto industry. There are three provisional objectives to be finalized in partnership with Accord supporters:
> - Enable all of the world’s blockchains to be powered by 100% renewables by the 2025 UNFCCC COP Conference
> - Develop an open-source accounting standard for measuring emissions from the cryptocurrency industry
> - Achieve net-zero emissions for the entire crypto industry, including all business operations beyond blockchains and retroactive emissions, by 2040
Similar to the Paris Agreement (2015), stakeholders appear to be setting their own targets for sustainability in accordance with the Crypto Climate Accord (2021). https://cryptoclimate.org/accord/
Someone who's not in renewables could launch e.g. a "Satoshi Nakamoto Clean Energy Fund: SNCEF" to receive donations from e.g. hash pools and connect nonprofits with sustainability managed renewables. How many SNCEFs did you give this year and why?
#CleanEnergy
Thankfully, cheap energy fit for Bitcoin mining is increasingly not found in fossil fuels, but in renewables, wasted and “stranded” sources of energy, as we’ll dive into next. [1]
[1]: https://bitcoinmagazine.com/business/bitcoin-will-save-our-e...
[2]: https://www.carbonbrief.org/solar-is-now-cheapest-electricit...
[3]: For example, to illustrate the point, I currently mine BTG on a spare GPU I had because my house needed to be heated anyways. I turn the miner on when I want heat in my house (about 600w) and turn it off when not needed... This is a micro example of what could be done across our entire society "Recent data suggest that space heating accounts for about 42 percent of energy use in U.S. residences and about 36 percent of energy use in U.S. commercial buildings."[4] . Every one of those space heaters could actually be a processor doing computation. In the future the "waster" will not be a miner but someone who turns electricity straight to heat without intermediate chains of value (such as computation or light).
[4]:https://www.epa.gov/rhc/renewable-space-heating#:~:text=Rece....
It's important to note for #3 that a mining GPU is a terribly inefficient space heater.
No, all space heaters are equally efficient. They all have perfect 100% efficiency, because they turn electrical power into heat. When your work product is heat and the waste product is also heat, then there really is no waste.
Technically in the case of cryptocurrency mining, some of the electrical power is turned into information rather than heat. In principle this reduces the amount of heat that you get, but in practice this isn’t even measurable. Most of the information is erased (discarded as useless), which turns it back into heat. Only a few hundred bits of information will be kept after successfully mining a block of transactions, and the amount of heat that costs you is fantastically small. Far smaller than you can measure.
More transistors per unit area, but also more efficient please! There should be demand for more efficient chips (semiconductors,) that are fully-utilized while depreciating on your ma's electricity bill (which is not yet (?) really determined by a market-based economy with intraday speculation to smooth over differences in supply and demand in the US). Oversupply of the electrical grid results in damage costs; which is why the price sometimes falls so low where there are intraday prices and supply has been over-subsidized pending the additional load from developing economies and EVs: Electric Vehicles.
New grid renewables (#CleanEnergy) are now less expensive than existing baseload; which makes renewables long term environment-rational and short term price-rational.
"Thermodynamics of Computation Wiki" (2018) https://news.ycombinator.com/item?id=18146854
> No, all space heaters are equally efficient. They all have perfect 100% efficiency, because they turn electrical power into heat. When your work product is heat and the waste product is also heat, then there really is no waste.
This heat must be distributed throughout the room somehow (i.e. a batteryless woodstove fan or a sterling engine that does work with the difference in entropy when there is a difference in entropy)
> Technically in the case of cryptocurrency mining, some of the electrical power is turned into information rather than heat. In principle this reduces the amount of heat that you get, but in practice this isn’t even measurable. Most of the information is erased (discarded as useless), which turns it back into heat.
See "Thermodynamics of Computation Wiki" re: a possible way to delete known observer-entangled bits while reducing heat/entropy (thus bypassing Landauer's limit for classical computation?)?
> Only a few hundred bits of information will be kept after successfully mining a block of transactions, and the amount of heat that costs you is fantastically small. Far smaller than you can measure.
Each n-symbol sequence in the hash function output does appear to have nearly equal frequency/probability of occurrence. Indeed, is Proof-of-Work worth the heat if you're not reusing the waste heat?
What does a PGP signature on a Git commit prove?
Totally though this was going to be some snarky article claiming they're worthless. Refreshing to see the article just ernestly answers the headline.
> To my knowledge, there are no effective attacks against sha1 as used by git
Perhaps im missing something, but wouldn't a chosen prefix collision be relavent here? I imagine the real reason is that cost to pull it off for sha1 is somewhere in the $10,000-$100,000 range (but getting cheaper every year) which is lots of $$$ to attack something without an obvious attack scenario that can justify it.
So the big problem with Git and SHA1 is that in many cases you are giving full control to an untrusted third party over the sha1 hash of something. For example, if you merge a binary file, it'd be quite easy for them to generate two different versions of the same file, with the same SHA1 digest, and then use the second version to cause problems in the future. You may also be able to modify a text file in the same way without getting noticed in review (I'm not up to speed on how advanced sha1 collision techniques are now).
Similarly, the git commits you merge themselves could have that done - the actual git commit serialization gives you a fair bit of ability to append stuff to it that isn't shown in the UI. That wouldn't affect the signed git commits. But it's still dubious to have the ability to change old history in a checkout.
Anyway, Git is apparently moving towards SHA256 support, so hopefully this problem will be fixed soon: https://lwn.net/Articles/823352/
> it'd be quite easy for them to generate two different versions of the same file
Citation needed. When SHA1 was cracked, it cost $110k worth of cloud computing. And there was some restriction on the two files which matched checksums. IIRC it was like the Birthday Paradox — you don’t pick one and find another sharing the same match, but you generate billions of mutations of similar binaries and statistically two would have the same checksum.
Not exactly easy, fast, cheap, or work with all use cases.
That nonce value could be ±\0 or 5,621,964,321e100; though for well-designed cryptographic hash functions it's far less likely that - at maximum difficulty - a low nonce value will result in a hash collision.
? How are nonces even involved in this?
Searching for the value to prepend or append that causes a hash collision is exactly the same as finding a nonce value at maximum difficulty (not less than the difficulty value, exactly equal to the target hash).
Mutate and check.
The term nonce has a broader meaning than how it is used in bitcoin. (Edit: i reworded this sentence from what i originally had)
That said, no, finding a collision and finding a preimage are very different things, and well the collision attacks on sha1 will involve a lot of guessing and checking, they are not generic birthday or bruteforce attacks but rely on weaknesses in sha-1 to be practical. They also do not make preimage attacks practical.
Brute forcing to find `hash(data_1+nonce) == hash(data_0)` differs very little from ``hash(data_1+nonce) < difficulty_level`. Write each and compare the cost/fitness/survival functions.
If the hash function is reversible - as may be discovered through e.g. mutation and selection - that would help find hashes that are equal and maybe also less than.
Practically, there are "rainbow tables" for very many combinations of primes and stacked transforms: it's not necessary to search the whole space for simple collisions and may not be necessary for preimages; we don't know and it's just a matter of time. "Collision attack" https://en.wikipedia.org/wiki/Collision_attack
Crytographic nonce > hashing: https://en.wikipedia.org/wiki/Cryptographic_nonce#Hashing
> Brute forcing
The attack being discussed is not a brute force attack (or not purely). If the best attack on sha1 was bruteforce than we would still be using it.
> to find `hash(data_1+nonce) == hash(data_0)` differs very little from ``hash(data_1+nonce) < difficulty_level`.
Neither of those are collision attacks (assuming you dont control the data variable). The first is a second pre-image and the second (with equality) would be a normal preimage.
The attack for sha1 under discussion (chosen prefix collision) is finding hash(a+b) == hash(c+d) where you control b and d (but not neccesarily a and c)
> Practically, there are "rainbow tables" for very many combinations of primes and stacked transforms:
What do primes or rainbow tables have to do with any of this? Primes especially. Rainbow tables are at least related to reversing hashes, if totally irrelavent to the subject at hand, but how did you get to primes?
Practically, iff browsers still relied upon SHA-1 to fingerprint and pin and verify certificates instead of the actual chain, and there were no file size limits on x.509 certificates, some fields in a cert (e.g. CommonName and SAN) would be chosen and other fields would then potentially be nonce.
In context to finding a valid cert with a known good hash fingerprint, how many prime keypairs could there be to precompute and cache/memoize when brute forcing.
"SHA-1 > Cryptanalysis and validation " does list chosen prefix collision as one of many weaknesses now identified in SHA-1: https://en.wikipedia.org/wiki/SHA-1#Cryptanalysis_and_valida...
This from 2008 re: the 200 PS3s it took to generate a rogue CA cert with a considered-valid MD5 hash: https://hackaday.com/2008/12/30/25c3-hackers-completely-brea...
... Was just discussing e.g. frankencerts the other day: https://news.ycombinator.com/item?id=26605647
Breakthrough for ‘massless’ energy storage
"Researchers from Chalmers University of Technology have produced a structural battery that performs ten times better than all previous versions"
It's 10 times more everything, just like every other battery breakthrough in the last several decades. AND, you get to use it as a building material?
- Sounds too good to be true: Check
- Sounds like every other "big" breakthrough: Check
- Article light on science and heavy on assertions: Check
- Skepticism engaged: Check
It may really be 10x, but you've mischaracterized what the paper is claiming a 10x breakthrough for. In fairness, the linked article is light on the specifics.
The claim is:
__This battery holds 10x more charge per kilo of material than previous attempts at STRUCTURAL (massless) battery materials__.
To be specific, this material holds only _ONE FIFTH_ the charge of what your smartphone's battery can hold per kilo of battery. No laws of thermodynamics is being broken and this is not at all about battery chemistry. It's all about the 'physics' of construction materials: About how this stuff is made in the factory and layered. The basic battery chemistry going on is not much different from what's been available for years - the interesting part is how this material encases it.
You can't make a car by building the chassis out of smartphone batteries. But the promise of this paper is that you CAN build the car chassis out of this battery, and even if this battery is only 20% as effective, a car chassis is rather large, and you needed it anyway, so every drop of power you can store in the chassis itself was effectively 'free' - hence the somewhat hyperbolous 'massless' terminology.
> You can't make a car by building the chassis out of smartphone batteries
They're called Structural batteries (or [micro]structural super/ultracapacitors)
"Carmakers want to ditch battery packs, use auto bodies for energy storage" (2020,) https://arstechnica.com/cars/2020/11/carmakers-want-to-ditch...
This is literally what TFA is about
OpenSSL Security Advisory
If the cloud companies just paid a team of five people 200K each to spend a year rewriting OpenSSL from scratch, they would save multiple millions in scrambling to deploy bug fixes.
Nope, they'd just create new software with different bugs that have not been discovered yet. Then we'd all be scrambling to fix those bugs.
Not if the implementation is formally verified, like miTLS [0] and EverCrypt [1]. Parts of the latter were integrated into Firefox, which even provided a performance boost (10x in one case) [2].
I think what is needed is something like EverCrypt but for TLS. Or in other words, something like miTLS but which extracts to C and/or Assembly, to avoid garbage collection and for easy interoperation with different programming languages (preferably including an OpenSSL-compatible API for backwards compatibility).
[0]: https://mitls.org/ [1]: https://hacl-star.github.io/HaclValeEverCrypt.html [2]: https://blog.mozilla.org/security/2020/07/06/performance-imp...
Formally verified against what? and what assumption were made?
miTLS is formally verified against the TLS spec on handshaking, assuming the lower-level crypto routine are good. It is not even free from timing attack.
EverCrypt have stronger proof, but it is only safe as in not crashing and correct as in match the spec in all valid input. It is not proved to be free from DoS or invalid input attack.
OpenSSL do more then TLS. Lots of interesting thing are in the cert format parsing and management.
> Formally verified against what? and what assumption were made?
Well, tell me again, what is OpenSSL formally verified against? What assumptions were made in OpenSSL?
Formal verification does not eliminate very large classes of bugs only when absolutely everything is formally verified, including timing attacks. Instead, it consistently produces more reliable software, many times even when only certain basic properties are proved (such as, memory safety, lack of integer overflows or division by zero, etc).
Formal verification can be a continuum, like testing, but proven to work for all inputs (that possibly meet certain conditions, under certain assumptions). The assumptions and properties that are proven can always be strengthened later, as seen in multiple real-world projects (such as seL4 and others).
The result is code that is almost always a lot more bug-free than code that is not formally verified. And as I said, more properties can be proven over time, especially as new classes of attacks are discovered (e.g. timing attacks, speculation attacks in CPUs, etc).
To formally verify an implementation you need a... formal description of what is to be implemented and verified.
Constructing a formal specification from RFC8446 is possible.
Constructing a formal specification for PKIX... is not. PKIX is specified by a large number of RFCs and ITU-T/ISO specs, some of which are more formal than others. E.g., constructing a formal specification for ASN.1 should be possible (though a lot of work), while constructing a formal specification for certificate validation is really hard, especially if you must support complex PKIs like DoD's. Checking CRLs and OCSP, among other things, requires support for HTTP, and even LDAP, so now you have... a bunch more RFCs to construct formal descriptions of.
And there had better be no bugs in the formal descriptions you construct from all these specs! Recall, they're mostly not formal specs at all -- they're written in English, with smatterings of ASN.1 (which is formal, though the specs for it, though they're very very good, mostly aren't formal).
The CVE in question is in the PKIX part of OpenSSL, not the TLS implementation.
What you're asking for is not 5 man-years worth of work, but tens of man-decades. The number of people with deep knowledge of all this stuff is minute as it is -- maybe just a handful, tens at most. The number of people with deep knowledge of a lot of this stuff is larger, but still minute. So you're asking to spend decades' worth of a tiny band of people's time on this project, when there are other valuable things for them to do.
The number of people who can do dev work in this space is much larger, of course -- in the thousands. But very few of them have the right expertise to work on a formal, verified implementation of PKIX.
Plus, it's all a moving target.
Sure, we could... train a lot of people just for such a project, but it takes time to do that, and it still takes time from that tiny band of people who know this stuff really well.
I'm afraid you're asking for unobtanium.
EDIT: Plus, there's probably tens of millions of current dollars' (if not more) worth of development embodied in OpenSSL as it stands. It's probably at least that much to replace it with a verified implementation, and probably much more because the value of programmers expert enough to do it is much more than the $200K/year suggested above (even if you train new ones, it would take years of training, and then they would be just as valuable). I think a proper , formally verified replacement of OpenSSL would probably run into the hundreds of millions, especially if it's one huge project since those tend to fail.
Well, sure, if your start with those premises, then I'm not surprised that you reach the conclusion that the goal is unachievable.
First of all, if constructing a formal specification for PKIX is not possible, then that should be telling you that it either needs to be simplified, better specified or scrapped altogether for something better (the latter would require an extremely large transition period, I'm imagining, so the first two are much preferred in this situation).
Otherwise, how can you be sure that any implementation in fact implements it correctly?
> And there had better be no bugs in the formal descriptions you construct from all these specs!
Well, I don't think that is true. You should in fact allow bugs in the formal description, otherwise how will you ever get anything done in such a project?
You see, having a formal description with a bug is much better than having no formal description at all.
If you have no formal description, you can't really tell if your code has bugs. If you have a buggy formal description, then you are able to catch some bugs in the implementation and the implementation can also catch some bugs in the formal description.
Also, some parts of the formal description can catch bugs in other parts of the formal description.
So the end result can be strictly better than the status quo.
> Plus, it's all a moving target.
Sure, but hopefully it's a moving target moving in the direction of simplification rather than getting more complex, otherwise things will just get worse rather than get better, regardless if we keep with the status quo or not. I'm not a TLS expert by any means but I think TLS 1.3 moved in that direction for some parts of the protocol, at least (if I'm not mistaken).
Also, I think you are not fully appreciating that formal verification can be done incrementally.
You can start by doing the minimum possible, i.e. simply verifying that your code is free of runtime errors, which would eliminate all memory safety-related bugs, including Heartbleed.
This would already be better than reimplementing in Rust, because the latter can protect from memory safety bugs but not other runtime bugs (such as division by zero, unexpected panics, etc).
BTW, this minimal verification effort would already eliminate the second bug in this security advisory.
You can then verify other simple properties, even function by function, no complicated models are even necessary at first.
For example, you could verify that your function that verifies a CA certificate, when passed the STRICT flag, is really more strict than when not passed the STRICT flag.
This would eliminate the first bug in this security advisory and all other similar bugs in the same function call-chain.
BTW, what I just said is really easy to specify, and I'm guessing it's also very easy to prove, since I'm guessing that the strict checks are just additional checks, while the normal checks are shared between the two verification modes.
Many other such properties are also easy to prove. The more difficult properties/models, or even full functional verification, can be implemented by more expert developers or even mathematicians.
I think the problem is also that OpenSSL devs, like the vast majority of devs, probably have no desire/intention/motivation/ability to do formal verification, otherwise you could even do this in the OpenSSL code base itself (although that is not ideal because verifying a program written in C requires more manual work than verifying it in a simpler language whose code extracts to C).
I'm also guessing that your budget estimate is an exaggeration since miTLS and EverCrypt, although admittedly projects whose scope has still not reached your ambitious goals (i.e. full functional verification of all layers in the stack), was probably done with a much smaller budget.
And it's not like you can't build on top of that and incrementally verify more properties over time, e.g. more layers of the stack or whatever.
You don't need a huge mega-project, just a starting point, and sufficient motivation.
> Well, sure, if your start with those premises, then I'm not surprised that you reach the conclusion that the goal is unachievable.
I didn't say unachievable. I said costly.
> First of all, if constructing a formal specification for PKIX is not possible, then that should be telling you that it either needs to be simplified, better specified or scrapped altogether for something better (the latter would require an extremely large transition period, I'm imagining, so the first two are much preferred in this situation).
The specs for the new thing would basically have to be written in Coq or similar. Even if you try real hard to keep it small and not make the... many mistakes made in PKIX's history... it would still be huge. And it would be even less accessible than PKIX already is.
> For example, you could verify that your function that verifies a CA certificate, when passed the STRICT flag, is really more strict than when not passed the STRICT flag.
That's just an argument for better testing. That's not implementation verification.
> This would eliminate the first bug in this security advisory and all other similar bugs in the same function call-chain.
Only if you thought to write that test to begin with. Writing a test for everything is... not really possible. SQLite3, one of the most tested codebases in the world, has a private testsuite that gets 100% branch coverage, and even that is not the same as testing every possible combination of branches.
> BTW, what I just said is really easy to specify, and I'm guessing it's also very easy to prove, since I'm guessing that the strict checks are just additional checks, while the normal checks are shared between the two verification modes.
It's not. The reason the strictness flag was added was that OpenSSL was historically less strict than the spec demanded. It turns out that when you're dealing with more than 30 years of history, you get kinks in the works. It wouldn't be different for whatever thing replaces PKIX.
> I think the problem is also that OpenSSL devs, like the vast majority of devs, probably have no desire/intention/motivation/ability to do formal verification, otherwise you could even do this in the OpenSSL code base itself [...]
You must be very popular at parties.
> I'm also guessing that your budget estimate is an exaggeration since miTLS and EverCrypt, [...]
Looking at miTLS, it only claims to be an implementation of TLS, not PKIX. Not surprising. EverCrypt is a cryptography library, not a PKIX library.
> That's just an argument for better testing. That's not implementation verification.
No, I'm not talking about testing, I'm talking about really basic formal verification:
forall (x: Certificate), verify_certificate(x, flags = 0) == invalid ==> verify_certificate(x, flags = STRICT) == invalid
This is trivial to specify and almost as trivial to prove to be correct for all inputs. Testing can't do that, no matter how good your testing.
> Only if you thought to write that test to begin with. Writing a test for everything is... not really possible.
I agree, but I'm not talking about testing. I'm talking about formal verification.
> It's not. The reason the strictness flag was added was that OpenSSL was historically less strict than the spec demanded. It turns out that when you're dealing with more than 30 years of history, you get kinks in the works. It wouldn't be different for whatever thing replaces PKIX.
That's totally fine and wouldn't affect the verification of that function at all. This is a very simple property to verify and it would avoid this bug and all similar bugs in that function (even if that function calls other functions, no matter how large or complex they are).
Other functions also have such easy to verify properties, this is not hard at all to come up with (although sure, some more difficult properties might be harder to verify).
> You must be very popular at parties.
I didn't want to be dismissive of OpenSSL devs or other devs in general, I just find it frustrating that there are so many myths surrounding this topic, and a lot less education, interest and investment than I think there should be, nowadays.
You are confusing verification as in "certificate" with verification as in "theorem proving", and you are still assuming a formal description of what to verify (which I'll remind you: doesn't exist). And then you go on to talk about myths and uneducated and uninterested devs. Your approach is like tilting at windmills, and will achieve exactly as much.
If I understood you correctly, then I am not confusing those two things.
Maybe you haven't noticed but I actually wrote a theorem about a hypothetical verify_certificate() function in my previous comment. Maybe you also haven't noticed, but I didn't need a formal description of how certificate validation needs to be done to write that theorem.
And I assure you, if the implementation of verify_certificate() is anything except absolute garbage, it would be very easy to prove the theorem to be correct. I actually have some experience doing this, you know? I'm not just parroting something I read, I've proved code to be 100% correct multiple times using formal verification (i.e. theorem proving) tools. That is, according to certain basic and reasonable assumptions, of course, like e.g. the hardware is not faulty or buggy while running the code, and that the compiler itself is not buggy, which would be a lot more rare than a bug in my code -- and even then, note that compilers and CPUs can be (and have been) formally verified.
Maybe you don't agree with my approach but I think it's the most realistic one for a project such as this and I am absolutely confident it would be practical, would completely eliminate most (i.e. more than 50%, at least) existing bugs with minimal verification effort (which almost anyone would be capable of doing with minimal training), and would steadily become more and more bug-free with additional, incremental verification (i.e. theorem proving) and refactoring effort.
https://project-everest.github.io/ :
> Focusing on the HTTPS ecosystem, including components such as the TLS protocol and its underlying cryptographic algorithms, Project Everest began in 2016 aiming to build and deploy formally verified implementations of several of these components in the F* proof assistant.
> […] Code from HACL*, ValeCrypt and EverCrypt is deployed in several production systems, including Mozilla Firefox, Azure Confidential Consortium Framework, the Wireguard VPN, the upcoming Zinc crypto library for the Linux kernel, the MirageOS unikernel, the ElectionGuard electronic voting SDK, and in the Tezos and Concordium blockchains.
S2n is Amazon's formally verified TLS library. https://en.wikipedia.org/wiki/S2n
IDK about a formally proven PKIX. https://www.google.com/search?q=formally+verified+pkix lists a few things.
A formally verified stack for Certificate Transparency would be a good way to secure key distribution (and revocation); where we currently depend upon a TLS library (typically OpenSSL), GPG + HKP (HTTP Key Protocol).
Fuzzing on an actual hardware - with stochastic things that persist bits between points in spacetime - is a different thing.
Funny, the first hit for that search you linked is... my comment above. The "few things" other than that are for alternatives to PKIX, which is fine and good, but PKIX will be with us for a long time yet. As for Everest, it jives with what I wrote above, that verified implementations of TLS are feasible (Everest also implements QUIC and similar), but -surprise!- not listed is PKIX.
I know, it sounds crazy, really crazy, but PKIX is much bigger than TLS. It's big. It's just big.
The crypto, you can verify. The session and presentation layers, you can verify. Heck, maybe you can verify your app. PKIX implementations of course can be verified in principle, but in fact it would require a serious amount of resources -- it would be really expensive. I hope someone does it, to be sure.
I suppose the first step would be to come up with a small profile of PKIX that's just enough for the WebPKI. Though don't be fooled, that's not really enough because people do use "mTLS" and they do use PKINIT, and they do use IPsec (mostly just for remote access) with user certificates, and DoD has special needs and they're not the only ones. But a small profile would be a start -- a formal specification for that is within the realm of the achievable in reasonably short order, though still, it's not small.
Both a gap and an opportunity; someone like an agency or a FAANG with a budget for something like this might do well to - invest in the formal methods talent pipeline and - very technically interface with e.g. Everest about PKIX as a core component in need of formal methods.
"The SSL landscape: a thorough analysis of the X.509 PKI using active and passive measurements" (2011) ... "Analysis of the HTTPS certificate ecosystem" (2013) https://scholar.google.com/scholar?oi=bibs&hl=en&cites=16545...
TIL about "Frankencerts": Using Frankencerts for Automated Adversarial Testing of Certificate Validation in SSL/TLS Implementations (2014) https://scholar.google.com/scholar?cites=3525044230307445257... :
> Our first ingredient is "frankencerts," synthetic certificates that are randomly mutated from parts of real certificates and thus include unusual combinations of extensions and constraints. Our second ingredient is differential testing: if one SSL/TLS implementation accepts a certificate while another rejects the same certificate, we use the discrepancy as an oracle for finding flaws in individual implementations.
> Differential testing with frankencerts uncovered 208 discrepancies between popular SSL/TLS implementations such as OpenSSL, NSS, CyaSSL, GnuTLS, PolarSSL, MatrixSSL, etc.
W3C ld-signatures / Linked Data Proofs, and MerkleProof2017: https://w3c-ccg.github.io/lds-merkleproof2017/
"Linked Data Cryptographic Suite Registry" https://w3c-ccg.github.io/ld-cryptosuite-registry/
ld-proofs: https://w3c-ccg.github.io/ld-proofs/
W3C DID: Decentralized Identifiers don't solve for all of PKIX (x.509)?
"W3C DID x.509" https://www.google.com/search?q=w3c+did+x509
Thanks for the link about frankencerts!
How much total throughput can your wi-fi router really provide?
Out of curiosity does some company benchmark other types of commercial networking devices like this? Things like load balancers, DDoS protection, switches, etc.? How do businesses shop for such products without having a standard measure from an independent third party?
netperf and iperf are utilities for measuring network throughput: https://en.wikipedia.org/wiki/Iperf
It's possible to approximate the https://dslreports.com/speedtest using the flent CLI or QT GUI (which calls e.g. fping and netperf) and isolate out ISP variance by running a netperf server on a decent router and/or a workstation with a sufficient NIC (at least 1Gbps). https://flent.org/tests.html
`dslreports_8dn`: https://github.com/tohojo/flent/blob/master/flent/tests/dslr...
From https://flent.org/ :
> RRUL: Create the standard graphic image used by the Bufferbloat project to show the down/upload speeds plus latency in three separate charts:
> `flent rrul -p all_scaled -l 60 -H address-of-netserver -t text-to-be-included-in-plot -o filename.png`
In 2021, most routers - even with OpenWRT and hardware-offloading - cannot actually push 1 Gigabit over wired Ethernet, though the port spec does say 1000 Mbps.
The Most Important Scarce Resource Is Legitimacy
It seems that nearly every "public good" in the Ethereum ecosystem got gobbled up by VCs and pivoted to for-profit endeavors. I don't mind VCs and I don't mind people seeking profit, but this moral high ground that Ethereum has been trying to build for itself is delusional. In practice it is just a marketing gimmick.
An astute observer may have noticed that Vitalik has a habit of coming up with definitions that only himself and the Ethereum foundation can meet the standards of.
The reality is everyone is just trying to make money. I think we are better off coming to terms with that instead of living in a collective fantasy.
Public goods are by definition non-rivalrous and non-excludable. How can they be gobbled?
https://en.wikipedia.org/wiki/Public_good_(economics)
(I'm not being pedantic here. The concept of a public good, in this technical sense, is a very important concept to have -- and it's specifically the kind of "public goods" the article was talking about, along with ideas for improving their tendency to be under-supplied. Goods that can be gobbled up by someone tend to be produced at closer to the optimum amount, since there's a less diffuse private incentive for their production.)
Public goods ... Welfare economics ... Social choice theory, Arrow's, Indifference curve: https://en.wikipedia.org/wiki/Indifference_curve
People do collectibles; commemorative plates.
A few notes on message passing
[deleted]
The big problem nobody talks about with actors and message passing is non-determinism and state.
If two otherwise independent processes A and B are sending messages to a single process C, it's non-deterministic which arrives at C first. This can create a race condition. C may respond differently depending on which messages arrives first - A's or B's. This is also effectively an example of shared mutable state - the state of C, which can be mutated whenever a message is received - is shared with A and B because they're communicating with it and their messages work differently based on the current state.
Non-determinism, race-conditions, shared mutable state: the absolute opposite of what we want when dealing with concurrency and parallelism.
> Luckily, global orders are rarely needed and are easy to impose yourself (outside distributed cases): just let all involved parties synchronize with a common process.
When there are multiple agents/actors in a distributed system, and the timestamp resolution is datetime64, and clock synchronization and network latency are variable, and non-centralized resilience is necessary to eliminate single points of failure, global ordering is impractical to impossible because there is no natural unique key with which to impose a [partial] preorder [1][2]: there are key collisions when you try and merge the streams.
Just don't cross the streams.
[1] https://en.wikipedia.org/wiki/Preorder_(disambiguation)
[2] https://en.wikipedia.org/wiki/Partially_ordered_set
The C in CAP theorem is for Consistency [3][4]. Sequential consistency is elusive because something probably has to block/lock somewhere unless you've optimally distributed the components of the CFG control flow graph.
[3] https://en.wikipedia.org/wiki/Consistency_model
[4] https://en.wikipedia.org/wiki/CAP_theorem
FWIU, TLA+ can help find such issues. [5]
[5] https://en.wikipedia.org/wiki/TLA%2B
Wouldn't a partial order be possible using logical clocks? Vector clocks probably don't satisfy "practical" but a hybrid logical clock is practical if you can assume clock synchronisation is within some modest delta.
> if you can assume clock synchronisation is within some modest delta.
From experience with clocks, you should really be actively testing the delta, and not just assuming it.
The assumption would be the limit you actively test for.
The Lamport timestamp: https://en.wikipedia.org/wiki/Lamport_timestamp :
> The Lamport timestamp algorithm is a simple logical clock algorithm used to determine the order of events in a distributed computer system. As different nodes or processes will typically not be perfectly synchronized, this algorithm is used to provide a partial ordering of events with minimal overhead, and conceptually provide a starting point for the more advanced vector clock method.
Duolingo's language notes all on one page
Succinct. What a useful reference.
An IPA (International Phonetic Alphabet) reference would be helpful, too. After taking linguistics in college, I found these Sozo videos of US english IPA consonants and vowels that simultaneously show {the ipa symbol, example words, someone visually and auditorily producing the phoneme from 2 angles, and the spectrogram of the waveform} but a few or a configurable number of [spaced] repetitions would be helpful: https://youtu.be/Sw36F_UcIn8
IDK how cartoonish or 3d of an "articulatory phonetic" model would reach the widest audience. https://en.wikipedia.org/wiki/Articulatory_phonetics
IPA chart: https://en.wikipedia.org/wiki/International_Phonetic_Alphabe...
IPA chart with audio: https://en.wikipedia.org/wiki/IPA_vowel_chart_with_audio
All of the IPA consonant chart played as a video: "International Phonetic Alphabet Consonant sounds (Pulmonic)- From Wikipedia.org" https://youtu.be/yFAITaBr6Tw
I'll have to find the link of the site where they playback youtube videos with multiple languages' subtitles highlighted side-by-side along with the video.
Found it: https://www.captionpop.com/
It looks like there are a few browser extensions for displaying multiple subtitles as well; e.g. "YouTube Dual Subtitles", "Two Captions for YouTube and Netflix"
Ask HN: The easiest programming language for teaching programming to young kids?
Hi,
I want to start a small community pilot project to help young kids, 8 and above, get interested in programming. We will use video games and robotics projects. We want to keep our tech stack as simple as possible. Here are some of the choices:
Godot + Aurdino: We can use C in Godot and Aurdino. Aurdino might be more interesting for kids as opposed neatly packaged Lego Kits.
Apple SpriteKit + Lego Mindstorm: We can use Swift with Legos. But cost will be higher.
Some of the projects we are thinking are:
Game-ish:
1. Sound visualizer like how Winamp and old school visualization were. Use speakers. And various other ideas around these concepts.
2. AR project that shows the world around you in cartoonish style. Swap faces etc.
3. Of cousre, platform games.
Robotics projects:
I see a lot of tutorials for Arduino such as robots that follow sound or light, or stuff like lights display. We will use mostly those.
Some harder project ideas I have are for drones, boats, and other navigational vehicles. This is why I want to use Arduino. But is C going to be too hard for young kids to play with?
What do you recommend? If this works, I would like to expand it and start a company around it.
awesome-python-in-education > "Python suitability for education" lists a few justifications for Python: https://github.com/quobit/awesome-python-in-education#python...
There is a Scratch Jr for Android and iOS. You can view Scratch code as JS. JS does run in a browser, until it needs WASI.
awesome-robotics-libraries: https://github.com/jslee02/awesome-robotics-libraries
FWIU, ROS (Robot Operating System) is now installable with Conda/Mamba. There's a jupyter-ros and a jupyterlab-ros extension: https://github.com/RoboStack/jupyter-ros
I just found this: https://coderdojotc.readthedocs.io/projects/python-minecraft...
> This documentation supports the CoderDojo Twin Cities’ Build worlds in Minecraft with Python code group. This group intends to teach you how to use Python, a general purpose programming language, to mod the popular game called Minecraft. It is targeted at students aged 10 to 17 who have some programming experience in another language. For example, in Scratch.
K12CS Framework has your high-level CS curriculum: https://k12cs.org/ [PDF]: https://k12cs.org/wp-content/uploads/2016/09/K%E2%80%9312-Co...
Educational technology > See also links to e.g. "Evidence-based education" and "Instructional theory" https://en.wikipedia.org/wiki/Educational_technology https://en.wikipedia.org/wiki/Educational_technology
Thank you for these resources, reviewing them now.
Yw. Np. So I just searched for "site: readthedocs.io kids python" https://www.google.com/search?q=site%3Areadthedocs.io+kids+p... and found a few new and old things:
SensorCraft (pyglet (Python + OpenGL)) from US AFRL Sensors Directorate has e.g. Gravity, Rocket Launch, and AI tutorials:
> Most people are familiar with Minecraft [...] for this project we are using a Minecraft type environment created in the Python programming language. The Air Force Research Laboratory (AFRL) Sensors Directorate located in Dayton, Ohio created this guide to inspire kids of all ages to learn to program and at the same time get an idea of what it is like to be a Scientist or Engineer for the Air Force. We created this YouTube video about SensorCraft
https://sensorcraft.readthedocs.io/en/latest/intro.html
`conda install -c conda-forge -y pyglet` should probably work. Miniforge on Win/Mac/Lin is an easy way to get Python installed on anything including ARM64 for a RPi or similar; `conda create -n scraft; conda install -c conda-forge -y python=3.8 jupyterlab jupytext jupyter-book pyglet` . If you're in a conda env, `pip install` should install things within that conda env. Here's the meta.yaml in the conda-forge pyglet-feedstock: https://github.com/conda-forge/pyglet-feedstock/blob/master/...
"BBC micro:bit MicroPython documentation" https://microbit-micropython.readthedocs.io/en/latest/
$25 for a single board-computer with a battery pack and a case (and curricula) is very reasonable: https://en.wikipedia.org/wiki/Micro_Bit
> The [micro:bit] is described as half the size of a credit card[10] and has an ARM Cortex-M0 processor, accelerometer and magnetometer sensors, Bluetooth and USB connectivity, a display consisting of 25 LEDs, two programmable buttons, and can be powered by either USB or an external battery pack.[2] The device inputs and outputs are through five ring connectors that form part of a larger 25-pin edge connector. (V2 adds a Mic and a Speaker)
Raspberry Pi for Kill Mosquitoes by Laser
This work (I think) worked with the mosquitos 30cm away with a servo scanning Pi Camera (1080p) and a 1W laser.
To work in the real world - cover a whole room or terrace - presumably a much higher resolution camera (or much faster scanning system) would be required. Even a 1W laser is dangerous to eyesight, if it was being fired at targets mingling with people.
The system could be mounted on small drones that would patrol larger areas - but the idea of robotic drones armed with lasers roaming around is beginning to sound worse than the mosquitos.
Good luck detecting a mosquito optically from a distance of several meters using a cheap camera and Raspberry Pi. Oh and you want to do from a moving drone. That will certainly make it work!
Just look at the images in the article - the guy's best result was detecting a black speck appearing on a nearby white wall with some 60-70% reliability (based on his own numbers). So you would be missing a lot of mosquitoes - but will be happy firing the laser at random shadows and what not. And that was in a completely stationary setup and controlled lab conditions, i.e. not at all something resembling a typical poorly lit room!
This article is BS. Preprints are not peer reviewed (i.e. nobody has checked anything in it - so could even be a complete hoax), it is a pretty typical gadgetry style paper (we do it because we can, not because it makes sense) you do at when you need to fill up your resume with research papers (e.g. for keeping/obtaining a job reasons).
The "save the world" (mosquito control, diseases, etc.) justification is also par for the course for this type of crappy paper. Anyone who seriously thinks that one could control mosquito problem by shooting them one by one by a laser is delusional.
But neural networks and "AI" are being used, so it has to be cutting edge groundbreaking stuff, right?
BTW, this nonsense idea has been floated as a publicity stunt a few years ago (including a slow motion video of a laser burning off wing of a mosquito in flight) and it seems that some Russian PhD student from a fairly obscure uni either didn't do their research or has reinvented the wheel (or just plain copied the thing without attribution). The list of irrelevant or only very tangentially relevant (it is about mosquitoes, so in scope, right?) references is a dead giveaway there (paper on mosquitoes spreading zika? seriously?).
Here, it was even on National Geographic in 2010(!): https://www.youtube.com/watch?v=BKm8FolQ7jw
Oh and that was supposed to be a handheld device to boot. With the same "save the world from malaria" spiel too. I wonder what are the owners of the company that was pushing this concept to investors back then trying to sell today ...
There are actually multiple videos on Youtube showing products from different companies that were attempting to push this as some sort of viable concept.
> using a cheap camera
even if you had a RED Komodo feeding uncompressed 4K DCI 60fps video to a pci-express bus capture card, the sensor resolution and tiny size of mosquitoes means that unless the lighting conditions are just right, and the mosquito is somehow highlighted against a background, it's going to be very hard to pick them out at distances of 2 or more meters.
and that's before you get into the software problem of processing the fire hose of data that is 4096x2160 at 60fps raw. and the hardware cost of a very serious workstation class PC capable of taking the capture at 1:1 realtime.
possibly a lidar based sensor or something might be more suitable to locating the x/y/z position of mosquitoes in a few meter area.
Mosquitos, as we all know, have a highly distinctive auditory signature.
A phased microphone array is the only sensible approach to localized mosquito detection. It would probably work reasonably well.
The problem is that the entire field is patent encumbered, because Myhrvold's company Intellectual Ventures has done some research on it, and no on in their right mind would go up against those guys.
Yeah, they already did sharks with lasers. IDK what the licensing terms are on that
ONE MILLION DOLLARS
Donate Unrestricted
The article mentioned the Gates foundation which has terrible records in education initiatives:
- The big failure in small schools initiative: https://marginalrevolution.com/marginalrevolution/2010/09/th...
- The big failure in teacher evaluation initiative: https://www.businessinsider.com/bill-melinda-gates-foundatio...
- The big failure in Common Core initiative: https://www.washingtonpost.com/news/answer-sheet/wp/2016/06/...
Now it is funding anti-racist math: https://equitablemath.org/
Unbelievable.
Rather than diminishing the efforts of others, you could start helping by describing your own efforts to improve education (in order to qualify your ability to assess the mentioned and other efforts to improve education and learning)
In context to seed and series funding for a seat on a board of a for-profit venture, an NGO non-profit organization can choose whether to accept restricted donations and government organizations have elected public servant leaders who lead and find funding.
Works based on Faust: https://en.wikipedia.org/wiki/Works_based_on_Faust
Bitcoin Is Time
I'm ready to be downvoted to oblivion, especially by all the Bitcoin purists, but I hope my message sparks at least some curiosity.
Bitcoin is getting "weaker" every year for several reasons:
- emerging centralization due to economies of scale. If this trend continues the mining power will be so consolidated that a 51% attack will be likely. The Nakamoto coefficient is already at 4, and tending towards 3.
- energy usage due to Proof of Work is growing astronomically, and the higher the price of bitcoin, the less incentive there is to use renewable energy. Bitcoin uses more energy than the country of Argentina.
- transaction times are SLOW, and expensive. The lightning network is incredibly buggy and won't actually solve the problems it's promising.
There is a better alternative. RaiBlocks, named nano since 2018. It got a bad rap due to the BitGrail hack, but it's picking up steam again. The developer community is great, the main dev team on the Nano protocol has been consistently chugging along regardless of the 3 years of crypto winter on a shoestring budget and the nano community is made up of users, not speculators.
This article sums it up quite nicely: https://senatusspqr.medium.com/why-nano-is-the-ultimate-stor...
I'm happy to receive downvotes, but all I ask in return is that you give it a read with an open mind.
> The lightning network is incredibly buggy and won't actually solve the problems it's promising.
Not an expert but I found that a readable summary of different layer 2 scaling strategies and why Ethereum developers prefer Rollups instead of Channels (like Lightning):
"Bitcoin scalability problem" could link to the Ethereum design docs: https://en.wikipedia.org/wiki/Bitcoin_scalability_problem
The Ethereum design docs could link to direct-listed premined [stable] coins as a solution for Proof of Work and TPS reports: https://github.com/flare-eng/coston#smart-contracts-with-xrp
(edit) re: n-layer solutions: The https://interledger.org/ RFCs and something like Transaction Permission Layer (TPL) will probably be helpful for interchain compliance.
> Interledger is not tied to a single company, blockchain, or currency.
From https://tplprotocol.org/ :
> The challenge: Current blockchain-based protocols lack an effective governance mechanism that ensures token transfers comply with requirements set by the project that issued the token.
> Projects need to set requirements for a variety of reasons. For instance, remaining compliant with securities laws, limiting transfer to beta testers, or limiting transfer to a particular geo-spatial location. Whatever your reason, if a requirement can be verified by a third-party, TPL will be able to help.
In the US, S-Corps can't have international or more than n shareholders, for example; so if firms even wanted to issue securities on a first-layer network, they'd need an extra-chain compliance mechanism to ensure that their issuance is legal pursuant to local, sovereign, necessary policies. Re-issuing stock certificates is something that has to be done sometimes. When is it possible to cancel outstanding tokens?
Foundational Distributed Systems Papers
From "Ask HN: Learning about distributed systems?" https://news.ycombinator.com/item?id=23932271 :
> Papers-we-love > Distributed Systems: https://github.com/papers-we-love/papers-we-love/tree/master...
> awesome-distributed-systems also has many links to theory: https://github.com/theanalyst/awesome-distributed-systems
And links to more lists of distributed systems papers under "Meta Lists": https://github.com/theanalyst/awesome-distributed-systems#me...
In reviewing this awesome list, today I learned about this playlist: "MIT 6.824 Distributed Systems (Spring 2020)" https://youtube.com/playlist?list=PLrw6a1wE39_tb2fErI4-WkMbs...
> awesome-bigdata lists a number of tools: https://github.com/onurakpolat/awesome-bigdata
It makes my flesh crawl to see college reduced purely to ROI, but at least they're honest that that's what they're doing.
I just hope people at least consider all of the other "outputs" of going to college when reading something like this. The ROI analysis is good data, it answers an important question. But there are many, many other important questions worth answering. (And most of them aren't quantifiable, so it's not like you could do a study on them if you wanted to.)
College can be an awful experience for some people even if they end up making good money. And it can be an excellent life experience for people who end up making dirt. I got lucky -- I had a great experience, I learned a lot of important non-academic things, I broke out of my shell, and it's pretty directly responsible for a large portion of my earnings. (And yes, the numbers for my university and major from the study match my current income and age quite well.)
But now I have a kid in high school, and I'm facing all the questions about what futures to keep open and what ones to sacrifice in the service of others. College is much more expensive now. My family was dirt poor and so I had massive financial aid; any financial aid my kid gets will have to be merit-based. Degree inflation has sucked away a lot of the value of having a degree. I can afford to support a less secure (low-risk) path through life if I think it would be better for my kid as a human being. These are not easy questions to be facing.
Increasingly I see the idea of thinking of college in any other terms as a deliberate meme designed to make people pay extremely high prices and put them in a debt trap, and that the sources of that meme like it that way.
Before college can be creating well-rounded citizenry or provide "excellent life experiences" or any of the other things people want of it, it must first provide a good ROI. This is a necessary foundation. If it is not doing that, then all the other fancy things are merely digging the debt trap in deeper because you're paying for all these things while failing to obtain a method of paying them back.
As a culture, in decades past we were used to college generally being so cheap that providing a good ROI wasn't that big a deal. It's much easier to get a good ROI out of something cheap. So we focused on the higher layers and were able to neglect the fact that the higher layers were always built on a foundation of the fact that college in general was a good ROI. Consequently we have come to misinterpret those higher level things as the purpose of college.
But it is necessary that these aspirational benefits be setting on a good foundation of good ROI to not be abusive to the customer.
It is not and must not be considered some sort of betrayal to be worried about ROI for college. It must simply be seen as an understanding of the fact that as the costs have changed, the way we must analyze college has also changed. It was always true, it's just now it has manifested in a larger way.
College wouldn’t need to provide a good roi if it were much mess expensive. Just like, strictly speaking, going to a movie doesn’t provide a good roi.
It’s one thing to take four years of your life deepening your thinking ability and stretching your mind without a good roi if you come out the other end without crushing debt and prospects for employment that can sustain you, even if those prospects have nothing to do with your college experience. This is what college was like until the last four decades or so.
It’s entirely another to come out the other side in crippling debt and no prospects for any kind of sustaining job.
If it provides the same value, but costs less, then by definition, the ROI is better. OTOH, if it puts you in crippling debt, but you have no prospects of a job to support that - the ROI becomes bad or even negative. So doesn't it capture all of that already?
In some sense, twice the cost for twice the benefit might be worth it to those who can afford it, because the time-wise investment (4 years or so of your life) remain constant.
No, it doesn't capture all of that already.
A "better" ROI doesn't necessarily mean a positive ROI.
Getting negative 10% return on a $1,000 investment doesn't matter as much as getting a negative 10% return on a $100,000 investment.
Magnitude certainly is relevant to vector comparisons; but, if we define ROI as nominal rate of return, gross returns are not relevant to a comparison by that metric.
Return on Investment: https://en.wikipedia.org/wiki/Return_on_investment
From https://en.wikipedia.org/wiki/Vector_(mathematics_and_physic... :
> A Euclidean vector is thus an equivalence class of directed segments with the same magnitude (e.g., the length of the line segment (A, B)) and same direction (e.g., the direction from A to B).[3] In physics, Euclidean vectors are used to represent physical quantities that have both magnitude and direction, but are not located at a specific place, in contrast to scalars, which have no direction.[4] For example, velocity, forces and acceleration are represented by vectors
Quantitatively and Qualitatively quantify the direct and external benefits of {college, other alternatives} with criteria in additional to real monetary ROI?
From https://en.wikipedia.org/wiki/Welfare_economics
> Welfare economics also provides the theoretical foundations for particular instruments of public economics, including cost–benefit analysis,
From https://news.ycombinator.com/item?id=18833730 :
>> Why would people make an investment with insufficient ROI (Return on Investment)?
> Insufficient information.
> College Scorecard [1] is a database with a web interface for finding and comparing schools according to a number of objective criteria. CollegeScorecard launched in 2015. It lists "Average Annual Cost", "Graduation Rate", and "Salary After Attending" on the search results pages. When you review a detail page for an institution, there are many additional statistics; things like: "Typical Total Debt After Graduation" and "Typical Monthly Loan Payment".
> The raw data behind CollegeScorecard can be downloaded from [2]. The "data_dictionary" tab of the "Data Dictionary" spreadsheet describes the data schema.
> [1] https://collegescorecard.ed.gov/
> [2] https://collegescorecard.ed.gov/data/
> Khan Academy > "College, careers, and more" [3] may be a helpful supplement for funding a full-time college admissions counselor in a secondary education institution
> [3] https://www.khanacademy.org/college-careers-more
https://www.khanacademy.org/college-careers-more/college-adm... :
- [ ] Video & exercise / Jupyter notebook under Exploring college options for Return on Investment (according to e.g. CollegeScorecard data)