Contents^
Items^
The Trillion-Dollar Annual Interest Payment
> Given the recent actions of Congress, and the years of prior inaction in changing the nation’s fiscal path, the U.S. government’s annual interest payment will eclipse annual defense spending in only six years. By 2025, annual interest costs on the national debt will reach $724 billion, while annual defense spending will reach $706 billion. To put that into perspective, in the 2018 fiscal year, the U.S. government spent $325 billion in interest payments and spent $622 billion in defense (Exhibit 2).
Why would you cut taxes and debt finance our nation's future?
Oak, a Free and Open Certificate Transparency Log
Great use case for blockchain technology
CT logs are already chained
> Great use case for blockchain technology
>> CT logs are already chained
Trillian is a centralized Merkle tree: it doesn't support native replication (AFAIU?) and there is a still a password that can delete or recreate the chain (though we can track for any such inappropriate or errant modifications (due to e.g. solar flares) by manually replicating and verifying every entry in the chain, or trusting that everything before whatever we consider to be a known hash (that could be colliding) is unmodified (since the last time we never verified those entries)).
According to the trillian README, trillian depends upon MySQL/MariaDB and thus internal/private replication is as good as the SQL replication model (which doesn't have a distributed consensus algorithm like e.g. paxos).
A Merkle tree alone is not a blockchain; though it provides more assurance of data integrity than a regular tree, verifying that the whole chain of hashes actually is good and distributed replication without configuring e.g. SSL certs are primary features of blockchains.
There are multiple certificate issuers, multiple logs, and multiple log verifiers. With no single point of failure, that doesn't sound centralized to me?
Which components of the system are we discussing?
PKI is necessarily centralized: certs depend upon CA certs which can depend upon CA certs. If any CA is compromised (e.g. by theft or brute force (which is inestimably infeasible given current ASIC resources' preference for legit income)) that CA can sign any CRL. A CT log and a CT log verifier can help us discover that a redundant and so possibly unauthorized cert has been issued for a given domain listed in an x.509 cert CN/SAN.
The CT log itself - trillian, for Google and now LetsEncrypt, too - though, runs on MySQL; which has one root password.
The system of multiple independent, redundant CT logs is built upon databases that depend upon presumably manually configured replication keys.
Does my browser call a remote log verifier API over (hopefully pinned with a better fingerprint than MD5) HTTPS?
There are multiple issuers, so from an availability point of view, if one is down, you could choose another. They submit to at least two logs, so if one log is unavailable you could read the other one. This is a form of decentralization.
Now, from a security point of view, it only takes breaking into one issuer to issue bad certificates. But maybe classifying everything as either centralized or decentralized is too simple?
Centralized and decentralized are overloaded terms. We could argue that every system that depends upon DNS is a centralized (and thus has a single point of failure).
We could describe replication models as centralized or decentralized. Master/master SQL replication is still not decentralized (regardless of whether there are multiple A records or multiple static IPs configured in the client).
With PKI, we choose the convenience of trusting a CA bundle over having to manually check every cert fingerprint.
Whether a particular chain is centralized or decentralized is often bandied about. When there are a few mining pools that effectively choose which changes are accepted, that's not decentralized either.
That there are multiple redundant independent CT logs is a good thing.
How do I, as a concerned user, securely download (and securely mirror?) one or all of the CT logs and verify that none of the record hashes don't depend upon the previous hash? If the browser relies upon a centralized API for checking hash fingerprints, how is that decentralized?
Looks like there is a bit here about how to get started: https://security.stackexchange.com/questions/167366/how-can-...
Most people aren't going to do it, but I think that's not really the point, any more than every user needs to review Linux kernel patches. But I wonder if there are enough "eyes" on this and how would we check?
Death rates from energy production per TWh
Apparently the deaths are justified because energy.
Are the subsidies and taxes (incentives and penalties) rational in light of the relative harms of each form of energy?
"Study: U.S. Fossil Fuel Subsidies Exceed Pentagon Spending" https://www.rollingstone.com/politics/politics-news/fossil-f...
> The IMF found that direct and indirect subsidies for coal, oil and gas in the U.S. reached $649 billion in 2015. Pentagon spending that same year was $599 billion.
> The study defines “subsidy” very broadly, as many economists do. It accounts for the “differences between actual consumer fuel prices and how much consumers would pay if prices fully reflected supply costs plus the taxes needed to reflect environmental costs” and other damage, including premature deaths from air pollution.
IDK whether they've included the costs of responding to requests for help with natural disasters that are more probable due to climate change caused by these "externalties" / "external costs" of fossil fuels.
Energy saves lives so some risk probably is justified.
Why isn't the market choosing the least harmful, least lethal energy sources? Energy is for the most part entirely substitutable: switching costs for consumers like hospitals are basically zero.
(Everyone is free to invest in clean energy at any time)
Switching costs for the entire society are far from zero.
100% Renewable Energy https://en.wikipedia.org/wiki/100%25_renewable_energy
> The main barriers to the widespread implementation of large-scale renewable energy and low-carbon energy strategies are political rather than technological. According to the 2013 Post Carbon Pathways report, which reviewed many international studies, the key roadblocks are: climate change denial, the fossil fuels lobby, political inaction, unsustainable energy consumption, outdated energy infrastructure, and financial constraints.
We need to make the external costs of energy production internal in order to create incentives to prevent these fossil fuel deaths and other costs.
Use links not keys to represent relationships in APIs
A thing may be identified by a URI (/person/123) for which there are zero or more URL routes (/person/123, /v1/person/123). Each additional route complicates caching; redirects are cheap for the server but slower for clients.
JSONLD does define a standard way to indicate that a value is a link: @id (which can be specified in a/an @context) https://www.w3.org/TR/json-ld11/
One additional downside to storing URIs instead of bare references is that it's more complicated to validate a URI template than a simple regex like \d+ or [abcdef\=\d+]+
And on a more fundamental standpoint I get that disk space is cheap theses days. But you just doubled, if not worse, the storage space required for a key for a vague reason.
It may never make any difference on a small dataset, where storage was anyway unaware of differences between integer and text. But it would be hiding in the dark. And maybe in a few year a new recruit will have the outstanding idea to convert text-link to bigint to save some space...
No Python in Red Hat Linux 8?
/usr/bin/python can point to either /usr/bin/python3 or (as PEP 394 currently recommends) /usr/bin/python2
$ alternatives --config python
JMAP: A modern, open email protocol
What are the optimizations in JMAP that make it faster than, say, Solid? Solid is built on a bunch of W3C Web, Security, and Linked Data Standards; LDP: Linked Data Protocol, JSON-LD: JSON Linked Data, WebID-TLS, REST, WebSockets, LDN: Linked Data Notifications. [1][2] Different worlds, I suppose.
There's no reason you couldn't represent RFC5322 data with RDF as JSONLD. There's now a way to do streaming JSON-LD.
LDP does paging and querying.
Solid supports pubsub with WebSockets and LDN. It may or may not (yet?) be as efficient for synchronization as JMAP, but it's definitely designed for all types of objects with linked data web standards; and client APIs can just parse JSON-LD.
[1] https://github.com/solid/information#solid-specifications
[2] https://github.com/solid/solid-spec/issues/123 "WebSockets and HTTP/2" SSE (Server-Side Events)
JMAP: JSON Meta Application Protocol https://en.wikipedia.org/wiki/JSON_Meta_Application_Protocol
Is there a OpenAPI Specification for JMAP? There are a bunch of tools for Swagger / OpenAPIs: DRY interactive API docs, server implementations, code generators: https://swagger.io/tools/open-source/ https://openapi.tools/
Does JMAP support labels; such that I don't need to download a message and an attachment and mark it as read twice like labels over IMAP?
How does this integrate with webauthn; is that a different layer?
(edit) Other email things: openpgpjs; Web Key Directory /.well-known/openpgpkey/*; if there's no webserver on the MX domain, you can use the ACME DNS challenge to get free 3-month certs from LetsEncrypt.
I would say your comment, and the first reply, both demonstrate quite effectively why JMAP is probably a better choice for email.
> It may or may not (yet?) be as efficient for synchronization as JMAP, but it's definitely designed for all types of objects
If we hypothetically allow for equal adoption & mindshare of both, and assume both are non-terrible designs, I'd guess the one designed for "all types of objects" is less likely to ever be as efficient as the one designed with a single use-case in mind.
And narrow focus is not only good for optimising specific use-cases, it's also good for adoption as people immediately understand what your protocol is for and how to use it when it has a single purpose and a single source of truth for reference spec, rather than a series of disparate links and vague all-encompassing use-cases.
Solid has brilliant people behind it, but it's too broad, too ambitious, and very much lacks focus, and that will impair adoption because it isn't the "one solution" for anyone's "one problem".
--
To take another perspective on this, there are other commenters in this thread bemoaning the loss of non-HTTP-based protocols. Funnily enough, HTTP itself is a broadly used, broadly useful protocol than can be used for pretty much anything (and had TBL behind it also). The big difference was that Tim wasn't proposing that HTTP be the solution to all our internet problems and needs in 1989—it was just for hypertext documents. It's only now, post-adoption, that it is used for so much more than that.
> If we hypothetically allow for equal adoption & mindshare of both, and assume both are non-terrible designs, I'd guess the one designed for "all types of objects" is less likely to ever be as efficient as the one designed with a single use-case in mind.
This is a generalization that is not supported by any data.
Standards enable competing solutions. Competing solutions often result in performance gains and efficiency.
Hopefully, there will be performant implementations and we won't need to reinvent the wheel in order to synchronize and send notifications for email, contacts, and calendars.
> There's no reason you couldn't represent RFC5322 data with RDF as JSONLD. There's now a way to do streaming JSON-LD.
Is there a reason you'd want to? I clicked all your links but I still have no idea what Solid is.
To eliminate the need for domain-specific parser implementations on both server and client, make it easy to index and search this structured data, and to link things with URIs and URLs like other web applications that also make lots of copies.
Solid is a platform for decentralized linked data storage and retrieval with access controls, notifications, WebID + OAuth/OpenID. The Wikipedia link and spec documents have a more complete description that could be retrieved and stored locally.
Grid Optimization Competition
From "California grid data is live – solar developers take note" https://news.ycombinator.com/item?id=18855820 :
>> It looks like California is at least two generations of technology ahead of other states. Let’s hope the rest of us catch up, so that we have a grid that can make an asset out of every building, every battery, and every solar system.
> +1. Are there any other states with similar grid data available for optimization; or any plans to require or voluntarily offer such a useful capability?
How do these competitions and the live actual data from California-only (so far; AFAIU) compare?
Are there standards for this grid data yet? Without standards, how generalizable are the competition solutions to real-world data?
Blockchain's present opportunity: data interchange standardization
What are the current standards efforts for blockchain data interchange?
W3C JSON-LD, ld-signatures + lds-merkleproof2017 (normalize the data before signing it so that the signature is representation-independent (JSONLD, RDFa, RDF, n-triples)), W3C DID Decentralized Identifiers, W3C Verifiable Claims, Blockcerts.org
W3C Credentials Community Group: https://w3c-ccg.github.io/community/work_items.html#draft-sp... (DID, Multihash (IETF), [...])
"Blockchain Credential Resources; a gist" https://gist.github.com/westurner/4345987bb29fca700f52163c33...
Specifically for payments:
https://www.w3.org/TR/?title=payment (the W3C Payment Request API standardizes browser UI payment/checkout workflows)
ILP: Interledger Protocol https://interledger.org/rfcs/0027-interledger-protocol-4/
> W3C JSON-LD
https://www.w3.org/TR/json-ld/ (JSON-LD 1.0)
https://www.w3.org/TR/json-ld11/ (JSON-LD 1.1)
> ld-signatures + lds-merkleproof2017 (normalize the data before signing it so that the signature is representation-independent (JSONLD, RDFa, RDF, n-triples))
https://w3c-dvcg.github.io/ld-signatures/
https://w3c-dvcg.github.io/lds-merkleproof2017/ (2017 Merkle Proof Linked Data Signature Suite)
> W3C DID Decentralized Identifiers
https://w3c-ccg.github.io/did-primer/
>> A Decentralized Identifier (DID) is a new type of identifier that is globally unique, resolveable with high availability, and cryptographically verifiable. DIDs are typically associated with cryptographic material, such as public keys, and service endpoints, for establishing secure communication channels. DIDs are useful for any application that benefits from self-administered, cryptographically verifiable identifiers such as personal identifiers, organizational identifiers, and identifiers for Internet of Things scenarios. For example, current commercial deployments of W3C Verifiable Credentials heavily utilize Decentralized Identifiers to identify people, organizations, and things and to achieve a number of security and privacy-protecting guarantees.
> W3C Verifiable Claims
https://github.com/w3c/verifiable-claims
https://w3c.github.io/vc-data-model/ (Data Model)
https://w3c.github.io/vc-use-cases/ (Use Cases: Education, Healthcare, Professional Credentials, Legal Identity,)
Ask HN: Value of “Shares of Stock options” when joining a startup
I got an offer from a US start-up (well +25 employees) which has an office in EU where I would join them.
The offer's base salary is good (ie. higher than average for senior positions for that location) but I intend to negotiate it further, as I have possible other options. patio11's negotiation guide was a great read in that regard.
However, I'm relocating from a non-EU/US country, and I don't have a single idea about the financial systems, stock markets, and how to evaluate "15k shares of stock-options" or what "Stock Option and Grant Plan" means, I'm asking you fellow HNers about this part.
Do I just treat them as worthless and focus on base salary (as some internet sources suggest) or is there a formula to evaluate what they would be worth in say 2 years for instance ?
There are a number of options/equity calculators:
https://tldroptions.io/ ("~65% of companies will never exit", "~15% of companies will have low exits*", "~20% of companies will make you money")
https://comp.data.frontapp.com/ "Compensation and Equity Calculator"
http://optionsworth.com/ "What are my options worth?"
http://foundrs.com/ "Co-Founder Equity Calculator"
CMU Computer Systems: Self-Grading Lab Assignments (2018)
These look fun; in particular the "Attack Lab".
Dockerfiles might be helpful and easy to keep updated. Alpine Linux or just busybox are probably sufficient?
The instructor set could extend FROM the assignment image and run a few tests with e.g. testinfra (pytest)
You can also test code written in C with gtest.
I haven't read through all of the materials: are there suggested (automated) fuzzing tools? Does OSS-Fuzz solve?
Are there references to CWE and/or the SEI CERT C Coding Standard rules? https://wiki.sei.cmu.edu/confluence/plugins/servlet/mobile?c...
"How could we have changed our development process to catch these bugs/vulns before release?"
"If we have 100% [...] test coverage, would that mean we've prevented these vulns?"
What about 200%?
What on earth would 200% test coverage mean?
Coverage for all the code that was written, and all the code that has yet to be typed, of course.
All thanks to quantum computing :)
Show HN: Debugging-Friendly Tracebacks for Python
pytest also has helpful tracebacks; though only for test runs.
With nose-progressive, you can specify --progressive-editor or update the .noserc so that traceback filepaths are prefixed with your preferred editor command.
vim-unstack parses paths from stack traces / tracebacks (for a number of languages including Python) and opens each in a split at that line number. https://github.com/mattboehm/vim-unstack
Here's the Python regex from my hackish pytb2paths.sh script:
'\s+File "(?P<file>.*)", line (?P<lineno>\d+), in (?P<modulestr>.*)$'
Why isn't 1 a prime number?
Funny true story about this: One time a man came up to me and handed me his phone, and asked me to call his mom. He was about to pass out and he asked me not call an ambulance (God bless America) , he appeared to have a concussion. By this time there was about 10-15 people around him and he could barely talk. I asked him what his phone code was and instead of just giving it to me he said "it's the first four prime numbers". Immediately, about five people shout "1,2,3,5". I am no longer holding the phone, because I handed to someone else to make sure it was okay. Sure enough, I was in a mathematical proofs class and we had just discussed this topic. So, I say "one is not a prime number". Of course, we get the phone unlocked in the second try with "2,3,5,7" and the guys mom is on the way. Everyone thought I was a genius a hero.
You can also dial emergency contacts without unlocking the phone. They are accessible from the medical ID page on iOS, I assume Android has similar.
> You can also dial emergency contacts without unlocking the phone. They are accessible from the medical ID page on iOS, I assume Android has similar.
You can set a Lock Screen Message by searching for "Lock Screen Message" in the Android Settings.
You can also create an "ICE (In Case of Emergency)" contact.
How do we know when we’ve fallen in love? (2016)
Where is the feeling felt? Is there an associated color or a shape or a kinesiologic position?
"Love styles" https://en.wikipedia.org/wiki/Love_styles
"Greek words for love" https://en.wikipedia.org/wiki/Greek_words_for_love
Rare and strange ICD-10 codes
These are too funny: https://www.empr.com/home/features/the-strangest-and-most-ob...
Choice selection:
V91.07X - Burn Due to Water Skis on Fire(?!)
W61.42XA. Struck By Turkey, initial encounter. If a duck is involved, there's a code for that too. (W61.62X)
R46.1. Bizarre Personal Appearance.
My favorite: T63.012D: Toxic effect of rattlesnake venom, intentional self-harm, subsequent encounter
So basically you have a case where someone not only intentionally wanted to harm themselves with rattlesnake venom once, but at least twice!
> So basically you have a case where someone not only intentionally wanted to harm themselves with rattlesnake venom once, but at least twice!
No, you misunderstand the terminology. "Subsequent encounter" means with the doctor not with the rattlesnake. AKA followup care during or after recovery.
> No, you misunderstand the terminology. "Subsequent encounter" means with the doctor not with the rattlesnake
You can reference ICD codes with the schema.org/code property of schema.org/MedicalEntity and subclasses. https://schema.org/docs/meddocs.html
"Subsequent encounter" is poorly defined. IMHO, there should be a code for this.
> "Subsequent encounter" is poorly defined.
"Poorly defined" is poorly defined. Explanations of when to use the D make perfect sense to me.
"The 7th character for “subsequent encounter” is to be used for all encounters after the patient has received active treatment of the condition and is receiving routine care for the condition during the healing or recovery phase. Examples of subsequent encounters include cast change or removal, x-ray to check healing status of a fracture, removal of external or internal fixation device, medication adjustments, and other aftercare and follow-up visits following active treatment of the injury or condition. Encounters for rehabilitation, such as physical and occupational therapy, are another example of the use of the “subsequent encounter” 7th character. For aftercare following an injury, the acute injury code should be assigned with the 7th character for subsequent encounter."
This parses the ICD 10 CM with lxml: https://github.com/westurner/pycd10api/blob/master/pycd10api...
Python Requests III
So what's the difference to the predecessor?
asyncio, HTTP/2, connection pooling, timeouts, Python 3.6+
README > "Feature Support" https://github.com/kennethreitz/requests3/blob/master/README...
https://github.com/aio-libs/aiohttp already supports asyncio. Why do I need this for asyncio?
Anyway, I see that Issues are disabled for the repo. Is that the new way to develop? /s
Tools built with aiohttp: https://libraries.io/pypi/aiohttp/usage
Tools built with requests: https://libraries.io/pypi/requests/usage
Post-surgical deaths in Scotland drop by a third, attributed to a checklist
My former roommate is a pilot. When I first met him, I noticed that he uses checklists for just about everything, even the most basic everyday tasks.
After some time, I decided to apply that same mentality to my own life. Both in private and work situations.
I get it now. Checklists reduce cognitive load tremendously well, even for basic tasks. As an example: I have a checklist for when I need to travel, it contains stuff like what to pack, asking someone to feed my cat, check windows are closed, dishwasher empty, heating turned down, etc. Before the checklist, I would always be worried I forgot something, now I can relax.
Also, checklists are a great way to improve processes. Basically a way to debug your life. For instance: I once forgot to empty the trash bin before a long trip, I added that to my checklist and haven't had a smelly surprise ever since ;)
This is described in the book called "the checklist manifesto". Very good book by the way.
https://www.amazon.com/Checklist-Manifesto-How-Things-Right/...
The medical community uses checklists. They've been using checklists for a while. A number of studies found that they only help for a few months, while they're new.
That said, "the medical community" is not a homogeneous monolith, and you can absolutely find regional variation in what checklists are used for, how detailed they are, how closely they're followed, how people are accountable for keeping to them, etc.
"The Checklist Manifesto" chose to overlook the studies about how transient the benefit of checklists is.
Are these paper based? Any checklist apps out there for our daily use?
GitHub and GitLab support task checklists in Markdown and also project boards which add and remove labels like 'ready' and 'in progress' when cards are moved between board columns; like kanban:
- [ ] not complete
- [x] completed
Other tools support additional per-task workflow states:
- [o] open
- [x (2019-04-17)] completed on date
I worked on a large hospital internal software project where the task was to build a system for reusable checklists editable through the web that prints them out in duplicate or triplicate at nearby printers. People really liked having the tangible paper copy.
"The Checklist Manifesto" by Atul Gawande was published while I worked there. TIL pilots have been using checklists for process control in order to reduce error for many years.
Evernote, RememberTheMilk, Google Tasks, and Google Keep all support checklists. Asana and Gitea and TaskWarrior support task dependencies.
A person could carry around a Hipster PDA with Bullet Journal style tasks lists with checkboxes; printed from a GTD service with an API and a @media print CSS stylesheet: https://en.wikipedia.org/wiki/Hipster_PDA
I'm not aware of very many tools that support authoring reusable checklists with structured data elements and data validation.
...
There are a number of configuration management systems like Puppet, Chef, Salt, and Ansible that build a graph of completable and verifiable tasks and then depth-first traverse said graph (either with hash randomization resulting in sometimes different traversals or with source order as an implicit ordering)
Resource scheduling systems like operating systems and conference room schedulers can take ~task priority into account when optimally ordering tasks given available resources; like triage.
Scheduling algorithms: https://news.ycombinator.com/item?id=15267146
TodoMVC catalogs Todo list implementations with very many MV* JS Frameworks: http://todomvc.com
Reusable Checklists could be done with a simple text document that you duplicate every time you need it no ?
For sure. Though many tools don't read .txt (or .md/.markdown) files.
GitHub and GitLab support (multiple) Issue and Pull Request templates:
Default: /.github/ISSUE_TEMPLATE.md || Configure in web interface
/.github/ISSUE_TEMPLATE/Name.md || /.gitlab/issue_templates/Name.md
Default: /.github/PULL_REQUEST_TEMPLATE.md || Configure in web interface
/.github/PULL_REQUEST_TEMPLATE/Name.md || /.gitlab/merge_request_templates/Name.md
There are template templates in awesome-github-templates [1] and checklist template templates in github-issue-templates [2].
I really want slack to adopt these as well...
I'll often be on call with customer and create a checklist on MacOS Notes on the fly. Then will copy paste that in slack or github for simple tracking.
Mattermost supports threaded replies and Markdown with checklist checkboxes
You can post GitHub/GitLab project updates to a Slack/Mattermost channel with webhooks (and search for and display GH/GL issues with /slash commands); though issue edits and checkbox state changes aren't (yet?) included in the events that channels receive.
Apply to Y Combinator
The next Startup School course begins in July: https://www.startupschool.org/
> Learn how to start a startup with YC’s free 10-week online course.
> Here is a list of the top 100 Y Combinator companies by valuation (why valuation?), as of October 2018. We also included YC’s top 12 exits.
Here's the list of the 1,900 Y Combinator companies through Winter 2019 (W19) https://www.ycombinator.com/companies/
"Startup Playbook" by Sam Altman (YC Founder) and Illustrated by Gregory Koberger is also a good read: https://playbook.samaltman.com/
Trunk-Based Development vs. Git Flow
One major advantage of the gitflow/hubflow git workflows is that there is a standard way of merging across branches. For example, a 'hotfix' branch is merged into the stable master branch and also develop with one standard command; there's no need to re-explain and train new devs on how the branches were supposed to work here. I even copied the diagram(s) into my notes: https://westurner.github.io/tools/#hubflow
IMHO, `git log` on the stable master branch containing each and every tagged release is preferable to having multiple open release branches.
Requiring tests to pass before a PR gets merged is a good policy that's independent of the trunk or gitflow workflow decision.
Ask HN: Anyone else write the commit message before they start coding?
I feel like I just learned how to use Git: writing the message first thing has made me a lot more productive. I'm wondering if anyone else does this; I know test driven development is a thing, where people write tests before code, and this seems like a logical extension.
Ask HN: Datalog as the only language for web programming, logic and database
Can Datalog be used as the only language which we can use for writing server-side web application, complex domain business logic and database querying?
Are there any efforts made in this direction.
To quote myself from a post the other day https://news.ycombinator.com/item?id=19407170 :
> PyDatalog does Datalog (which is ~Prolog, but similar and very capable) logic programming with SQLAlchemy (and database indexes) and apparently NoSQL support. https://sites.google.com/site/pydatalog/
> Datalog: https://en.wikipedia.org/wiki/Datalog
> ... TBH, IDK about logic programming and bad facts. Resilience to incorrect and incredible information is - I suppose - a desirable feature of any learning system that reevaluates its learnings as additional and contradictory information makes its way into the datastores.
I'm not sure that Datalog is really necessary for most CRUD operations; SQLAlchemy and the SQLAlchemy ORM are generally sufficient for standard database querying CRUD.
The cortex is a neural network of neural networks
Is there a program like codeacademy but for learning sysadmin?
if not, anyone wanna build one?
A few sysadmin and devops curriculum resources; though none but Beaker and Molecule are interactive with any sort of testing AFAIU:
"System Administrator" https://en.wikipedia.org/wiki/System_administrator
"Software Configuration Management" (SCM) https://en.wikipedia.org/wiki/Software_configuration_managem...
"DevOps" https://en.wikipedia.org/wiki/DevOps
"OpsSchool Curriculum" http://www.opsschool.org
- Soft Skills 101, 201
- Labs Exercises
- Free. Contribute
awesome-sysadmin > configuration-management https://github.com/kahun/awesome-sysadmin/blob/master/README...
- This could list reusable module collections such as Puppet Forge and Ansible Galaxy;
- And module testing tools like Puppet Beaker and Ansible Molecule (that can use Vagrant or Docker to test a [set of] machines)
https://github.com/stack72/ops-books
- I'd add "Time Management for System Administrators" (2005)
https://landing.google.com/sre/books/
- There's now a "Site Reliability Workbook" to go along with the Google SRE book. Both are free online.
https://response.pagerduty.com
- The PagerDuty Incident Response Documentation is also free online.
- OpsGenie has a free plan also with incident response alerting and on-call management.
There are a number of awesome-devops lists.
Minikube and microk8s package Kubernetes into a nice bundle of distributed systems components that'll run on Lin, Mac, Win. You can convert docker-compose.yml configs to Kubernetes pods when you decide that it should've been HA with a load balancer SPOF and x.509 certs and a DRP (Disaster Recovery Plan) from the start!
Maybe You Don't Need Kubernetes
The argument that Kubernetes adds complexity is, in my opinion, bogus. Kubernetes is a "define once, forget about it" type of infrastructure. You define the state you want your infrastructure to and Kubernetes takes care of maintaining that state. Tools like Ansible and Puppet, as great as they are, do not guarantee your infrastructure will end up in the state you defined and you easily end up with broken services. The only complexity in kubernetes is the fact that it forces you to think and carefully design your infra in a way people aren't used to, yet. More upfront, careful thinking isn't complexity. It can only benefit you in the long run.
There is, however, a learning curve to Kubernetes, but it isn't this sharp. It does require you to sit down and read the doc for 8 hours, but that a small price to pay.
A few month back I wrote a blog post[1] that, through walking through the few different infrastructures my company experimented with over the years, surfaces many reasons one would want to use [a managed] Kubernetes. (For a shorter read, you can probably start at reading at [2])
[1]: https://boxunix.com/post/bare_metal_to_kube
[2]: https://boxunix.com/post/bare_metal_to_kube/#_hardware_infra...
It's pretty common for new technologies to advertise themselves as "adopt, and forget about it", but in my experience it's unheard of that any actually deliver on this promise.
Any technology you adopt today is a technology you're going to have to troubleshoot tomorrow. (I don't think the 15,000 Kubernetes questions on StackOverflow are all from initial setup.) I can't remember the last [application / service / file format / website / language / anything related to computer software] that was so simple and reliable that I wasn't searching the internet for answers (and banging my head against the wall because of) the very next month. It was probably something on my C=64.
As Kernighan said back in the 1970's, "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?" I've never used Kubernetes, but I've read some articles about it and watched some videos, and despite the nonstop bragging about its simplicity (red flag #1), I'm not sure I can figure out how to deploy with it. I'm fairly certain I wouldn't have any hope of fixing it when it breaks next month.
Hearing testimonials only from people who say "it doesn't break!" is red flag #2. No technology works perfectly for everyone, so I want to hear from the people who had to troubleshoot it, not the people who think it's all sunshine and rainbows. And those people are not kind, and make it sound like the cost is way more than just "8 hours reading the docs" -- in fact, the docs are often called out as part of the problem.
> As Kernighan said back in the 1970's, "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?"
What a great quote. Thanks
Some more quotes taken from http://typicalprogrammer.com/what-does-code-readability-mean that you might find interesting.
Any fool can write code that a computer can understand. Good programmers write code that humans can understand. – Martin Fowler
Just because people tell you it can’t be done, that doesn’t necessarily mean that it can’t be done. It just means that they can’t do it. – Anders Hejlsberg
The true test of intelligence is not how much we know how to do, but how to behave when we don’t know what to do. – John Holt
Controlling complexity is the essence of computer programming. – Brian W. Kernighan
The most important property of a program is whether it accomplishes the intention of its user. – C.A.R. Hoare
No one in the brief history of computing has ever written a piece of perfect software. It’s unlikely that you’ll be the first. – Andy Hunt
> Tools like Ansible and Puppet, as great as they are, do not guarantee your infrastructure will end up in the state you defined and you easily end up with broken services.
False dilemma. Ansible and Puppet are great tools for configuring kubernetes, kubernetes worker nodes, and building container images.
Kubernetes does not solve for host OS maintenance; though there are a number of host OS projects which remove most of what they consider to be unnecessary services, there's still need to upgrade kubernetes nodes and move pods out of the way first (which can be done with e.g. Puppet or Ansible).
As well, it may not be appropriate for monitoring to depend upon kubernetes; there again you have nodes to manage with an SCM tool.
Quantum Machine Appears to Defy Universe’s Push for Disorder
Maybe an example or related to time crystals?
"Scar (physics)": https://en.wikipedia.org/wiki/Scar_(physics)
> Scars are unexpected in the sense that stationary classical distributions at the same energy are completely uniform in space with no special concentrations along periodic orbits, and quantum chaos theory of energy spectra gave no hint of their existence
Pytype checks and infers types for your Python code
How does pytype compare with the PyAnnotate [1] and MonkeyType [2] dynamic / runtime PEP-484 type annotation type inference tools?
How I'm able to take notes in mathematics lectures using LaTeX and Vim
As a feat in itself this is definitely very impressive, but I wonder if it's really worth anyone's time to spend precious lecture time with your mind fully occupied in the mechanical task of taking notes rather than actually absorbing and engaging with the content. Especially for an extremely content-dense subject like Mathematics, where you need all your concentration just to process what you are reading and follow along the logic.
There is absolutely no dearth of study material on the internet. It's one thing to take notes as part of your study process. There it helps solidify your understanding. But surely taking notes on the fly when you have been barely introduced to the subject isn't going to help with that.
> mechanical task of taking notes rather than actually absorbing and engaging with the content
The mechanical task of taking notes is one of the most important parts of actually absorbing the material. It is not an either-or. Hearing/seeing the information, processing it in a way that makes sense to you individually, and then mechanically writing it down in a legible manner is one of the main methods that your brain learns. It's one of the primary reasons that taking notes is important in the first place. This is referred to as the "encoding hypothesis" [1].
There are actually even studies [2] that show that tools that assist in more efficient note taking, such as taking notes via typing rather than by hand, are actually detrimental to absorbing information, as it makes it easier for you to effectively pass the information directly from your ears to your computer without actually doing the processing that is required when writing notes by hand. This is why many universities prefer (or even require) notes to be taken by hand, and disallow laptops in class.
1: https://www.sciencedirect.com/science/article/pii/0361476X78...
2: https://www.npr.org/2016/04/17/474525392/attention-students-...
> The mechanical task of taking notes is one of the most important parts of actually absorbing the material. It is not an either-or. Hearing/seeing the information, processing it in a way that makes sense to you individually, and then mechanically writing it down in a legible manner is one of the main methods that your brain learns. It's one of the primary reasons that taking notes is important in the first place. This is referred to as the "encoding hypothesis" [1].
There's almost certainly an advantage to learning to think about math using a publishable symbol set like LaTeX.
We learn by reinforcement; with feedback loops that may take until weeks later in a typical university course.
> There are actually even studies [2] that show that tools that assist in more efficient note taking, such as taking notes via typing rather than by hand, are actually detrimental to absorbing information, as it makes it easier for you to effectively pass the information directly from your ears to your computer without actually doing the processing that is required when writing notes by hand.
Handwriting notes is impractical for some people due to e.g. injury and illegibility.
The linked study regarding retention and handwritten versus typed notes has been debunked with references that are referenced elsewhere in comments on this post. There have been a few studies with insufficient controls (lack of randomization, for one) which have been widely repeated by educators who want to be given attention.
Doodling has been shown to increase information retention. Maybe doodling as a control really would be appropriate.
Banning laptops from lectures is not respectful of students with injury and illegible handwriting. Asking people to put their phones on silent (so they can still make and take emergency calls) and refrain from distracting other students with irrelevant content on their computers is reasonable and considerate.
(What a cool approach to math note-taking. I feel a bit inferior because I haven't committed to learning that valuable, helpful skill and so that's stupid and you're just wasting your time because that's not even necessary when all you need to do is retain the information you've paid for for the next few months at most. If course, once you get on the job, you'll never always be using that tool and e.g. latex2sympy to actually apply that theory to solving a problem that people are willing to pay for. So, thanks for the tips and kudos, idiot)
LHCb discovers matter-antimatter asymmetry in charm quarks
So, does this disprove all of supersymmetry? https://en.wikipedia.org/wiki/Supersymmetry
No, supersymmetry and charge-parity symmetry are different.
Ah, thanks.
"CPT Symmetry" https://en.wikipedia.org/wiki/CPT_symmetry
"CP Violations" https://en.wikipedia.org/wiki/CP_violation
"Charm quark" https://en.wikipedia.org/wiki/Charm_quark :
> The antiparticle of the charm quark is the charm antiquark (sometimes called anticharm quark or simply anticharm), which differs from it only in that some of its properties have equal magnitude but opposite sign.
React Router v5
Using react router on one of my personal projects (~ 30k loc) was probably one of my largest regrets. At every turn it seemed designed to do the thing I wouldn’t expect, or have arbitrary restrictions that made my life tougher.
Some examples:
* there's no relative routes https://github.com/ReactTraining/react-router/issues/2172
* there's no way to refresh the page https://github.com/ReactTraining/react-router/issues/1982 ("that's your responsibility, not ours")
* The scroll position will stick when you navigate to a new route, causing you to need to create a custom component wrapper to manage scrolling https://github.com/ReactTraining/react-router/issues/3950
* React router's <link> do not allow you to link outside of the current site https://github.com/ReactTraining/react-router/issues/1147 so you have to make an in those cases. This doesn’t sound bad, but it’s particularly frustrating when dealing with dynamic or user generated content. Why can’t they handle this simple case?
> there's no relative routes
> React router's <link> do not allow you to link outside of the current site
Both valid points. Not show stoppers though. And not enough to make me regret using the library. The solution to the external links issue is a one-liner.
> there's no way to refresh the page https://github.com/ReactTraining/react-router/issues/1982 ("that's your responsibility, not ours")
That is your responsibility. Why do you need the routing library to handle page refreshing for you?
> the scroll position will stick when you navigate to a new route
Making the page scroll to the top when navigating to a new page is trivial. I would 100% rather have this problem instead of the opposite: scroll jumping to the top when I don't want it to. That's so much harder to fix.
> Navigating to the same page would not actually reload the page, it would just trigger a componentDidMount() on all components in the page again, which led me to have a lot of bugs when I did some initialization in my constructor
That's exactly what it's supposed to do. It's a client side routing solution. (I'm also pretty sure that it doesn't remount)
From the issues that you've had with the library, it seems like client side routing is not actually what you're looking for. If you regret using it so much, may I ask what the alternative would be?
IMO a project either needs client side routing, or it doesn't. If it does, then React Router is the obvious choice. Otherwise, of course, don't use it and save yourself from unnecessary complexity.
> The solution to the external links issue is a one-liner.
It is most definitely not a one-liner.
> Making the page scroll to the top when navigating to a new page is trivial.
So do it for me. There’s no reason a routing library should break default behavior.
> That's exactly what it's supposed to do. It's a client side routing solution. (I'm also pretty sure that it doesn't remount)
Then it’s “supposed” to have inconsistent behavior. Navigating anywhere else in my site will call constructors to all my components. Navigating to the same page won’t.
> From the issues that you've had with the library, it seems like client side routing is not actually what you're looking for.
My site was a music site. I wanted clientside routing so I could keep music playing while you navigated between links. Seemed like a slam dunk for clientside routing to me.
> may I ask what the alternative would be?
Quite honestly I would go onto GitHub and look for any routing solution that didn’t have multiple closed unfixed issues with hundreds of thumbs up.
> So do it for me.
I think this will work:
Put this wherever you put your reusable functions:
`const scrollToTop = () => document.getElementById('root').scrollIntoView();`
And put this in the components / functions you want the scrollToTop effect to work on:
`useEffect(() => { scrollToTop() }, []);`
And put this in your CSS:
`html { scroll-behavior: smooth }`
(Edit to add: Not saying React Router is something you should / shouldn’t use. Just wanted to share that code in case it helps unblock anyone.)
Accidentally downvoted on mobile (and upvoted two others). Thanks for this.
"Scroll Restoration" https://reacttraining.com/react-router/web/guides/scroll-res...
Experimental rejection of observer-independence in the quantum world
Objective truth!? A question for epistemologists to decide.
How could they record their high entropy (?) solipsistic observations in an immutable datastore in such as way as to have probably zero knowledge of the other party's observations?
Anyways, that's why I only read the title and the abstract.
Wigner's friend experiment: https://en.wikipedia.org/wiki/Wigner%27s_friend
Show HN: A simple Prolog Interpreter written in a few lines of Python 3
Cool tests! PyDatalog does Datalog (which is ~Prolog, but similar and very capable) logic programming with SQLAlchemy (and database indexes) and apparently NoSQL support. https://sites.google.com/site/pydatalog/
Datalog: https://en.wikipedia.org/wiki/Datalog
... TBH, IDK about logic programming and bad facts. Resilience to incorrect and incredible information is - I suppose - a desirable feature of any learning system that reevaluates its learnings as additional and contradictory information makes its way into the datastores.
Thanks for the feedback :) I'll definitely check out Datalog - I didn't realize they had logic programming integrated with SQLAlchemy, so it definitely sounds interesting!
How to earn your macroeconomics and finance white belt as a software developer
Thanks for the wealth of resources in this post. Here are a few more:
"Python for Finance: Analyze Big Financial Data" (2014, 2018) https://g.co/kgs/qkY8J6 ... https://pyalgo.tpq.io also includes the "Finance with Python" course and this book as a PDF and Jupyter notebooks.
Quantopian put out a call for the best Value Investing algos (implemented in quantopian/zipline) awhile back. This post links to those and other value investing resources: https://westurner.github.io/hnlog/#comment-19181453 (Ctrl-F "econo")
"Lectures in Quantitative Economics as Python and Julia Notebooks" https://news.ycombinator.com/item?id=19083479 links to these excellent lectures and a number of tools for working with actual data from FRED, ECB, Eurostat, ILO, IMF, OECD, UNSD, UNESCO, World Bank, Quandl.
One thing that many finance majors, courses, and resources often fail to identify is the role that startup and small businesses play in economic growth and actual value creation: jobs, GDP, return on direct capital investment. Most do not succeed, but it is possible to do better than index funds and have far more impact in terms of sustainable investment than as an owner of a nearly-sure-bet index fund that owns some shares and takes a hands-off approach to business management, research, product development, and operations.
Is it possible to possess a comprehensive understanding of finance and economics but still not have personal finance down? Personal finance: r/personalfinance/wiki, "Consumer science (a.k.a. home economics) as a college major" https://news.ycombinator.com/item?id=17894632
Ask HN: Relationship between set theory and category theory
I have an idea about the relationship between set theory and category theory and I would like some feedback. I would like others to see it too, and I don't know how to do it. I think it's at least interesting to look at as a slightly crazy collage, but I was a bit more excited than normal when the idea hit, so I just had to dump it all at once in this image: https://twitter.com/FamilialRhino/status/1101777965724168193 (You will have to zoom the picture in order to be able to read the scribbles.)
It has to do with resonance in the energy flowing in emergent networks. Can't quite put my finger on it, so I'll be here to answer any questions.
Thanks for reading.
"Categorical set theory" > "References" https://en.wikipedia.org/wiki/Categorical_set_theory#Referen...
From "Homotopy category" > "Concrete categories" https://en.wikipedia.org/wiki/Homotopy_category#Concrete_cat... :
> While the objects of a homotopy category are sets (with additional structure), the morphisms are not actual functions between them, but rather a classes of functions (in the naive homotopy category) or "zigzags" of functions (in the homotopy category). Indeed, Freyd showed that neither the naive homotopy category of pointed spaces nor the homotopy category of pointed spaces is a concrete category. That is, there is no faithful functor from these categories to the category of sets.
My "understanding" of category theory is extremely shallow, but that's exactly why I think my proposal makes sense. It is the kind of thing that everybody ignores for decades precisely because it's transparently obvious, like a fish that doesn't understand water.
Here is the statement:
The meaning of no category is every category.
reference: https://terrytao.wordpress.com/2008/02/05/the-blue-eyed-isla...
This was already understood by everybody in the field, no doubt. It's just that somebody has to actually say it to someone else in order for the symmetry to break. The link above has the exact description of this, from Terence Tao.
The most popular docker images each contain at least 30 vulnerabilities
Although vulnerability scanners can be a useful tool, I find it very troublesome that you can utter the sentence "this package contains XX vulnerabilities, and that package contains YY vulnerabilities" and then stop talking. You've provided barely any useful information!
The quantity of vulnerabilities in an image is not really all that useful information. A large amount of vulnerabilities in a Docker image does not necessarily imply that there's anything insecure going on. Many people don't realize that a vulnerability is usually defined as "has a CVE security advisory", and that CVEs get assigned based on a worst-case evaluation of the bug. As a result, having a CVE in your container barely tells you anything about your actual vulnerability position. In fact, most of the time you will find that having a CVE in some random utility doesn't matter. Most CVEs in system packages don't apply to most of your containers' threat models.
Why not? Because an attacker is very unlikely to be able to use vulnerabilities in these system libraries or utilities. Those utilities are usually not in active use in the first place. Even if they are used, you are not usually in a position to exploit these vulnerabilities as an attacker.
Just as an example, a hypothetical outdated version of grep in one of these containers can hypothetically contain many CVEs. But if your Docker service doesn't use grep, then you would need to manually run grep to be vulnerable. And an attacker that is able to run grep in your Docker container has already owned you - it doesn't make a difference that your grep is vulnerable! This hypothetical vulnerable version of grep therefore makes no difference in the security of your container, despite containing many CVEs.
It's the quality of these vulnerabilities that matters. Can an attacker actually exploit the vulnerabilities to do bad things? The answer for almost all of these CVEs is "no". But that's not really the product that Snyk sells - Snyk sells a product to show you as many vulnerabilities as possible. Any vulnerability scanner company thinks it can provide most business value (and make the most money) by reporting as many vulnerabilities as it can. For sure it can help you to pinpoint those few vulnerabilities that are exploitable, but that's where your own analysis comes in.
I'm not saying there's not a lot to improve in terms of container security. There's a whole bunch to improve there. But focusing on quantities like "amount of CVEs in an image" is not the solution - it's marketing.
This entire depending on a base container for setup and then essentially throwing away everything you get from the package manager is part of the issue. There is no real package management for Docker. Hell, there isn't even an official way to determine if your image needs an upgrade (I wrote a ruby script that gets the latest tags from a docker repo, extracts the ones with numbers and sorts them in order and compares them to what I have running).
Relying on an Alpine/Debian/Ubuntu base helps to get dependencies installed quickly. Docker could have just created their own base distro and some mechanism to track package updates across images, but they did not.
There are guides for making bare containers, they contain nothing .. no ip, grep, bash .. only the bare minimum libraries and requirements to run your service. They are minimal, but incredibly difficult to debug (sysdig still sucks unless you shell out money for enterprise).
I feel like containers are alright, but Docker is a partial dumpster fire. cgroup isolation is good, the crazy way we deal with packages in container systems is not so good.
Sure if you're just checking for base-distro packages for security vulnerabilities, you're going to find security issues that don't apply (e.g. an exploit in libpng even though your container runs nothing that even links to libpng), but it does excuse the whole issue with the way containers are constructed.
I think this space is really open too, for people to find better systems that are also portable: image formats that are easy to build, easy to maintain dependencies for, and easy to run as FreeBSD jails OR Linux cgroup managed containers (Docker for FreeBSD translated images to jails, but it's been unmaintained for years).
I agree the tooling is a tire fire :(
> e.g. an exploit in libpng even though your container runs nothing that even links to libpng
it's a problem because some services or api's could be abused giving the attacker a path to these vulnerable resources and then use that vulnerability regardless if it is used currently by your service.
I like my images to only contain what's absolutely needed to do that job only. It's not so difficult to do, provided people would be willing to architect systems from the ground up, instead of pulling in a complete debian or fedora installation and then removing things (that should be outlawed imho lol). Not only do I get less attack surface but also smaller updates (which again then is incentive to do more often), less complexity, less logs, easier auditabile (now every log file or even log line might give valid clues), faster incident response, easier troubleshooting, sorry for going on and on ...
It's a cultural problem too: where people working in an environment where it's normal to have every command available on a production system (really?), and where there is no barrier to install anything new that is "required" without discussion & peer review (what are we pair programming for?) or where nobody even tracks the dead weight in production or whether configs are locked down?.
I sometimes think many companies lost control over this long ago. [citation needed] :(
I'm not familiar with Docker infrastructure but what is the alternative to "pulling in a complete debian or fedora installation and then removing things"? Compiling your own kernel and doing the whole "Linux From Scratch" thing? Isn't that incredibly time-intensive to do for every single container?
Just have an image with very a minimal user land. Compiling your own kernel is irrelevant because you need the host kernel to run the container, and container images don't contain a kernel.
The busybox image is a good starting point. Take that, then copy your executables and libraries. If you are willing to go further, you can rather easily compile your own busybox with most utilities stripped out. It's not time intensive because you need to do it just once, and it takes just an afternoon to figure out how.
I don't think this is a tooling problem at all.
"The tooling makes it too easy to do it wrong." Compared to shell scripts with package manager invocations? Nobody configures a system with just packages: there are always scripts to call, chroots to create, users and groups to create, passwords to set, firewall policies to update, etc.
There are a bunch of ways to create LXC containers: shell scripts, Docker, ansible. Shell scripts preceded Docker: you can write a function to stop, create an intermediate tarball, and then proceed (so that you don't have to run e.g. debootstrap without a mirror every time you manually test your system build script; so that you can cache build steps that completed successfully).
With Docker images, the correct thing to do is to extend FROM the image you want to use, build the whole thing yourself, and then tag and store your image in a container repository. Neither should you rely upon months-old liveCD images.
"You should just build containers on busybox." So, no package management? A whole ensemble of custom builds to manually maintain (with no AppArmor or SELinux labels)? Maintainers may prefer for distros to field bug reports for their own common build configurations and known-good package sets. Please don't run as root in a container ("because it's only a container that'll get restarted someday"). Busybox is not a sufficient OS distribution.
It's not the tools, it's how people are choosing to use them. They can, could, and should try and use idempotent package management tasks within their container build scripts; but they don't and that's not Bash/Ash/POSIX's fault either.
> With Docker images, the correct thing to do is to extend FROM the image you want to use, build the whole thing yourself, and then tag and store your image in a container repository. Neither should you rely upon months-old liveCD images.
This should rebuild all. There should be an e.g. `apt-get upgrade -y && rm -rf /var/lib/apt/lists` in there somewhere (because base images are usually not totally current (and neither are install ISOs)).
`docker build --no-cache --pull`
You should check that each Dockerfile extends FROM `tag:latest` or the latest version of the tag that you support. Its' not magical, you do have to work it.
Also, IMHO, Docker SHOULD NOT create another Linux distribution.
Tinycoin: A small, horrible cryptocurrency in Python for educational purposes
The 'dumbcoin' jupyter notebook is also a good reference: "Dumbcoin - An educational python implementation of a bitcoin-like blockchain" https://nbviewer.jupyter.org/github/julienr/ipynb_playground...
When does the concept of equilibrium work in economics?
"Modeling stock return distributions with a quantum harmonic oscillator" (2018) https://iopscience.iop.org/article/10.1209/0295-5075/120/380...
> We propose a quantum harmonic oscillator as a model for the market force which draws a stock return from short-run fluctuations to the long-run equilibrium. The stochastic equation governing our model is transformed into a Schrödinger equation, the solution of which features "quantized" eigenfunctions. Consequently, stock returns follow a mixed χ distribution, which describes Gaussian and non-Gaussian features. Analyzing the Financial Times Stock Exchange (FTSE) All Share Index, we demonstrate that our model outperforms traditional stochastic process models, e.g., the geometric Brownian motion and the Heston model, with smaller fitting errors and better goodness-of-fit statistics. In addition, making use of analogy, we provide an economic rationale of the physics concepts such as the eigenstate, eigenenergy, and angular frequency, which sheds light on the relationship between finance and econophysics literature.
"Quantum harmonic oscillator" https://en.wikipedia.org/wiki/Quantum_harmonic_oscillator
The QuantEcon lectures have a few different multiple agent models:
"Rational Expectations Equilibrium" https://lectures.quantecon.org/py/rational_expectations.html
"Markov Perfect Equilibrium" https://lectures.quantecon.org/py/markov_perf.html
"Robust Markov Perfect Equilibrium" https://lectures.quantecon.org/py/rob_markov_perf.html
"Competitive Equilibria of Chang Model" https://lectures.quantecon.org/py/chang_ramsey.html
... "Lectures in Quantitative Economics as Python and Julia Notebooks" https://news.ycombinator.com/item?id=19083479 (data sources (pandas-datareader, pandaSDMX), tools, latex2sympy)
"Econophysics" https://en.wikipedia.org/wiki/Econophysics
> Indeed, as shown by Bruna Ingrao and Giorgio Israel, general equilibrium theory in economics is based on the physical concept of mechanical equilibrium.
Simdjson – Parsing Gigabytes of JSON per Second
> Requirements: […] A processor with AVX2 (i.e., Intel processors starting with the Haswell microarchitecture released 2013, and processors from AMD starting with the Rizen)
Also noteworthy that on Intel at least, using AVX/AVX2 reduces the frequency of the CPU for a while. It can even go below base clock.
iirc, it's complicated. Some instructions don't reduce the frequency; some reduce it a little; some reduce it a lot.
I'm not sure AVX2 is as ubiquitous as the README says: "We assume AVX2 support which is available in all recent mainstream x86 processors produced by AMD and Intel."
I guess "mainstream" is somewhat subjective, but some recent Chromebooks have Celeron processors with no AVX2:
https://us-store.acer.com/chromebook-14-cb3-431-c5fm
https://ark.intel.com/products/91831/Intel-Celeron-Processor...
Because someone wanting 2.2GB/s JSON parsing is deploying to a chromebook...
It doesn't seem that laughable to me to want faster JSON parsing on a Chromebook, given how heavily JSON is used to communicate between webservers and client-side Javascript.
"Faster" meaning faster than Chromebooks do now; 2.2 GB/s may simply be unachievable hardware-wise with these cheap processors. They're kinda slow, so any speed increase would be welcome.
AVX2 also incurs some pretty large penalties for switching between SSE and AVX2. Depending on the amount of time taken in the library between calls, it could be problematic.
This looks mostly applicable to server scenarios where the runtime environment is highly controlled.
There is no real penalty for switching between SSE and AVX2, unless you do it wrong. What are you referring to specifically?
Are you talking about state transition penalties that can occur if you forget a vzeroupper? That's the only thing I'm aware of which kind of matches that.
A faster, more efficient cryptocurrency
Full disclosure: I work on the cryptocurrency in this article, Algorand.
There are a lot of questions and speculation here about this paper and Algorand. I would be happy to try an answer them to your satisfaction. Some context may be helpful first, though. This paper is an innovation about one aspect of our technology. Algorand has a very fast consensus mechanism and can add blocks as quickly as the network can deliver them. We become a victim of our success. The blockchain will grow very rapidly. A terabyte a month is possible. The storage issue associated with our performance can quickly become an issue. The Vault paper is focused on solving this and other storage scaling problems.
The Algorand pure proof-of-stake blockchain and associated cryptocurrency has many novel innovations aside from Vault. It possesses security and scalability properties beyond what any other blockchain technology allows while still being completely decentralized. Our website, algorand.com, and whitepaper are great places to start to learn more.
If you learn best from videos then I suggest you watch Turing award winner and cryptographic pioneer, Silvio Micali, talk about Algorand: https://youtu.be/NykZ-ZSKkxM. He is a captivating speaker and the founder of Algorand.
Are there reasons that e.g. Bitcoin and Ethereum and Stellar could not implement some of these more performant approaches that Algorand [1] and Vault [2] have developed, published, and implemented? Which would require a hard fork?
[2] https://dspace.mit.edu/handle/1721.1/117821
My understanding is that PoS approaches follow normal byzantine agreement theory which states that adversaries cannot control more than 1/3rd of the accounts (or money in the case of algorand). You can also delay new blocks more easily.
Ethereum is scared or that so they are implementing some hybrid form.
Bitcoin is doomed from my perspective, because of the focus on proof of work and the confirmation times. When you realize that algorand is super fast, there is no "confirmation time", and there is no waste in energy to mine, then it is hard to back up any cryptocurrency focusing on proof of work.
And what of decentralized premined chains (with no PoW, no PoS, and far less energy use) that release coins with escrow smart contracts over time such as Ripple and Stellar (and close a new ledger every few seconds)?
> Algorand has a very fast consensus mechanism and can add blocks as quickly as the network can deliver them. We become a victim of our success. The blockchain will grow very rapidly. A terabyte a month is possible. The storage issue associated with our performance can quickly become an issue. The Vault paper is focused on solving this and other storage scaling problems.
What prevents a person from using a chain like IPFS?
Ethereum Casper PoS has been under review for quite some time.
Why isn't all Bitcoin on Lightning Network?
Bitcoin could make bootstrapping faster by choosing a considered-good blockhash and balances, but AFAIU, re-verifying transactions like Bitcoin and derivatives do prevents hash collision attacks that are currently considered infeasible for SHA-256 (especially given a low block size).
There was an analysis somewhere where they calculated the cloud server instance costs of mounting a ~51% attack (which applies to PoW chains) for various blockchains.
Bitcoin is not profitable to mine in places without heavily subsidized dirty/clean energy anymore: energy and Bitcoin commodity costs and prices have intersected. They'll need any of: inexpensive clean energy, more efficient chips, higher speculative value.
Energy arbitrage (grid-scale energy storage) may be more profitable now. We need energy storage in order to reach 100% renewable energy (regardless of floundering policy support).
Ripple is not decentralized. I don't know enough about Stellar to answer.
Bitcoin is software and can easily implement these features but the community is divided and can't reach consensus on anything. Lightning Network as layer two solution is pretty good from what I know.
Ethereum improvements are coming along very slowly and that's good. They're the only blockchain with active engagement by thousands of multiple parties.
Aragaon and Vault's papers might sound good, but who knows how they'll turn out in production.
People argue this all day. There's a lot of FUD.
Ripple only runs ~7% of validator nodes; which is far less centralized control than major Bitcoin mining pools and businesses (who do the deciding in regards to the many Bitcoin hard forks); that's one form of decentralization.
Ripple clients can use their own UNL or use the Ripple-approved UNL.
Ripple is traded on a number of exchanges (though fewer than Bitcoin for certain); that's another form of decentralization.
As an open standard, ILP will further reduce vendor lock in (and increase interoperability between) networks that choose to implement it.
There are forks of Ripple (e.g. Stellar) just like there are forks of Bitcoin and Ethereum.
From https://ripple.com/insights/the-inherently-decentralized-nat... :
> In contrast, the XRP Ledger requires 80 percent of validators on the entire network, over a two-week period, to continuously support a change before it is applied. Of the approximately 150 validators today, Ripple runs only 10. Unlike Bitcoin and Ethereum — where one miner could have 51 percent of the hashing power — each Ripple validator only has one vote in support of an exchange or ordering a transaction.
How does your definition of 'decentralized' differ?
[deleted]
Git-signatures – Multiple PGP signatures for your commits
Is there anything out there that doesn't need GPG? Having a working GPG install is a huge lift for developers.
I take this to mean: apart from the barnacles on GPG, could there be a system which does what GPG does for software development (signing), without the non-functioning web-of-trust of GPG, or the hierarchical system of x509 signing? Something that deals with lost keys, compromised keys/accounts, loss of DNS control, MitMing, MitBing, etc?
I think it is probably in the class of problems where there are no great foolproof solutions. However, I can imagine that techniques like certificate transparency (all signed x509 certificates pushed to a shared log) would be quite useful. Even blockchain techniques. Maybe send someone to check on me, I'm feeling unwell having written that.
> I think it is probably in the class of problems where there are no great foolproof solutions. However, I can imagine that techniques like certificate transparency (all signed x509 certificates pushed to a shared log) would be quite useful.
Securing DNS: "https://news.ycombinator.com/item?id=19181362"
> Certs on the Blockchain: "Can we merge Certificate Transparency with blockchain?" https://news.ycombinator.com/item?id=18961724
> Namecoin (decentralized blockchain DNS): https://en.wikipedia.org/wiki/Namecoin
(Your first link is broken.)
My main problem with blockchain is the excessive energy consumption of PoW. I know there are PoS efforts, but they seem problematical.
I like the recent CertLedger paper: https://eprint.iacr.org/2018/1071.pdf
My mistake. How ironic. Everything depends upon the red wheelbarrow. Here's that link without the trailing ": https://news.ycombinator.com/item?id=19181362
> My main problem with blockchain is the excessive energy consumption of PoW. I know there are PoS efforts, but they seem problematical.
One report said that 78% of Bitcoin energy usage is from renewable sources (many of which would otherwise be curtailed and otherwise unfunded due to flat-to-falling demand for electricity). But PoW really is expensive and hopefully the market will choose less energy-inefficient solutions from the existing and future blockchain solutions while keeping equal or better security assurances.
>> Proof of Work (Bitcoin, ...), Proof of Stake (Ethereum Casper), Proof of Space, Proof of Research (GridCoin, CureCoin,)
The spec should be: DDOS resiliant (without a SPOF), no one entity with control over API and/or database credentials and database backups and the clock, and immutable.
Immutability really cannot be ensured with hashed records that incorporate the previous record's hash as a salt in a blocking centralized database because someone ultimately has root and the clock and all the backups and code vulnerable to e.g. [No]SQL injection; though distributed 'replication' and detection of record modification could be implemented. git push -f may be detected if it's on an already-replicated branch; but git depends upon local timestamps. google/trillian does Merkle trees in a centralized database (for Certificate Transparency).
In quickly reading the git-signatures shell script sources, I wasn't certain whether the git-notes branch with the .gitsigners that are fetched from all n keyservers (with DNS) is also signed?
I also like the "Table 1: Security comparison of Log Based Approaches to Certificate Management" in the CertLedger paper. Others are far more qualified to compare implementations.
You read my mind. I'd love if it could be rooted in a Yubikey.
Decoupling the "signing" and "verifying" parts seem like a good idea. As random Person signs something, how someone else figures out how to go trust that signature is a separate problem.
> I'd love if it could be rooted in a Yubikey.
FIDO2 and Yubico helped develop the new W3C WebAuthn standard: https://en.wikipedia.org/wiki/WebAuthn
But WebAuthn does not solve for WoT or PKI or certificate pinning.
> Decoupling the "signing" and "verifying" parts seem like a good idea. As random Person signs something, how someone else figures out how to go trust that signature is a separate problem.
Someone can probably help with terminology here. There's identification (proving that a person has the key AND that it's their key (biometrics, challenge-response)), signing (using a key to create a cryptographic signature – for the actual data or a reasonably secure cryptographic hash of said data – that could only could have been created with the given key), signature verification (checking that the signature was created by the claimed key for the given data), and then there's trusting that the given key is authorized for a specific purpose (Web of Trust (key-signing parties), PKI, ACME, exchange of symmetric keys over a different channel such as QKD) by e.g. signing a structured document that links cryptographic keys with keys for specific authorized functions and trusting the key(s) used to sign said authorizing document.
Private (e.g. Zero Knowledge) blockchains can be used for key exchange and key rotation. Public blockchains can be used for sharing (high-entropy) key components; also with an optional exchange of money to increase the cost of key compromise attempts.
There's also WKD: "Web Key Directory"; which hosts GPG keys over HTTPS from a .well-known URL for a given user@domain identifier: https://wiki.gnupg.org/WKD
Compared to existing PGP/GPG keyservers, WKD does rely upon HTTPS.
TUF is based on Thandy. TUF: "The Update Framework" does not presume channel security (is designed to withstand channel compromise) https://en.wikipedia.org/wiki/The_Update_Framework_(TUF)
The TUF spec doesn't mention PGP/GPG: https://github.com/theupdateframework/specification/blob/mas...
There's a derivative of TUF for automotive applications called Uptane: https://uptane.github.io
The Bitcoin article on multisignature; 1-of-2, 2-of-2, 2-of-3, 3-of-5, etc.: https://en.bitcoin.it/wiki/Multisignature
Running an LED in reverse could cool future computers
"Near-field photonic cooling through control of the chemical potential of photons" (2019) https://www.nature.com/articles/s41586-019-0918-8
Compounding Knowledge
Buffet’s approach to life is interesting for the same reason an Olympic gymnast is interesting. He has specialized to an extreme and is taking advantage of the rewards of that specialization and natural talent in a unique way.
It’s easy for me to feel shame that I don’t read 8 hours per day, as Warren and Charlie do. Buffett is a phenomenal investor but by all accounts, rather odd. He eats like crap, doesn’t exercise, had a profoundly weird relationship with his wife, and seems addicted to his work at the expense of everything else.
My point is this: his practice of reading income statements and business reports 8 hours per day for 60 years doesn’t make him the kind of person I wish to emulate.
Don’t get me wrong, I respect his levelheadedness toward money, lack of polish wrt PR, and generous philanthropic efforts. There’s a lot of good. But it’s easy to idolize the guy.
Controversial take: Buffett is actually not a great investor in the way people think. Via the float in his insurance companies, he receives a 0% infinite maturity loan to plow into the market. That financial leverage gives him the ability to beat the market year after year - not his own stockpicking prowess.
If you were to start with $1B, then get an extra $2B that you never had to pay back, you too would do quite well holding Coca-Cola and other safe companies with moats. Your returns would be 3x everyone else's.
Buffett himself has tried to explain the impact of float on Berkshire Hathaway but it never seems to sink in with people.
Ill take it even farther: Buffet isn't statistically different than average. He just found a strategy that happened to work, was stubborn enough to stick to it through bad times, and used copious amounts of leverage to juice returns.
A really interesting paper called "Buffet's Alpha" talks about this, and was able to replicate his performance by following a few simple rules. They found that he produced very little actual alpha. To his credit, he seems to have been observant enough to stumble into factors(value, quality, and low beta) before anyone else knew they existed, which is his real strength and contribution.
A summary of that paper from the authors is here: https://www.aqr.com/Insights/Research/Journal-Article/Buffet....
They work at AQR, a firm which is notable for being one of the only successful hedge funds to actually publish meaningful research.
The paper's conclusion is noteworthy:
> The efficient market counterargument is that Buffett was simply lucky. Our findings suggest that Buffett’s success is neither luck nor magic but is a reward for a successful implementation of value and quality exposures that have historically produced high returns. Second, we illustrated how Buffett’s record can be viewed as an expression of the practical implementability of academic factor returns after transaction costs and financing costs. We simulated how investors can try to take advan- tage of similar investment principles. Buffett’s success shows that the high returns of these academic factors are not simply “paper” returns; these returns can be realized in the real world after transaction costs and funding costs, at least by Warren Buffett. Furthermore, Buffett’s exposure to the BAB factor and his unique access to leverage are consistent with the idea that the BAB factor represents reward to the use of leverage.
BTW, AQR funded the initial development of pandas; which now powers tools like alphalens (predictive factor analysis) and pyfolio.
There's your 'compounding knowledge'.
(Days later)
"7 Best Community-Built Value Investing Algorithms Using Fundamentals" https://blog.quantopian.com/fundamentals-contest-winners/
(The Zipline backtesting library also builds upon Pandas)
How can we factor ESG/sustainability reporting into these fundamentals-driven algorithms in order to save the world?
I wonder to what extent Warren Buffet is Warren Buffet because of how he thinks and acts (like all of these non-fiction authors selling books by using his name would have us believe), and to what extent he is the product of media selection bias. -- If you take a large enough group of people who take risky stakes that are large enough (like the world of financial asset management), then one of them is bound to be as successful as Warren Buffet, even if they all behave randomly.
Funnily enough, Buffett actually calculated this in his essay "The Super Investors of Graham and Doddsville"
Basically, advocates of the Free Market Hypotheses believed that it was impossible for anyone to deliberately, repeatedly generate alpha. The market was rational, and success was probabilistically distributed. With a large enough population, you will get Buffett level returns - therefor Buffett is a fluke.
However, Buffett calculated that there weren't enough investors for his success to be a factor of random distribution, plus there were 2 dozen others who followed a similar strategy who also consistently generate alpha
"The Superinvestors of Graham and Doddsville" (1984) https://scholar.google.com/scholar?cluster=17265410477248371...
From https://en.wikipedia.org/wiki/The_Superinvestors_of_Graham-a... :
> The speech and article challenged the idea that equity markets are efficient through a study of nine successful investment funds generating long-term returns above the market index.
This book probably doesn't mention that he's given away over 71% to charity since Y2K. Or that it's really cold and windy and snowy in Omaha; which makes for lots of reading time.
"Warren Buffett and the Interpretation of Financial Statements: The Search for the Company with a Durable Competitive Advantage" (2008) [1], "Buffetology" (1999) [2], and "The Intelligent Investor" (1949, 2009) [3] are more investment-strategy-focused texts.
[1] https://smile.amazon.com/Warren-Buffett-Interpretation-Finan...
[2] https://smile.amazon.com/Buffettology-Previously-Unexplained...
[3] https://smile.amazon.com/Intelligent-Investor-Definitive-Inv...
Value Investing: https://en.wikipedia.org/wiki/Value_investing https://www.investopedia.com/terms/v/valueinvesting.asp
Why CISA Issued Our First Emergency Directive
There are a number of efforts to secure DNS (and SSL/TLS which generally depends upon DNS; and upon which DNS-over-HTTPS depends) and the identity proof systems which are used for record-change authentication and authorization.
Domain registrars can and SHOULD implement multi-factor authentication. https://en.wikipedia.org/wiki/Multi-factor_authentication
Are there domain registrars that support FIDO/U2F or the new W3C WebAuthn spec? https://en.wikipedia.org/wiki/WebAuthn
Credentials and blockchains (and biometrics): https://gist.github.com/westurner/4345987bb29fca700f52163c33...
DNSSEC: https://en.wikipedia.org/wiki/Domain_Name_System_Security_Ex...
ACME / LetsEncrypt certs expire after 3 months (*) and require various proofs of domain ownership: https://en.wikipedia.org/wiki/Automated_Certificate_Manageme...
Certificate Transparency: https://en.wikipedia.org/wiki/Certificate_Transparency
Certs on the Blockchain: "Can we merge Certificate Transparency with blockchain?" https://news.ycombinator.com/item?id=18961724
Namecoin (decentralized blockchain DNS): https://en.wikipedia.org/wiki/Namecoin
DNSCrypt: https://en.wikipedia.org/wiki/DNSCrypt
DNS over HTTPS: https://en.wikipedia.org/wiki/DNS_over_HTTPS
DNS over TLS: https://en.wikipedia.org/wiki/DNS_over_TLS
Chrome will Soon Let You Share Links to a Specific Word or Sentence on a Page
I would like to urge the browser developers/makers to adopt existing proposals which came through open consensus which do precisely cover the same use cases (and more!)
W3C Reference Note on Selectors and States: https://www.w3.org/TR/selectors-states/
It is part of the suite of specs that came through the W3C Web Annotation Working Group: https://www.w3.org/annotation/
More examples in W3C Note Embedding Web Annotations in HTML: https://www.w3.org/TR/annotation-html/
Different kinds of Web resources can combine multiple selectors and states. Here is a simple one using `TextQuoteSelector` handled by the https://dokie.li/ clientside application:
http://csarven.ca/dokieli-rww#selector(type=TextQuoteSelecto...
A screenshot/how-to: https://twitter.com/csarven/status/981924087843950595
"Integration with W3C Web Annotations" https://github.com/bokand/ScrollToTextFragment/issues/4
> It would be great to be able to comment on the linked resource text fragment. W3C Web Annotations [implementations] don't recognize the targetText parameter, so AFAIU comments are then added to the document#fragment and not the specified text fragment. [...]
> Is there a simplified mapping of W3C Web Annotations to URI fragment parameters?
Guidelines for keeping a laboratory notebook
My paper notebooks were always horrific. Writing has always been painful and awkward for me. But I could type like nobody's business thanks to programming. I survived college in the early 80s by being one of the first students to get a word processor.
By the time I was keeping a notebook, my work was generating mountains of computer readable data, source code, and so forth. We managed by agreeing on a format for data files, where the filename referenced a notebook page, and it worked OK.
Today, it's unavoidable that people are going to keep their notes electronically, and there are no perfect solutions for doing this. Wet chemists still like paper notebooks, since it's hard to get a computer close to the bench, and to type while wearing rubber gloves. Academic workers are expected to supply their own computers, and are nervous about getting them damaged or contaminated. Plus, drawing pictures and writing equations on a computer are both awkward.
Computation related fields lend themselves well to purely electronic notebooks, no surprise. Today, a lot of my work fits perfectly in a Jupyter notebook.
Commercial notebook software exists, but it tends to be sold largely for enterprise use, i.e., the solution it solves is how to control lab workers and secure their results, not how to enable independent, creative work.
> Computation related fields lend themselves well to purely electronic notebooks, no surprise. Today, a lot of my work fits perfectly in a Jupyter notebook.
Some notes and ideas regarding Jupyter notebooks as lab notebooks from "Keeping a Lab Notebook [pdf]": https://news.ycombinator.com/item?id=15710815
Superalgos and the Trading Singularity
Though others didn't, you might find this interesting: "Ask HN: Why would anyone share trading algorithms and compare by performance?" https://news.ycombinator.com/item?id=15802785 ( https://westurner.github.io/hnlog/#story-15802785 )
I think there is value in a back-testing module, however, sharing an algo doesn't make sense to me, until unless someone wants to buy mine for an absurd amount.
I think part of the value of sharing knowledge and algorithmic implementations comes from getting feedback from other experts; like peer review and open science and teaching.
Case in point: the first algorithm on this list [1] of community contributed algorithms that were migrated to their new platform is "minimum variance w/ constraint" [2]. Said algorithm showed returns of over 200% as compared with 77% returns from the SPY S&P 500 ETF over the same period, ceteris paribus. In the 69 replies, there are modifications by community members and the original author that exceed 300%.
Working together on open algorithms has positive returns that may exceed advantages of closed algorithmic development without peer review.
[1] https://www.quantopian.com/posts/community-algorithms-migrat...
[2] https://www.quantopian.com/posts/56b6021b3f3b36b519000924
How well does it do in production though and what happens when multiple algos execute the same trades? Does it cause the rest of the algos to adapt and change results? It makes sense to back-test together and work on it, but if it's proven to work, someone will create something to monitor volume on those trades and work against it. I'd be curious to see the same algo do 300% in production, and if so, then my bias would be uncalled for.
> How well does it do in production though and what happens when multiple algos execute the same trades?
Price inflation.
> Does it cause the rest of the algos to adapt and change results?
Trading index ETFs? IDK
> It makes sense to back-test together and work on it, but if it's proven to work, someone will create something to monitor volume on those trades and work against it.
Why does it need to do lots of trades? Is it possible for anyone other than e.g. SEC to review trades by buyer or seller?
> I'd be curious to see the same algo do 300% in production, and if so, then my bias would be uncalled for.
pyfolio does tear sheets with Zipline algos: pyfolio/examples/zipline_algo_example.ipynb https://nbviewer.jupyter.org/github/quantopian/pyfolio/blob/...
alphalens does performance analysis of predictive factors: alphalens/examples/pyfolio_integration.ipynb https://nbviewer.jupyter.org/github/quantopian/alphalens/blo...
awesome-quant lists a bunch of other tools for algos and superalgos: https://github.com/wilsonfreitas/awesome-quant
What's a good platform for paper trading (with e.g. zipline or moonshot algorithms)?
I disagree with price inflation just because everything is hedged, but it may be true.
The too many trades is if there are 300 algos, and I look in the order book and see different orders from different exchanges at the same price point, then I would be adapting to see what's happening, not myself, but there are people who watch order flows.
I don't paper trade, either it works in production with real money or not. Have to get a feel for spreads, commissions, and so on.
Also, in my case, I am hesitant to even use paid services as someone can be watching it, so most my tools are made by me. Good luck with your trading though, if it works out, let me know, I'd pay to use it along side my other trades.
Crunching 200 years of stock, bond, currency and commodity data
Coming from a CS/Econ/Finance background....
Efficiency as “the market fully processes all information” is decidedly untrue.
Efficiency as “Its very hard to arb markets without risk better than a passive strategy” is very true.
Most professional money managers don’t beat the market once you factor in fees. Many private equity funds produce outsize returns, but it’s based on higher risk and taking advantage of tax laws.
Over time, positive performance for mutual funds don’t persist. (If you’re in the top decile performance one year, you’re no more likely to be there next year)
Despite all of this, there are ways people make money. But it’s a small subset of professionals. It’s high frequency traders who find ways to cut the line on mounds of pennies. It inside information from hedge funds. Or earlier access to non public information. But it’s generally not in areas that normal people like you and me can access.
It inside information from hedge funds. Or earlier access to non public information. But it’s generally not in areas that normal people like you and me can access.
Asymmetric information is pretty far from what used to be said about the perfect market and rational actors. It's "there's a sucker born every minute" and "if it seems too good to be true it probably is" economics.
I might be misunderstanding what you're saying here, but are you sure you're right? Fama originally predicated the model of the efficient market (the efficient market hypothesis) on the idea of informational efficiency. Information asymmetry is a fundamental measure involved in the idealized model of an efficient market.
What you're mentioning about rational actors is actually a different topic altogether in economics.
Or have I misunderstood what you're getting at?
I was interested, so I did some research here.
Rational Choice Theory https://en.wikipedia.org/wiki/Rational_choice_theory
Rational Behavior https://www.investopedia.com/terms/r/rational-behavior.asp
> Most mainstream academic economics theories are based on rational choice theory.
> While most conventional economic theories assume rational behavior on the part of consumers and investors, behavioral finance is a field of study that substitutes the idea of “normal” people for perfectly rational ones. It allows for issues of psychology and emotion to enter the equation, understanding that these factors alter the actions of investors, and can lead to decisions that may not appear to be entirely rational or logical in nature. This can include making decisions based primarily on emotion, such as investing in a company for which the investor has positive feelings, even if financial models suggest the investment is not wise.
Behavioral finance https://www.investopedia.com/terms/b/behavioralfinance.asp
Bounded rationality > Relationship to behavioral economics https://en.wikipedia.org/wiki/Bounded_rationality
Perfectly rational decisions can be and are made without perfect information; bounded by the information available at the time. If we all had perfect information, there would be no entropy and no advantage; just lag and delay between credible reports and order entry.
Information asymmetry https://en.wikipedia.org/wiki/Information_asymmetry
Heed these words wisely: What foolish games! Always breaking my heart.
https://deepmind.com/blog/game-theory-insights-asymmetric-mu...
> Asymmetric games also naturally model certain real-world scenarios such as automated auctions where buyers and sellers operate with different motivations. Our results give us new insights into these situations and reveal a surprisingly simple way to analyse them. While our interest is in how this theory applies to the interaction of multiple AI systems, we believe the results could also be of use in economics, evolutionary biology and empirical game theory among others.
https://en.wikipedia.org/wiki/Pareto_efficiency
> A Pareto improvement is a change to a different allocation that makes at least one individual or preference criterion better off without making any other individual or preference criterion worse off, given a certain initial allocation of goods among a set of individuals. An allocation is defined as "Pareto efficient" or "Pareto optimal" when no further Pareto improvements can be made, in which case we are assumed to have reached Pareto optimality.
Which, I think, brings me to equitable availability of maximum superalgo efficiency and limits of real value creation in capital and commodities markets; which'll have to be a topic for a different day.
Show HN: React-Schemaorg: Strongly-Typed Schema.org JSON-LD for React
I have a slightly longer post describing this work and the reasoning behind it on dev.to[1].
[1]: https://dev.to/eyassh/react-schemaorg-strongly-typed-schemaorg-json-ld-for-react-4lhdhttps://dev.to/eyassh/react-schemaorg-strongly-typed-schemao...
Is there a good way to generate JSONschema and thus forms from schema.org RDFS classes and (nested, repeatable) properties?
By JSONschema do you mean [this standard](https://json-schema.org/)? I don't know of a tool that does that yet, the JSON Schema is general enough with allOf/anyOf that it should express schema.org schema as well.
Depends on the purpose here. With this, my goal was to speed up the Write-Update-Debug development loop. Depends on the use case, simply using Google's Structured Data Testing Tool [1] might be a better way to verify schema than JSON-schema?
[1]: https://search.google.com/structured-data/testing-toolThere are a number of tools for generating forms and requisite client and serverside data validations from JSONschema; but I'm not aware of any for RDFS (and thus the schema.org schema [1]). A different use case, for certain.
https://schema.org/docs/developers.html#defs
Definitely a cool area for exploration. I'm not aware of JSON Schema generators from RDFS either.
It should be possible to model the basics (nested structure, repeated structure, defining the "domain" and "range" of a value).
Schema.org definitions however have no conception of "required" values[], however, so some of the cool form validation we see in some of these tools might not apply here.
[*] _Consumers_ of Schema.org JSON-LD or microdata, however, might define their own data requirements. E.g. Google has some concept of required fields, which you can see when using the Structured Data Testing Tool.Consumer Protection Bureau Aims to Roll Back Rules for Payday Lending
From the article:
> The way payday loans work is that payday lenders typically offer small loans to borrowers who promise to pay the loans back by their next paycheck. Interest on the loans can have an annual percentage rate of 390 percent or more, according to a 2013 report by the CFPB. Another bureau report from the following year found that most payday loans — as many as 80 percent — are rolled over into another loan within two weeks. Borrowers often take out eight or more loans a year.
390%
From https://www.npr.org/2019/02/06/691944789/consumer-protection... :
> TARP recovered funds totalling $441.7 billion from $426.4 billion invested, earning a $15.3 billion profit or an annualized rate of return of 0.6% and perhaps a loss when adjusted for inflation.[2][3]
0.6%
Is your point that the US government should get into the payday lending business?
Lectures in Quantitative Economics as Python and Julia Notebooks
It's amazing how we are watching use cases for notebooks and spreadsheets converging. I wonder what the killer feature will be to bring a bigger chunk of the Excel world into a programmatic mindset... Or alternatively, whether we will see notebook UIs embedded in Excel in the future in place of e.g. VBA.
That’s not a bad idea. Spreadsheets are pure functional languages built that use literal spaces instead of namespaces.
Notebooks are cells of logic. You could conceivably change the idea of notebook cells to be an instance of a function that points to raw data and returns raw data.
Perhaps this just Alteryx though
This is brilliant.
I'm picturing the ability to write a Python function with the parameters being just like the parameters in an Excel function. You can drag the cell and have it duplicated throughout a row, updating the parameters to correspond to the rows next to it.
It would exponentially expand the power of excel. I wouldn't be limited to horribly unmaintainable little Excel functions.
VBA can't be used to do that, can it? As far as I understand (and I haven't investigated VBA too much) VBA works on entire spreadsheets.
Essentially, replace the excel formula `=B3-B4` with a Python function `subtract(b3, b4)` where Subtract is defined somewhere more conveniently (in a worksheet wide function definition list?).
This would require a reactive recomputing of cells to be anything like a spreadsheet. > Essentially, replace the excel formula `=B3-B4` with a Python function `subtract(b3, b4)`
as of now jupyter/ipython would not recompute `subtract(b3, b4)` if you change b3 or b4, this has positive and negative (reliance on hidden state and order of execution) effects.
I too would really like something like this, but I think it is pretty far away from where jupiter is now.
You can build something like this with Jupyter today.
> Traitlets is a framework that lets Python classes have attributes with type checking, dynamically calculated default values, and ‘on change’ callbacks. https://traitlets.readthedocs.io/en/stable/
> Traitlet events. Widget properties are IPython traitlets and traitlets are eventful. To handle changes, the observe method of the widget can be used to register a callback https://ipywidgets.readthedocs.io/en/stable/examples/Widget%...
You can definitely build interactive notebooks with Jupyter Notebook and JupyterLab (and ipywidgets or Altair or HoloViews and Bokeh or Plotly for interactive data visualization).
> Qgrid is a Jupyter notebook widget which uses SlickGrid to render pandas DataFrames within a Jupyter notebook. This allows you to explore your DataFrames with intuitive scrolling, sorting, and filtering controls, as well as edit your DataFrames by double clicking cells. https://github.com/quantopian/qgrid
Qgrid's API includes event handler registration: https://qgrid.readthedocs.io/en/latest/
> neuron is a robust application that seamlessly combines the power of Visual Studio Code with the interactivity of Jupyter Notebook. https://marketplace.visualstudio.com/items?itemName=neuron.n...
"Excel team considering Python as scripting language: asking for feedback" (2017) https://news.ycombinator.com/item?id=15927132
OpenOffice Calc ships with Python 2.7 support: https://wiki.openoffice.org/wiki/Python
Procedural scripts written in a general purpose language with named variables (with no UI input except for chart design and persisted parameter changes) are reproducible.
What's a good way to review all of the formulas and VBA and/or Python and data ETL in a spreadsheet?
Is there a way to record a reproducible data transformation script from a sequence of GUI interactions in e.g. OpenRefine or similar?
OpenRefine/OpenRefine/wiki/Jupyter
"Within the Python context, a Python OpenRefine client allows a user to script interactions within a Jupyter notebook against an OpenRefine application instance, essentially as a headless service (although workflows are possible where both notebook-scripted and live interactions take place. https://github.com/OpenRefine/OpenRefine/wiki/Jupyter
Are there data wrangling workflows that are supported by OpenRefine but not Pandas, Dask, or Vaex?
This interesting need to have a closer look, possibly refine can be more efficient? But haven't used it enough to know, just payed around with it a bit. Didn't realise you could combine it with jupyter.
There are undergraduate and graduate courses in each language:
Python version: https://lectures.quantecon.org/py/
Julia version: https://lectures.quantecon.org/jl/
Does anyone else find it strange that there is no real-world data in these notebooks? It's all simulations or abstract problems.
This gives me the sense, personally, that economists aren't interested in making accurate predictions about the world. Other fields would, I think, test their theories against observations.
pandas-datareader can pull data from e.g. FRED, Eurostat, Quandl, World Bank: https://pandas-datareader.readthedocs.io/en/latest/remote_da...
pandaSDMX can pull SDMX data from e.g. ECB, Eurostat, ILO, IMF, OECD, UNSD, UNESCO, World Bank; with requests-cache for caching data requests: https://pandasdmx.readthedocs.io/en/latest/#supported-data-p...
The scikit-learn estimator interface includes a .score() method. "3.3. Model evaluation: quantifying the quality of predictions" https://scikit-learn.org/stable/modules/model_evaluation.htm...
statsmodels also has various functions for statistically testing models: https://www.statsmodels.org/stable/
"latex2sympy parses LaTeX math expressions and converts it into the equivalent SymPy form" and is now merged into SymPy master and callable with sympy.parsing.latex.parse_latex(). It requires antlr-python-runtime to be installed. https://github.com/augustt198/latex2sympy https://github.com/sympy/sympy/pull/13706
IDK what Julia has for economic data retrieval and model scoring / cost functions?
If Software Is Funded from a Public Source, Its Code Should Be Open Source
From the US Digital Services Playbook [1]:
> PLAY 13
> Default to open
> When we collaborate in the open and publish our data publicly, we can improve Government together. By building services more openly and publishing open data, we simplify the public’s access to government services and information, allow the public to contribute easily, and enable reuse by entrepreneurs, nonprofits, other agencies, and the public.
> Checklist
> - Offer users a mechanism to report bugs and issues, and be responsive to these reports
> [...]
> - Ensure that we maintain contractual rights to all custom software developed by third parties in a manner that is publishable and reusable at no cost
> [...]
> - When appropriate, publish source code of projects or components online
> [...]
> Key Questions
> [...]
> - If the codebase has not been released under an open source license, explain why.
> - What components are made available to the public as open source?
> [...]
Apache Arrow 0.12.0
> Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. It also provides computational libraries and zero-copy streaming messaging and interprocess communication. Languages currently supported include C, C++, C#, Go, Java, JavaScript, MATLAB, Python, R, Ruby, and Rust.
Statement on Status of the Consolidated Audit Trail (2018)
U.S. Federal District Court Declared Bitcoin as Legal Money
"Application of FinCEN's Regulations to Persons Administering, Exchanging, or Using Virtual Currencies" (2013) https://www.fincen.gov/resources/statutes-regulations/guidan...
"Legality of bitcoin by country or territory" https://en.wikipedia.org/wiki/Legality_of_bitcoin_by_country...
"Know your customer" https://en.wikipedia.org/wiki/Know_your_customer
"Anti-money-laundering measures by region" https://en.wikipedia.org/wiki/Money_laundering#Anti-money-la...
"Anti-money-laundering measures by region > United States" https://en.wikipedia.org/wiki/Money_laundering#United_States
Post Quantum Crypto Standardization Process – Second Round Candidates Announced
> As the latest step in its program to develop effective defenses, the National Institute of Standards and Technology (NIST) has winnowed the group of potential encryption tools—known as cryptographic algorithms—down to a bracket of 26. These algorithms are the ones NIST mathematicians and computer scientists consider to be the strongest candidates submitted to its Post-Quantum Cryptography Standardization project, whose goal is to create a set of standards for protecting electronic information from attack by the computers of both tomorrow and today.
> “These 26 algorithms are the ones we are considering for potential standardization, and for the next 12 months we are requesting that the cryptography community focus on analyzing their performance,”
Links to the 17 public-key encryption and key-establishment algorithms and 9 digital signature algorithms are here: "Round 2 Submissions" https://csrc.nist.gov/Projects/Post-Quantum-Cryptography/Rou...
"Quantum Algorithm Zoo" has moved to https://quantumalgorithmzoo.org .
Ask HN: How do you evaluate security of OSS before importing?
What tools can I use to evaluate the security posture of an OSS project before I approve its usage with high confidence?
Oddly, whether a project has at least one CVE reported could be interpreted in favor of the project. https://www.cvedetails.com
Do they have a security disclosure policy? A dedicated security mailing list?
Do they pay bounties or participate in e.g Pwn2own?
Do they cryptographically sign releases?
Do they cryptographically sign VCS tags (~releases)? commits? `git tag -s` / `git commit/merge -S` https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work
Downstream packagers do sometimes/often apply additional patches and then sign their release with the repo (and thus system global) GPG key.
Whether they require "Signed-off-by" may indicate that the project has mature controls and possibly a formal code review process requirement. (Look for "Signed-off-by:" in the release branch (`git commit/merge -s/--signoff`)
How have they integrated security review into their [iterative] release workflow?
Is the software formally verified? Are parts of the software implementation or spec formally verified?
Does the system trust the channel? The host? Is it a 'trustless' system?
What are the single points of failure?
How is logging configured? To syslog?
Do they run the app as root in a Docker container? Does it require privileged containers?
If it has to run as root, does it drop privileges at startup?
Does the package have an SELinux or AppArmor policy? (Or does it say e.g. "just set SELinux to permissive mode)
Is there someone you can pay to support the software in an enterprise environment? Open or closed, such contacts basically never accept liability; but if there is an SLA, do you get a pro-rated bill?
As far as indicators of actual software quality:
How much test coverage is there? Line coverage or statement coverage?
Do they run static analysis tools for all pull requests and releases? Dynamic analysis? Fuzzing?
Of course, closed or open source projects may do none or all of these and still be totally secure, insecure, or unsecure.
This is a pretty extensive list. Thanks for sharing!
Ask HN: How can I use my programming skills to support nonprofit organizations?
Lately I've been thinking about doing programming for nonprofits, both because I want to help out with what I'm good at but also to hone my skills and potentially get some open source credit.
So far I've had a hard time finding nonprofit projects where I can just pick up something and start programming. I know about freecodecamp.org, but they force you to go through their courses, and as I already have multiple years of experience as a developer, I feel like that would be a waste of time.
Isn't there a way to contribute to nonprofit organization in a more direct and simple manner like how you would contribute to an open source project on GitHub?
There are lots of project management systems with issue tracking and kanban boards with swimlanes. Because it's unreasonable to expect all volunteers to have a GH account or even understand what GH is for, support for external identity management and SSO may be essential to getting people to actually log in and change their password regularly.
Sidling a nonprofit with custom built software with no other maintainers is not what they need. Build (and pay for development, maintenance, timely security upgrades and security review) or Buy (where is our data? who backs it up? how much does it cost for a month or a few years? Is it open source with a hosted option; so that we can pay a developer to add or fix what we need?)
"Solutions architect" may be a more helpful objective title for what's needed. https://en.wikipedia.org/wiki/Solution_architecture
What are their needs? Marketing, accounting, operations, HR
Marketing: web site, maps service, directions, active social media presence that speaks to their defined audience
Accounting: Revenue and expenses, payroll/benefits/HR, projections, "How can we afford to do more?", handle donations and send receipts for tax purposes, reports to e.g. https://charitynavigator.org/ and infographics for wealth-savvy donors
Operations: Asset inventory, project management, volunteer scheduling
HR: payroll, benefits, volunteer scheduling, training, turnover, retaining selfless and enlightenedly-self-interested volunteers
Create a spreadsheet. Rows: needs/features/business processes. Columns: essential, nice to have, software products and services.
Create another spreadsheet. Rows: APIs. Columns: APIs.
Training: what are the [information systems] processes/workflows/checklists? How can I suggest a change? How do we reach consensus that there's a better way to do this? Is there a wiki? Is there a Q&A system?
"How much did you sink on that? Probably seemed like the best option according to the information available at the time, huh? Do you have a formal systems acquisition process? Who votes according to what type of prepared analysis? How much would it cost to switch? What do we need to do to ETL (extract, transform, and load) into a newer better system?"
When estimating TCO for a nonprofit, turnover is a very real consideration. People move. Chances are, as with most organizations TBH, there's a patchwork of partially-integrated and maybe-integrable systems that it may or may not be more cost-effective and maintainable to replace with a cloud ERP specifically designed for nonprofits.
Who has access rights to manually update which parts of the website? How can we include dynamic ([other] database-backed) content in our website? What is a CMS? What is an ERP? What is a CRM? Are these customers, constituents, or both? When did we last speak with those guys? How can people share our asks with social media networks?
If you're not willing or able to make a long-term commitment, the more responsible thing to do is probably to disclose any conflicts of interest recommend a SaaS solution hosted in a compliant data center.
q="nonprofit erp"
q="nonprofit crm"
q="nonprofit cms" + donation campaign visibility
What time of day are social media posts most likely to get maximum engagement from which segments of our audience? What is our ~ARPU "average revenue per user/follower"?
... As a volunteer and not a FTE, it may be a worthwhile exercise to build a prototype of the new functionality with whatever tools you happen to be familiar with with the expectation that they'll figure out a way to accomplish the same objectives with their existing systems. If that's not possible, there may be a business opportunity: are there other organizations with the same need? Is there a sustainable market for such a solution? You may be building to be acquired.
Ask HN: Steps to forming a company?
Hey guys, I'm leaving my firm very shortly to form a startup.
Does why have a checklist of proper ways to do things?
Ie. 1. Form Chapter C Delaware company with Clerky 2. Hire payroll company x 3. use this company for patents.
any info there?
From "Ask HN: What are your favorite entrepreneurship resources" https://news.ycombinator.com/item?id=15021659 :
> USA Small Business Administration: "10 steps to start your business." https://www.sba.gov/starting-business/how-start-business/10-...
> "Startup Incorporation Checklist: How to bootstrap a Delaware C-corp (or S-corp) with employee(s) in California" https://github.com/leonar15/startup-checklist
> FounderKit has reviews for Products, Services, and Software for founders: https://founderkit.com
... I've heard good things about Gusto for payroll, HR, and benefits through Guideline: https://gusto.com/product/pricing
A Self-Learning, Modern Computer Science Curriculum
Outstanding resource.
jwasham/coding-interview-university also links to a number of also helpful OER resources: https://github.com/jwasham/coding-interview-university
MVP Spec
> The criticism of the MVP approach has led to several new approaches, e.g. the Minimum Viable Experiment MVE[19] or the Minimum Awesome Product MAP[20].
https://en.wikipedia.org/wiki/Minimum_viable_product#Critici...
Can we merge Certificate Transparency with blockchain?
From "REMME – A blockchain-based protocol for issuing X.509 client certificates" https://news.ycombinator.com/item?id=18868540 :
""" In no particular order, there are a number of blockchain PKI (and DNS (!)) proposals and proofs of concept.
"CertLedger: A New PKI Model with Certificate Transparency Based on Blockchain" (2018) https://arxiv.org/pdf/1806.03914 https://scholar.google.com/scholar?q=related:LF9PMeqNOLsJ:sc...
"TABLE 1: Security comparison of Log Based Approaches to Certificate Management" (p.12) lists a number of criteria for blockchain-based PKI implementations:
- Resilient to split-world/MITM attack
- Provides revocation transparency
- Eliminates client certificate validation process
- Eliminates trusted key management
- Preserves client privacy
- Require external auditing
- Monitoring promptness
... These papers also clarify why a highly-replicated decentralized trustless datastore — such as a blockchain — is advantageous for PKI. WoT is not mentioned.
"Blockchain-based Certificate Transparency and Revocation Transparency" (2018) https://fc18.ifca.ai/bitcoin/papers/bitcoin18-final29.pdf
https://scholar.google.com/scholar?q=related:oEsKmJvdn-MJ:sc...
Who can update and revoke which records in a permissioned blockchain (or a plain old database, for that matter)?
Letsencrypt has a model for proving domain control with ACME; which AFAIU depends upon DNS, too. """
TLA references "Certificate Transparency Using Blockchain" (2018) https://eprint.iacr.org/2018/1232.pdf https://scholar.google.com/scholar?q="Certificate+Transparen...
Thanks for the references! The main issue isn't the support and maintenance of a such distributed network, but its integration with current solutions and avoiding centralized middleware services that will weaken the schema described in the documents.
> The main issue isn't the support and maintenance of a such distributed network,
Running a permissioned blockchain is nontrivial. "Just fork XYZ and call it a day" doesn't quite describe the amount of work involved. There's read latency at scale. There's merging things to maintain vendor strings,
> but its integration with current solutions
- Verify issuee identity
- Update (domain/CN/subjectAltName, date) index
- Update cached cert and CRL bundles
- Propagate changes to all clients
> and avoiding centralized middleware services that will weaken the schema described in the documents.
Eventually, a CDN will look desireable. IPFS may fit the bill, IDK?
> Running a permissioned blockchain is nontrivial.
You are right. It's needed a relevant BFT protocol, a lot of work with masternodes community and smart economic system inside. You can look at an example of a such protocol: https://github.com/Remmeauth/remme-core/tree/dev
google/trillian https://github.com/google/trillian
> Trillian is an implementation of the concepts described in the Verifiable Data Structures white paper, which in turn is an extension and generalisation of the ideas which underpin Certificate Transparency.
> Trillian implements a Merkle tree whose contents are served from a data storage layer, to allow scalability to extremely large trees.
Why Don't People Use Formal Methods?
Which universities teach formal methods?
- q=formal+verification https://www.class-central.com/search?q=formal+verification
- q=formal-methods https://www.class-central.com/search?q=formal+methods
Is formal verification a required course or curriculum competency for any Computer Science or Software Engineering / Computer Engineering degree programs?
Is there a certification for formal methods? Something like for engineer-status in other industries?
What are some examples of tools and [OER] resources for teaching and learning formal methods?
- JsCoq
- Jupyter kernel for Coq + nbgrader
- "Inconsistencies, rolling back edits, and keeping track of the document's global state" https://github.com/jupyter/jupyter/issues/333 (jsCoq + hott [+ IJavascript Jupyter kernel], STLC: Simply-Typed Lambda Calculus)
- TDD tests that run FV tools on the spec and the implementation
What are some examples of open source tools for formal verification (that can be integrated with CI to verify the spec AND the implementation)?
What are some examples of formally-proven open source projects?
- "Quark : A Web Browser with a Formally Verified Kernel" (2012) (Coq, Haskell) http://goto.ucsd.edu/quark/
What are some examples of projects using narrow and strong AI to generate perfectly verified software from bad specs that make the customers and stakeholders happy?
From reading though comments here, people don't use formal methods because: cost-prohibitive, inflexibile, perceived as incompatible with agile / iterative methods that are more likely to keep customers who don't know what formal methods are happy, lack of industry-appropriate regulation, and cognitive burden of often-incompatible shorthand notations.
Almost University of New South Wales (Sydney, Australia) alum, and it's the biggest difference between the Software Engineering course and Comp Sci. There are two mandatory formal methods and more additional ones to take. They are really hard, and mean a lot of the SENG cohort ends up graduating as COMPSCI, since they can't hack it and don't want to repeat formal methods.
Steps to a clean dataset with Pandas
To add to the three points in the article:
Data quality https://en.wikipedia.org/wiki/Data_quality
Imputation https://en.wikipedia.org/wiki/Imputation_(statistics)
Feature selection https://en.wikipedia.org/wiki/Feature_selection
datacleaner can drop NaNs, do imputation with "the mode (for categorical variables) or median (for continuous variables) on a column-by-column basis", and encode "non-numerical variables (e.g., categorical variables with strings) with numerical equivalents" with Pandas DataFrames and scikit-learn. https://github.com/rhiever/datacleaner
sklearn-pandas "[maps] DataFrame columns to transformations, which are later recombined into features", and provides "A couple of special transformers that work well with pandas inputs: CategoricalImputer and FunctionTransformer" https://github.com/scikit-learn-contrib/sklearn-pandas
Featuretools https://github.com/Featuretools/featuretools
> Featuretools is a python library for automated feature engineering. [using DFS: Deep Feature Synthesis]
auto-sklearn does feature selection (with e.g. PCA) in a "preprocessing" step; as well as "One-Hot encoding of categorical features, imputation of missing values and the normalization of features or samples" https://automl.github.io/auto-sklearn/master/manual.html#tur...
auto_ml uses "Deep Learning [with Keras and TensorFlow] to learn features for us, and Gradient Boosting [with XGBoost] to turn those features into accurate predictions" https://auto-ml.readthedocs.io/en/latest/deep_learning.html#...
Reahl – A Python-only web framework
I feel bad about projects like these, I really do. I'm sure a lot of effort have been put in and it probably satisfies what OP wants to do. But in the end, no one serious would ever consider using it.
>But in the end, no one serious would ever consider using it.
Why not?
I would say it is the other way around. For any web application project of decent size there is almost always a transpilation step that converts whatever programming language your source code is in, into JavaScript (usually ES5). We also had very successful and widely used projects like GWT for years (GWT was first released in 2006!!!).
Before GWT, there was Wt framework (C++); and then JWt (Java), which do the server and clientsides (with widgets in a tree).
Wt: https://en.wikipedia.org/wiki/Wt_(web_toolkit)
JWt: https://en.m.wikipedia.org/wiki/JWt_(Java_web_toolkit)
GWT: https://en.wikipedia.org/wiki/Google_Web_Toolkit
Now we have Babel, ES YYYY, and faster browser release cycles.
[deleted]
Ask HN: How can you save money while living on poverty level?
I freelance remotely, making roughly $1200 a month as a programmer because I only work 10 hours maximum each week (limited by my contract). I share the apartment with my mom, and It's a section 8 so our rent contributions are based on the income we make. My contribution towards rent is $400 a month.
Although I make more money than my mom (she's of retirement age and only works 1-2 days a week), while I'm looking for more work I want to figure out how to move out and live more independently on only $1200 a month.
I need to live frugally and want to know what I can cut more easily. I own a used car (already paid in full), and pay my own car insurance, electricity, phone and internet. After all that I have about $400 left each month which can be eaten up by going out or some emergency funds.
More recently I had to pay for my new city parking sticker so that's $100 more in expenses this particular month. I would be satisfied just living in a far off town paying the same $400 a month, I feel my dollars would stretch further since I now get 100% more privacy for the same price.
On top of that this job is a contract job so I need to put money aside to pay my own taxes. This $1200 is basically living on poverty level. Any ideas to make saving work? Is it very possible for people in the US to still save while on poverty?
That's not a living wage (or a full time job). There are lots of job search sites.
Spending some time on a good resume / CV / portfolio would probably be a good investment with positive ROI.
Is there a nonprofit that you could volunteer with to increase your hireability during the other 158 hours of the week?
Or an online course with a credential that may or may not have positive ROI as a resume item?
Is there a code school in your city with a "you don't pay unless you land a full time job with a living wage and benefits" guarantee?
What is your strategy for business and career networking?
From https://westurner.github.io/hnlog/#comment-17894632 :
> Personal Finance (budgets, interest, growth, inflation, retirement)
Personal Finance https://en.wikipedia.org/wiki/Personal_finance
Khan Academy > College, careers, and more > Personal finance https://www.khanacademy.org/college-careers-more/personal-fi...
"CS 007: Personal Finance For Engineers" https://cs007.blog
A DNS hijacking wave is targeting companies at an almost unprecedented scale
> The National Cybersecurity and Communications Integration Center issued a statement [1] that encouraged administrators to read the FireEye report. [2]
[1] https://www.us-cert.gov/ncas/current-activity/2019/01/10/DNS...
[2] https://www.fireeye.com/blog/threat-research/2019/01/global-...
Show HN: Generate dank mnemonic seed phrases in the terminal
From https://github.com/lukechilds/doge-seed :
> The first four words will be a randomly generated Doge-like sentence.
The seed phrases are fully valid checksummed BIP39 seeds. They can be used with any cryptocurrency and can be imported into any BIP39 compliant wallet.
> […] However there is a slight reduction in entropy due to the introduction of the doge-isms. A doge seed has about 19.415 fewer bits of entropy than a standard BIP39 seed of equivalent length.
Can you sign a quantum state?
> Abstract. Cryptography with quantum states exhibits a number of surprising and counterintuitive features. In a 2002 work, Barnum et al. argued informally that these strange features should imply that digital signatures for quantum states are impossible [6].
> In this work, we perform the first rigorous study of the problem of signing quantum states. We first show that the intuition of [6] was correct, by proving an impossibility result which rules out even very weak forms of signing quantum states. Essentially, we show that any non-trivial combination of correctness and security requirements results in negligible security.
> This rules out all quantum signature schemes except those which simply measure the state and then sign the outcome using a classical scheme. In other words, only classical signature schemes exist.
> We then show a positive result: it is possible to sign quantum states, provided that they are also encrypted with the public key of the intended recipient. Following classical nomenclature, we call this notion quantum signcryption. Classically, signcryption is only interesting if it provides superior efficiency to simultaneous encryption and signing. Our results imply that, quantumly, it is far more interesting: by the laws of quantum mechanics, it is the only signing method available.
> We develop security definitions for quantum signcryption, ranging from a simple one-time two-user setting, to a chosen-ciphertext-secure many-time multi-user setting. We also give secure constructions based on post-quantum public-key primitives. Along the way, we show that a natural hybrid method of combining classical and quantum schemes can be used to “upgrade” a secure classical scheme to the fully-quantum setting, in a wide range of cryptographic settings including signcryption, authenticated encryption, and chosen-ciphertext security.
"Quantum signcryption"
Lattice Attacks Against Weak ECDSA Signatures in Cryptocurrencies [pdf]
From the paper:
Abstract. In this paper, we compute hundreds of Bitcoin private keys and dozens of Ethereum, Ripple, SSH, and HTTPS private keys by carrying out cryptanalytic attacks against digital signatures contained in public blockchains and Internet-wide scans.
REMME – A blockchain-based protocol for issuing X.509 client certificates
pity it's blockchain based
It's unclear to me why you would want a distributed PKI to authenticate a centralized app. Or maybe it's only for dapps?
In no particular order, there are a number of blockchain PKI (and DNS (!)) proposals and proofs of concept.
"CertLedger: A New PKI Model with Certificate Transparency Based on Blockchain" (2018) https://arxiv.org/pdf/1806.03914 https://scholar.google.com/scholar?q=related:LF9PMeqNOLsJ:sc...
"TABLE 1: Security comparison of Log Based Approaches to Certificate Management" (p.12) lists a number of criteria for blockchain-based PKI implementations:
- Resilient to split-world/MITM attack
- Provides revocation transparency
- Eliminates client certificate validation process
- Eliminates trusted key management
- Preserves client privacy
- Require external auditing
- Monitoring promptness
... These papers also clarify why a highly-replicated decentralized trustless datastore — such as a blockchain — is advantageous for PKI. WoT is not mentioned.
"Blockchain-based Certificate Transparency and Revocation Transparency" (2018) https://fc18.ifca.ai/bitcoin/papers/bitcoin18-final29.pdf
https://scholar.google.com/scholar?q=related:oEsKmJvdn-MJ:sc...
Who can update and revoke which records in a permissioned blockchain (or a plain old database, for that matter)?
Letsencrypt has a model for proving domain control with ACME; which AFAIU depends upon DNS, too.
California grid data is live – solar developers take note
> It looks like California is at least two generations of technology ahead of other states. Let’s hope the rest of us catch up, so that we have a grid that can make an asset out of every building, every battery, and every solar system.
+1. Are there any other states with similar grid data available for optimization; or any plans to require or voluntarily offer such a useful capability?
Why attend predatory colleges in the US?
> Why would people attend predatory colleges?
Why would people make an investment with insufficient ROI (Return on Investment)?
Insufficient information.
College Scorecard [1] is a database with a web interface for finding and comparing schools according to a number of objective criteria. CollegeScorecard launched in 2015. It lists "Average Annual Cost", "Graduation Rate", and "Salary After Attending" on the search results pages. When you review a detail page for an institution, there are many additional statistics; things like: "Typical Total Debt After Graduation" and "Typical Monthly Loan Payment".
The raw data behind CollegeScorecard can be downloaded from [2]. The "data_dictionary" tab of the "Data Dictionary" spreadsheet describes the data schema.
[1] https://collegescorecard.ed.gov
[2] https://collegescorecard.ed.gov/data/
Khan Academy > "College, careers, and more" [3] may be a helpful supplement for funding a full-time college admissions counselor in a secondary education institution
[3] https://www.khanacademy.org/college-careers-more
(I haven't the time to earn 10 academia.stackexchange points in order to earn the prestigious opportunity to contribute this answer to such a forum with threaded comments. In the academic journal system, journals sell academics' work (i.e. schema.org/ScholarlyArticle PDFs, mobile-compatible responsive HTML 5, RDFa, JSON-LD structured data) and keep all of the revenue).
"Because I need money for school! Next question. CPU: College Textbook costs and CPI: All over time t?!"
Ask HN: Data analysis workflow?
What kind of workflow do you employ when designing a data-flow or analyzing data?
Let me give a concrete example. For the past year, I have been selling stuff on the interwebs through two payment processors one of them being PayPal.
The selling process was put together with a bunch of SaaS hooking everything together through webhooks and notifications.
Now I need to step it that control and produce a proper flow to handle sign up, subscription and payment.
Before doing that I'm analyzing and trying to conciliate all transactions to make sure the books are OK and nothing went unseen. There lies the problem. I have data coming from different sources such as databases, excel files, CSV exports and some JSON files.
At first, I started dealing with it by having all the data in CSV files and trying to make sense of them using code and running queries within the code.
As I found holes in the data I had to dig up more data from different sources and it became a pain to continue with code. I now imported everything into Postgres and have been "debugging" with SQL.
As I advanced through the process I had to generate a lot of routines to collect and match data. I also have to keep all the data files around and organized which is very hard to do because I'm all over the place trying to find where the problem is.
How do you handle with it? What kind of workflow? Any best practices or recommendations from people who do this for a living?
Pachyderm may be basically what you're looking for. It does data version control with/for language-agnostic pipelines that don't need to always redo the ETL phase. https://www.pachyderm.io
Dask-ML works with {scikit-learn, xgboost, tensorflow, TPOT,}. ETL is your responsibility. Loading things into parquet format affords a lot of flexibility in terms of (non-SQL) datastores or just efficiently packed files on disk that need to be paged into/over in RAM. http://ml.dask.org/examples/scale-scikit-learn.html
Sklearn.pipeline.Pipeline API: {fit(), transform(), predict(), score(),} https://scikit-learn.org/stable/modules/generated/sklearn.pi...
https://docs.featuretools.com can also minimize ad-hoc boilerplate ETL / feature engineering :
> Featuretools is a framework to perform automated feature engineering. It excels at transforming temporal and relational datasets into feature matrices for machine learning.
The PLoS 10 Simple Rules papers distill a number of best practices:
"Ten Simple Rules for Reproducible Computational Research" http://www.ploscompbiol.org/article/info%3Adoi%2F10.1371%2Fj...
“Ten Simple Rules for Creating a Good Data Management Plan” http://journals.plos.org/ploscompbiol/article?id=10.1371/jou...
In terms of the scientific method, a null hypothesis like "there is no significant relation between the [independent and dependent] variables" may be dangerously unprofessional p-hacking and data dredging; and may result in an overfit model that seems to predict or classify the training and test data (when split with e.g. sklearn.model_selection.train_test_split and a given random seed).
One of these days (in the happy new year!) I'll get around to updating these notes with the aforementioned tools and docs: https://wrdrd.github.io/docs/consulting/data-science#scienti...
IDK what https://kaggle.com/learn has specifically in terms of analysis workflow? Their docker containers have very many tools configured in a reproducible way: https://github.com/Kaggle/docker-python/blob/master/Dockerfi...
Ask HN: What is your favorite open-source job scheduler
Too many business scripts rely on cron(8) to run. Classic cron cannot handle task duration, fail (only with email), same-task piling, linting, ...
So what is your favorite open-source, easy to bundle/deploy job scheduler, that is easy to use, has logging capacity, config file linting, and can handle common use-cases : kill if longer than, limit resources, prevent launching when previous one is nor finished, ...
systemd-crontab-generator may be usable for something like linting classic crontabs? https://github.com/systemd-cron/systemd-cron
Systemd/Timers as a cron replacement: https://wiki.archlinux.org/index.php/Systemd/Timers#As_a_cro...
Celery supports periodic tasks:
> Like with cron, the tasks may overlap if the first task doesn’t complete before the next. If that’s a concern you should use a locking strategy to ensure only one instance can run at a time (see for example Ensuring a task is only executed one at a time).
http://docs.celeryproject.org/en/latest/userguide/periodic-t...
How to Version-Control Jupyter Notebooks
Mentioned in the article: manual nbconvert, nbdime, ReviewNB (currently GitHub only), jupytext.
Jupytext includes a bit of YAML in the e.g. Python/R/Julia/Markdown header. https://github.com/mwouts/jupytext
Huge +1s for both nbdime and jupytext. Excellent tools both.
Really enjoying jupytext. I do a bunch of my training from Jupyter and it has made my workflow better.
Teaching and Learning with Jupyter (A book by Jupyter for Education)
Still under heavy development, but already useful. Add we accept Pull Requests to add items or fix issues. Join us!
A small suggestion.
Since Jupyter Notebook can be used for both programming and documentation, why don't you use Jupyter Notebook itself as the source of your document?
It is actually very easy to setup a Jupyter Notebook driven .ipynb -> .html publishing pipeline with github + a local Jupyter instance
Here is a toy example (for my own github page)
https://ontouchstart.github.io/
https://github.com/ontouchstart/ontouchstart.github.io
The convert script is here (also a Jupyter Notebook)
https://github.com/ontouchstart/ontouchstart.github.io/blob/...
You got the ideas.
BTW, to make the system fully replicable, I use docker for the local Jupyter instance, which can be launched via the Makefile
https://github.com/ontouchstart/ontouchstart.github.io/blob/...
Here is the custom Dockerfile:
https://github.com/ontouchstart/ontouchstart.github.io/blob/...
Margin Notes: Automatic code documentation with recorded examples from runtime
Slightly related are Go examples - they're tests, and documentation at the same time. It'd be nice if someone hooks in a listener to automatically collect examples tho
And Elixir's doctests https://elixir-lang.org/getting-started/mix-otp/docs-tests-a...
And Python's Axe!
I mean doctest: https://docs.python.org/3/library/doctest.html
1. sys.settrace() for {call, return, exception, c_call, c_return, and c_exception}
2. Serialize as/to doctests. Is there a good way to serialize Python objects as Python code?
3. Add doctests to callables' docstrings with AST
Mutation testing tools may have already implemented serialization to doctests but IDK about docstring modification.
... MOSES is an evolutionary algorithm that mutates and simplifies a combo tree until it has built a function with less error for the given input/output pairs.
Time to break academic publishing's stranglehold on research
Add in that the quality of the system is massively massively broken...peer review is about as accurate as the flip of a coin. It does not promote gorund breaking or novel research, it barely (arguably doesn't) even contribute to quality research. I had a colleague recently be told by a journal editor 'we don't publish critiques from junior scholars.' So much for the nature of peer review being entirely driven by the quality of the work.
As one of those academics...I keep getting requests to peer review, I respectfully make clear I don't review for non open source journals anymore. Same with publishing. I'm not tenure-track so am not primarily evaluated based on output.
Publishing is broken, but it is really just part of the broader and even more broken nature of academic research.
What would be a good alternative to peer-review though? Genuinely interested.
Open publishing and commenting would be a good start---having a dialog, like is done at conferences. Older academic journal articles (pre 1900) read much more like discussions than like the hundred dollar word vomits of modern academic publishing. The broken incentives are at the core of this rotten fruit, though. Just making journals open isn't enough.
We have (almost) open publishing and open commenting. Did that improve anything?
There's open commenting? I've never seen the back and forth of the review process be published. It should be published.
https://hypothes.is supports threaded comments on anything with a URI; including PDFs and specific sentences or figures thereof. All you have to do is register an account and install the browser extension or include the JS in the HTML.
It's based on open standards and an open platform.
W3C Web Annotations: http://w3.org/annotation
About Hypothesis: https://web.hypothes.is/about/
Ask HN: How can I learn to read mathematical notation?
There are a lot of fields I'm interested in, such as machine learning, but I struggle to understand how they work as most resources I come across are full of complex mathematical notation that I never learned how to read in school or University.
How do you learn to read this stuff? I'm frequently stumped by an academic paper or book that I just can't understand due to mathematical notation that I simply cannot read.
https://en.wikipedia.org/wiki/List_of_logic_symbols
https://en.wikipedia.org/wiki/Table_of_mathematical_symbols
These might help a bit.
But as someone with similar problems, I'm beginning to think there's no real solution other than thousands of hours of studying.
There are a number of Wikipedia pages which catalog various uses of symbols for various disciplines:
Outline_of_mathematics#Mathematical_notation https://en.wikipedia.org/wiki/Outline_of_mathematics#Mathema...
List_of_mathematical_symbols https://en.wikipedia.org/wiki/List_of_mathematical_symbols
List_of_mathematical_symbols_by_subject https://en.wikipedia.org/wiki/List_of_mathematical_symbols_b...
Greek_letters_used_in_mathematics,_science,_and_engineering https://en.wikipedia.org/wiki/Greek_letters_used_in_mathemat...
Latin_letters_used_in_mathematics https://en.wikipedia.org/wiki/Latin_letters_used_in_mathemat...
For learning the names of symbols (and maybe also their meaning as conventially utilized in a particular field at a particular time in history), spaced repetition with flashcards with a tool like Anki may be helpful.
For typesetting, e.g. Jupyter Notebook uses MathJax to render LaTeX with JS.
latex2sympy may also be helpful for learning notation.
… data-science#mathematical-notation https://wrdrd.github.io/docs/consulting/data-science#mathema...
New law lets you defer capital gains taxes by investing in opportunity zones
I believe the way that the program works is you can defer taxes from your original capital gains (and the cost basis gets increased so your deferred taxes are less than you would pay otherwise), reinvest them in an "opportunity zone", and not pay capital gains on your investment on the opportunity zone, if you hold it long enough
e.g. Bob bought Apple stock for $50 a share back in the day, and sells it for $200/share. He defers his taxes until 2026. Instead of paying capital gains tax on $150/share, the cost basis is adjusted by 15% so in addition to benefiting from the time value of money, future Bob will only be taxed on $142.50 of capital gains. Bob can buy a house in an "opportunity zone" (from scrolling around the embedded map, there's plenty of million dollar+ houses in these areas. There's also lots of sports teams and stadiums in these areas, so maybe Bob buys an NFL team or a parking lot next to their stadium), rent it out for 10 years, sell it, and not have to pay any capital gains tax on the appreciation. Definitely not a bad deal for him!
Yes its more akin to a 1031 exchange, but opened up so that non-real estate capital gains are eligible.
Is it just capital gains? Wondering if it applies to any other forms of active or passive income.
How are profits from these investments treated?
Can you "swap til you drop" like with a 1031 exchange?
> Is it just capital gains? Wondering if it applies to any other forms of active or passive income.
I would also like some information about this.
+1 for investing in distressed areas; self-nominated with intent or otherwise.
If it's capital gains only, -1 on requiring sale of capital assets in order to be sufficiently incentivized. (Because then the opportunity to tax-advantagedly invest in Opportunity Zones is denied to persons without assets to liquidate; i.e. unequal opportunity).
How to Write a Technical Paper [pdf]
A few years ago i found a great version of a similar piece that proposed something like a 'sandwich' model for each section, and the work as a whole. A sandwich layer was a hook (something interesting), the meat (what you actually did), and a summary.
I failed to save it and i haven't been able to dig it up again, but I liked the idea. The paper was written in the style it described, as well.
5 paragraph essay? https://en.wikipedia.org/wiki/Five-paragraph_essay
> The five-paragraph essay is a format of essay having five paragraphs: one introductory paragraph, three body paragraphs with support and development, and one concluding paragraph. Because of this structure, it is also known as a hamburger essay, one three one, or a three-tier essay.
The digraph presented in the OP really is a great approach, IMHO:
## Introduction
## Related Work, System Model, Problem Statement
## Your Solution
## Analysis
## Simulation, Experimentation
## Conclusion
... "Elements of the scientific method" https://en.wikipedia.org/wiki/Scientific_method#Elements_of_...
no, it was on arxiv somewhere. It was written as a journal article.
edit: AH! it was linked upthread on bioarxiv.
JSON-LD 1.0: A JSON-Based Serialization for Linked Data
JSON-LD 1.1 https://www.w3.org/TR/json-ld11/
"Changes since 1.0 Recommendation of 16 January 2014" https://www.w3.org/TR/json-ld11/#changes-from-10
Jeff Hawkins Is Finally Ready to Explain His Brain Research
Cortical column: https://en.wikipedia.org/wiki/Cortical_column
> In the neocortex 6 layers can be recognized although many regions lack one or more layers, fewer layers are present in the archipallium and the paleopallium.
What this means in terms of optimal artificial neural network architecture and parameters will be interesting to learn about; in regards to logic, reasoning, and inference.
According to "Cliques of Neurons Bound into Cavities Provide a Missing Link between Structure and Function" https://www.frontiersin.org/articles/10.3389/fncom.2017.0004... , the human brain appears to be [at most] 11-dimensional (11D); in terms of algebraic topology https://en.wikipedia.org/wiki/Algebraic_topology
Relatedly,
"Study shows how memories ripple through the brain" https://www.ninds.nih.gov/News-Events/News-and-Press-Release...
> The [NeuroGrid] team was also surprised to find that the ripples in the association neocortex and hippocampus occurred at the same time, suggesting the two regions were communicating as the rats slept. Because the association neocortex is thought to be a storage location for memories, the researchers theorized that this neural dialogue could help the brain retain information.
Re: Topological graph theory [1], is it possible to embed a graph on a space filling curve [2] (such as a Hilbert R-tree [3])?
[1] https://en.wikipedia.org/wiki/Topological_graph_theory
[2] https://en.wikipedia.org/wiki/Space-filling_curve
[3] https://en.wikipedia.org/wiki/Hilbert_R-tree
[4] https://github.com/bup/bup (git packfiles)
Interstellar Visitor Found to Be Unlike a Comet or an Asteroid
It's too bad we're not ready to launch probes to visit and explore such transient objects at a moment's notice.
I think it would make sense to plan a probe mission where the probe would be put into storage to stand by for these kind of events. It would still be a challenge and might be impossible to find a launch window and reach the passing object in time, but it would be worth trying.
Just imagine how tragic it would be, if an ancient artifact of an alien civilization drifts by earth and we don't manage to have at least a look at it.
Wouldn’t such an object be emitting radio signals that earth could receive?
Not if it's something like another civilization's Tesla Roadster.
> Not if it's something like another civilization's Tesla Roadster.
'Oumuamua is red and headed toward Pegasus (the winged horse) after a very long journey starting longtime in spacetime ago. It is wildly tumbling off-kilter and potentially creating a magnetic field that would be useful for interplanetary spacetravel.
They're probably pointing us to somewhere else from somewhere else.
If this is any indication of the state of another civilization's advanced physics, and it missed us by a wide margin, they're probably laughing at our energy and water markets; and indicating that we should be focused on asteroid impact avoidance (and then we will really laugh about rockets and red electromagnetic kinetic energy machines and asteroid mining). https://en.wikipedia.org/wiki/Asteroid_impact_avoidance
"Amateurs"
[We watch it fly by, heads all turning]
Maybe it would've been better to have put alone starman in the passenger seat or two starpeoples total?
Given the skull shape of October 2015 TB145 [1] (due to return in November 2018), maybe 'Oumuamua [2] is a pathology of Mars and an acknowledgement of our spacefaring intentions? Red, subsurface water, disrupted magnetic field.
[1] https://en.wikipedia.org/wiki/2015_TB145
[2] https://en.wikipedia.org/wiki/%CA%BBOumuamua
In regards to a red, unshielded, earth vehicle floating in solar orbit with a suited anthropomorphic creature whose head is too big for the windshield:
"What happened here?"
Publishing more data behind our reporting
Publishing raw data itself is definitely a good start but there also needs to be a push towards a standardized way of sharing data along with it's lineage (dependent sources, experimental design/generation process, metadata, graph relationship of other uses, etc.).
> Publishing raw data itself is definitely a good start but there also needs to be a push towards a standardized way of sharing data along with it's lineage (dependent sources, experimental design/generation process, metadata, graph relationship of other uses, etc.).
Linked Data based on URIs is reusable. ( https://5stardata.info )
The Schema.org Health and Life Sciences extension is ahead of the game here, IMHO. MedicalObservationalStudy and MedicalTrial are subclasses of https://schema.org/MedicalStudy . {DoubleBlindedTrial, InternationalTrial, MultiCenterTrial, OpenTrial, PlaceboControlledTrial, RandomizedTrial, SingleBlindedTrial, SingleCenterTrial, and TripleBlindedTrial} are subclasses of schema.org/MedicalTrial.
A schema.org/MedicalScholarlyArticle (a subclass of https://schema.org/ScholarlyArticle ) can have a https://schema.org/Dataset. https://schema.org/hasPart is the inverse of https://schema.org/isPartOf .
More structured predicates which indicate the degree to which evidence supports/confirms or disproves current and other hypotheses (according to a particular Person or Persons on a given date and time; given a level of scrutiny of the given information) are needed.
In regards to epistemology, there was some work on Fact Checking ( e.g. https://schema.org/ClaimReview ) in recent times. To quote myself here, from https://news.ycombinator.com/item?id=15528824 :
> In terms of verifying (or validating) subjective opinions, correlational observations, and inferences of causal relations; #LinkedMetaAnalyses of documents (notebooks) containing structured links to their data as premises would be ideal. Unfortunately, PDF is not very helpful in accomplishing that objective (in addition to being a terrible format for review with screen reader and mobile devices): I think HTML with RDFa (and/or CSVW JSONLD) is our best hope of making at least partially automated verification of meta analyses a reality.
"#LinkedReproducibility"; "#LinkedMetaAnalyses", "#StudyGraph"
CSV 1.1 – CSV Evolved (for Humans)
Well, if you want to improve tabular data formats:
1. Add a version identifier / content-type on the first line!
2. Create a formal grammar for this CSV format
3. Specify preferred character-encoding
4. Provide some tooling (validation, CSV 1.1 => HTML, CSV => Excel)
5. Add the option to specify column type (string, int, date)
6. Specify ISO-8601 as the preferred date format
7. Allow 'reheading' the columns in the file itself. This is useful in streaming data.
8. Specify the format of the newlines.
CSVW: CSV on the Web https://w3c.github.io/csvw/
"CSV on the Web: A Primer" http://www.w3.org/TR/tabular-data-primer/
"Model for Tabular Data and Metadata on the Web" http://www.w3.org/TR/tabular-data-model/
"Generating JSON from Tabular Data on the Web" (csv2json) http://www.w3.org/TR/csv2json/
"Generating RDF from Tabular Data on the Web" (csv2rdf) http://www.w3.org/TR/csv2rdf/
...
N. Allow authors to (1) specify how many header rows are metadata and (2) what each row is. For example: 7 metadata header rows: {column label, property URI [path], datatype URI, unit URI, accuracy, precision, significant figures}
With URIs, we can merge, join, and concatenate data (when e.g. study control URIs for e.g. single/double/triple blinding/masking indicate that the https://schema.org/Dataset meets meta-analysis inclusion criteria).
"#LinkedReproducibility"; "#LinkedMetaAnalyses"
Ask HN: Which plants can be planted indoors and easily maintained?
Chlorophytum comosum (spider plants) are good air-filtering houseplants that are also easy to take starts of: https://en.wikipedia.org/wiki/Chlorophytum_comosum
Houseplant: https://en.wikipedia.org/wiki/Houseplant
Graduate Student Solves Quantum Verification Problem
"Classical Verification of Quantum Computations" Mahadev. (2018)
https://arxiv.org/abs/1804.01082
https://www.arxiv-vanity.com/papers/1804.01082/
https://scholar.google.com/scholar?cluster=10138991277567750...
The down side to wind power
>To estimate the impacts of wind power, Keith and Miller established a baseline for the 2012‒2014 U.S. climate using a standard weather-forecasting model. Then, they covered one-third of the continental U.S. with enough wind turbines to meet present-day U.S. electricity demand. The researchers found this scenario would warm the surface temperature of the continental U.S. by 0.24 degrees Celsius, with the largest changes occurring at night when surface temperatures increased by up to 1.5 degrees. This warming is the result of wind turbines actively mixing the atmosphere near the ground and aloft while simultaneously extracting from the atmosphere’s motion.
I am confused: How does the warming work exactly and is this actually a global climate effect? Because this part of the article makes it sound to me as if it's just a very localised change of temperature caused by the exchange of different air layers, which can't be right? Because you couldn't really compare that to climate change on a global scale.
The example is clearly hypothetical only. We're never going to cover one third of the continental US with wind turbines.
The more important information to me is that neither wind nor solar have the power density that has been claimed.
For wind, we found that the average power density — meaning the rate of energy generation divided by the encompassing area of the wind plant — was up to 100 times lower than estimates by some leading energy experts
...
For solar energy, the average power density (measured in watts per meter squared) is 10 times higher than wind power, but also much lower than estimates by leading energy experts.
Then you have the separate problem that the wind doesn't always blow and the sun doesn't always shine, so you need a huge storage infrastructure (batteries, presumably) alongside the wind and solar generating infrastructure.
IMO nuclear is the only realistic alternative to coal to provide reliable, zero-emission "base load" power generation. Wind and solar could make sense in some use cases but not in general.
> IMO nuclear is the only realistic alternative to coal to provide reliable, zero-emission "base load" power generation. Wind and solar could make sense in some use cases but not in general.
How much heat energy does a reactor with n meters of concrete around it, located on a water supply in order to use water in an open closed loop, protected with national security resources, waste into the environment?
I'd be interested to see which power sources the authors of this study would choose as a control for these just sensational stats.
From https://news.ycombinator.com/item?id=17806589 :
> Canada (2030), France (2021), and the UK (2025) are all working to entirely phase out coal-fired power plants for very good reasons (such as neonatal health).
Would you burn a charcoal grill in an enclosed space like a garage? No.
Thermodynamics of Computation Wiki
"Quantum knowledge cools computers: New understanding of entropy" (2011) https://www.sciencedaily.com/releases/2011/06/110601134300.h...
> The new study revisits Landauer's principle for cases when the values of the bits to be deleted may be known. When the memory content is known, it should be possible to delete the bits in such a manner that it is theoretically possible to re-create them. It has previously been shown that such reversible deletion would generate no heat. In the new paper, the researchers go a step further. They show that when the bits to be deleted are quantum-mechanically entangled with the state of an observer, then the observer could even withdraw heat from the system while deleting the bits. Entanglement links the observer's state to that of the computer in such a way that they know more about the memory than is possible in classical physics.
"The thermodynamic meaning of negative entropy" (2011) https://www.nature.com/articles/nature10123
Landauer's principle: https://en.wikipedia.org/wiki/Landauer%27s_principle
"Thin film converts heat from electronics into energy" (2018) http://news.berkeley.edu/2018/04/16/thin-film-converts-heat-...
> This study reports new records for pyroelectric energy conversion energy density (1.06 Joules per cubic centimeter), power density (526 Watts per cubic centimeter) and efficiency (19 percent of Carnot efficiency, which is the standard unit of measurement for the efficiency of a heat engine).
"Pyroelectric energy conversion with large energy and power density in relaxor ferroelectric thin films" (2018) https://www.nature.com/articles/s41563-018-0059-8
Carnot heat engine > Carnot cycle, Carnot's theorem, "Real heat engines": https://en.wikipedia.org/wiki/Carnot_heat_engine
Carnot's theorem > Applicability to fuel cells and batteries: https://en.wikipedia.org/wiki/Carnot%27s_theorem_(thermodyna...
> Since fuel cells and batteries can generate useful power when all components of the system are at the same temperature [...], they are clearly not limited by Carnot's theorem, which states that no power can be generated when [...]. This is because Carnot's theorem applies to engines converting thermal energy to work, whereas fuel cells and batteries instead convert chemical energy to work.[6] Nevertheless, the second law of thermodynamics still provides restrictions on fuel cell and battery energy conversion
[deleted]
Is there enough heat energy from a datacenter to -- rather than heating oceans (which can result in tropical storms) -- turn a turbine (to convert heat energy back into electrical energy)?
Is there a statistic which captures the amount of heat energy discharged into ocean/river/lake water? "100% clean energy with PPAs (Power Purchase Agreements)" while bleeding energy into the oceans isn't quite representative of the total system.
"How to Reuse Waste Heat from Data Centers Intelligently" (2016) https://www.datacenterknowledge.com/archives/2016/05/10/how-...
> There are two big issues with data center waste heat reuse: the relatively low temperatures involved and the difficulty of transporting heat. Many of the reuse applications to date have used the low-grade server exhaust heat in an application physically adjacent to the data center, such as a greenhouse or swimming pool in the building next door. This is reasonable given the relatively low temperatures of data center return air, usually between 28° and 35°C (80-95°F), and the difficulty in moving heat around. Moving heat energy frequently requires insulated ducting or plumbing instead of cheap, convenient electrical cables. Trenching and installation to run a hot water pipe from a data center to a heat user may cost as much as $600 per linear foot. Just the piping to share heat with a facility one-quarter mile away might add $750,000 or more to a data center construction project. There’s currently not much that can be done to reduce this cost.
> To address the low-temperature issue, some data center operators have started using heat pumps to increase the temperature of waste heat, making the thermal energy much more valuable, and marketable. Waste heat coming out of heat pumps at temperatures in the range of 55° to 70°C (130-160°F) can be transferred to a liquid medium for easier transport and can be used in district heating, commercial laundry, industrial process heat, and many more. There are even High Temperature (HT) and Very High Temperature (VHT) heat pumps capable of moving low-grade data center heat up to 140°C.
Heat Pump: https://en.wikipedia.org/wiki/Heat_pump
"Data Centers That Recycle Waste Heat" https://www.datacenterknowledge.com/data-centers-that-recycl...
Why Do Computers Use So Much Energy?
> Also, to foster research on this topic we have built a wiki, combining lists of papers, websites, events pages, etc. We highly encourage people to visit it, sign up, and start improving it; the more scientists get involved, from the more fields, the better!
Thermodynamics of Computation Wiki https://centre.santafe.edu/thermocomp/Santa_Fe_Institute_Col...
Justice Department Sues to Stop California Net Neutrality Law
Sorry if I get some basic understanding of the law wrong but...
Isn't this the same thing as regulating car emissions? Doesn't 822 only apply to providers in the state itself? Wouldn't it be that the telecoms are welcome to engage in another method of end-customer billing in other states?
What am I missing?
> Like California’s auto emissions laws that forced automakers to adopt the standards for all production, the state’s new net neutrality rules could push broadband providers to apply the same rules to other states.
I think you're right, the motives are very similar.
> Attorney General Jeff Sessions said that California’s net neutrality law was illegal because Congress granted the federal government, through the F.C.C., the sole authority to create rules for broadband internet providers. “States do not regulate interstate commerce — the federal government does,” Mr. Sessions said in a statement.
I thought Republicans were pro-states rights and limited government? How does their position on this jive with their ideology?
Expansion of federal jurisdiction under the Commerce Clause is an egregious violation of Constitutional law.
Does the federal government have the enumerated right under the Commerce Clause to, for example, ban football for anyone that doesn't have a disability? No!
Was the Commerce Clause sufficient authorization for Federal prohibition of alcohol? No! An Amendment to the Constitution was necessary. And, Federal Alcohol and the unequal necessary State Alcohol prohibitions miserably failed to achieve the intended outcomes.
Where is the limit? How can they claim to support a states' rights, limited government position while expanding jurisdiction under the Interstate Commerce Clause? "Substantially affecting" interstate commerce is a very slippery slope.
Furthermore, de-classification from Title II did effectively - as the current administration's FCC very clearly argued (in favor of special interests over those of the majority) - relieve the FCC of authority to regulate ISPs: they claimed that it's FTC's job and now they're claiming it's their job.
Without Title II classification, FCC has no authority to preempt state net neutrality regulation. California and Washington have the right to regulate ISPs within their respective states.
Outrageous!
Limited government: https://en.wikipedia.org/wiki/Limited_government
States' rights: https://en.wikipedia.org/wiki/States%27_rights
[Interstate] Commerce Clause: https://en.wikipedia.org/wiki/Commerce_Clause
Net neutrality in the United States > Repeal of net neutrality policy: https://en.m.wikipedia.org/wiki/Net_neutrality_in_the_United...
The limit is established, and constantly reevaulated by the Supreme Court. For example it was held that gun control cannot be done through the Commerce Clause.
Car emissions and ISPs are different. As ISPs are very much perfect examples of truly local things (they need to reach your devices with EM signals either via cables or air radio), the Federal government might try to argue that the net neutrality regulation of California affects the whole economy substantially, because it allows too much interstate competition due to the lack of bundling/throttling by ISPs.
Similarly, the problem with car emissions might be that requiring thing at the time of sale affects what kind of cars are sold to CA.
Is the Commerce Clause too vague? Yes. Is there a quick and sane way to fix it? I see none. Is it at least applied consistently? Well, sort of. But we shall see.
ISPs are the very opposite of local, as the only reason I have an ISP is to deliver bits from the rest of the world. Of course, the FCC doesn't seem to understand that...
To summarize the points made in [1]: products can be sold across state lines, internet service sold in one state cannot be sold across state lines.
[1] https://news.ycombinator.com/item?id=18111651
In my opinion, the court has significantly erred in redefining interstate commerce to include (1) intrastate-only-commerce; and (2) non-commerce (i.e. locally grown and unsold wheat)
Furthermore - and this is a bit off topic - unalienable natural rights (Equality, Life, Liberty, and pursuit of Happiness) are of higher precedence. I mention this because this is yet another case where the court will be interpreting the boundary between State and Federal rights; and it's very clear that the founders intended for the powers of the federal government to be limited -- certainly not something that the Commerce Clause should be interpreted to supersede.
What penalties and civil fines are appropriate for States or executive branch departments that violate the Constitution; for failure to uphold Oaths to uphold the Constitution?
The problem is, someone has to interpret what kind of economy the Founders intended.
Is it okay if a State opts to withdraw from the interstate market for wheat? Because without power to meddle with intra-state production, consumption and transactions, it's entirely possible.
White House Drafts Order to Probe Google, Facebook Practices
This sounds like they want the equal representation policies that the Republican Party got rolled back in the 80s (ruled unconstitutional iirc). It’s what allowed the rise of partisan “news”. It seems like any “equal exposure” policies would hit the same issues.
That said the primary “imbalanced exposure” seems to be due to evicting people who simply spend their time attacking minorities, attack equal rights, and promoting violence towards anyone that they dislike. For whatever reason the Republican Party seems to have decided that those people represent “conservative” views that private companies should have to support.
i. e. : People who are exercising their right to free speech.
'Right to free speech' does not exist outside the government. It never has, unless there's an amendment to the first amendment that no one is telling us about..
Repeating what I said in an earlier comment....they were able to grow to the size they have become because they are exempted from liable laws under safe harbor. The argument for that was that they were neutral platforms. They no longer are so they either need to remove the protections or be subject to the first amendment but they should not be able to have it both ways.....and I don't think there is case law to back this one way or the other yet...
> they were able to grow to the size they have become because they are exempted from liable laws under safe harbor
This was not a selective protection. When the government grants limited resources like electromagnetic spectrum and right of way, they're not directly making a monopoly, but the FCC does then claim right to regulate speech.
In the interest of fairness, the FCC classed telecommunication service providers as common carriers; thus authorizing FCC to pass net neutrality protections which require equal prioritization of internet traffic. (No blocking, No throttling, No paid prioritization). The current administration doesn't feel that that's fair, and so they've moved to dismantle said "burdensome regulations".
The current administration is now apparently attempting to argue that information service providers - which are all equally granted safe harbor and obligated to comply with DMCA - have no right to take down abuse and harassment because anti-trust monopoly therefore Freedom of Speech doesn't apply to these corporation persons.
Selective bias, indeed! Broadcast TV and Radio are subject to different rules than Cable (non-broadcast) TV.
Other regimes have attempted to argue that the government has the right to dictate the media as well.
Taking down abuse and harassment is necessary and well within the rights of a person and a corporation in the United States. Taking down certain content is now legally required within 24 hours of notice from the government in the EU.
Where is the line between a media conglomerate that produces news entertainment and an information service provider? If there is none, and the government has the right to regulate "equal time" on non-granted-spectrum media outlets, future administrations could force ConservativeNewsOutletZ and LiberalNewsOutletZ to carry specific non-emergency content, to host abusive and offense rhetoric, and to be sued for being forced to do so because no safe harbor.
Can anyone find the story of how the GOP strongarmed and intimidated Facebook into "equal time" (and then we were all shoved full of apparently Russian conservative "fake news" propaganda) before the most recent election where the GOP won older radio, TV, and print voters and young people didn't vote because it appeared to be unnecessary?
Meanwhile, the current administration rolled back the "burdensome regulation" that was to prevent ISPs from selling complete internet usage history; regardless of age.
Maybe there's an exercise that would be helpful for understanding the "corporate media filter" and the "social media filter"?
You, having no money -- while watching corporate profits soar and income inequality grow to unprecedented heights -- will choose to take a job that requires you to judge whether thousands of reported pieces of content a day are abusive, harassing, making specific threats, inciting specific destructive acts, recruiting for hate groups, depicting abuse; or just good 'ol political disagreement over issues, values, and the appropriate role of the punishing and/or nurturing state. You will do this for weeks or months, because that's your best option, because nobody else is standing in the mirror behind these people who haven't learned to respectfully disagree over facts and data (evidence).
Next, you will plan segments of content time interspersed with ads paid for by people who are trying to sell their products, grow their businesses, and reach people. You will use a limited amount of our limited electromagnetic spectrum which the government has sold your corporate overlords for a limited period of time, contingent upon your adherence to specific and subjective standards of decency as codified in the stated regulations.
In both cases, your objective is to maximize profit for shareholders.
Your target audiences may vary from undefined (everyone watching), to people who only want to review fun things that they agree with in their safe little microcosm of the world, to people who know how to find statistics like corporate profits, personal savings rate, infant morality, healthcare costs per capita, and other Indicators identified as relevant to the Targets and Goals found in the UN Sustainable Development Goals (Global Goals Indicators).
Do you control what the audience shares?
Ask HN: Books about applying the open source model to society
I've been thinking for some time now that as productivity keeps growing, not all people will need to work any more. Society will eventually start to resemble an open source project where a few core contributors do the real work (and get to decide the direction), some others help around, and the majority of people just benefit without having to do anything. I'm wondering if any books have been written to explore this concept further?
> I've been thinking for some time now that as productivity keeps growing, not all people will need to work any more.
How much energy do autotrophs and heterotrophs need to thrive?
"But then we'll be rewarding laziness!"
Some people do enjoy the work they've chosen to do. We enjoy the benefits of upward mobility here in the US; the land of opportunity.
Why would I fully retire at 65 (especially if lifespan extension really is in reach)?
> Society will eventually start to resemble an open source project where a few core contributors do the real work (and get to decide the direction), some others help around, and the majority of people just benefit without having to do anything.
Open-source governance https://en.wikipedia.org/wiki/Open-source_governance
Free-rider problem https://en.wikipedia.org/wiki/Free-rider_problem
As we continue to reward work, the people who are investing in the means of production (energy, labor, automation, raw materials) and science (research and development; education) continue to amass wealth and influence.
This concentration of wealth -- wealth inequality -- has historically presaged and portended unrest.
How contributions to open source projects are reinforced, what motivates people who choose to contribute (altruism, enlightened self interest, compassion, acceptance,), and what makes a competitive and thus sustainable open source project is an interesting study.
... Business models for open-source software: https://en.wikipedia.org/wiki/Business_models_for_open-sourc...
... Political Science: https://en.wikipedia.org/wiki/Political_science
... National currencies are valued in FOREX markets: https://en.wikipedia.org/wiki/Foreign_exchange_market
> I'm wondering if any books have been written to explore this concept further?
"The Singularity is Near: When Humans Transcend Biology" (2005) contains a number of extrapolated predictions; chief among these is that there will continue to be exponential growth in technological change https://en.wikipedia.org/wiki/The_Singularity_Is_Near
... Until we reach limits; e.g. the carrying capacity of our ecosystem, the edge of the universe.
"The Limits to Growth" (1972, 2004) https://en.wikipedia.org/wiki/The_Limits_to_Growth
"Leverage Points: Places to Intervene in a System" (2010) https://news.ycombinator.com/item?id=17781927
Who owns what and who 'gets to' just chill while the solar robots brush their teeth? Heady questions. "Tired yet?"
The Aragon Project has a really interesting take on open source governance:
""" IMAGINE A NATION WITHOUT LAND AND BORDERS
A digital jurisdiction
> Aragon Network will be the first community governed decentralized organization whose goal is to act as a digital jurisdiction, an online decentralized court system that isn’t bound by traditional artificial barriers such as national jurisdictions or the borders of a single country.
Aragon organizations can be upgraded seamlessly using our aragonOS architecture. They can solve disputes between two parties by using the decentralized court system, a digital jurisdiction that operates only online and utilizes your peers to resolve issues.
The Aragon Network Token, ANT, puts the power into the hands of the people participating in the operation of the Network. Every single aspect of the Network will be governed by those willing to make an effort for a better future. """
Today, Europe Lost The Internet. Now, We Fight Back
Here's a quote from this excellent article:
> An error rate of even one percent will still mean tens of millions of acts of arbitrary censorship, every day.
And a redundant -- positively defiant -- link and page title:
"Today, Europe Lost The Internet. Now, We Fight Back." https://www.eff.org/deeplinks/2018/09/today-europe-lost-inte...
Firms with 50 or less employees should stay that small, really.
VPN providers in North and South America FTW.
> VPN providers in North and South America FTW.
Article 13 will affect the entire Internet, not just people in Europe. Most people on the Internet use large, multinational platforms. Those platforms will set rules according to the lowest common denominator, because it's the easiest to implement.
This means that people all over the world are going to have a much more difficult time with any user-generated content. That's true even for user-created content, even if there is no other copyright holder involved (look at how badly Content ID has played out).
Or they just stop doing business in Europe.
Since europe is a quite big market, maybe not an option and easier to geolocate and restrict just EU-Traffic
Or just not, if there's no nexus to europe. I suspect for a small business, the right thing to do would be to simply ignore EU directives. I would not be surprised if the US passes a law to make judgements against US companies without a European nexus (users in Europe would not count) unenforceable in the US. That was done with the SPEECH act to stop libel tourism.
Technically, the phrase "Useful Arts and Sciences" in the Copyright Clause of the US Constitution applies to just that; the definitions of which have coincidentally changed over the years.
The harms to Freedom of Speech -- i.e. impossible 99% accuracy in content filtering still results in far too much censorship -- so significantly outweigh the benefits for a limited number of special interests intending to thwart inferior American information services which also currently host "art" and content pertaining to the "useful arts"; that it's hard to believe this new policy will have it's intended effects.
Haven't there been multiple studies which show that free marketing from e.g. content piracy -- people who experience and recommended said goods at $0 -- is actually a net positive for the large corporate entertainment industry? That, unimpeded, content spreads like the common cold through word of mouth; resulting in greater number of artful impressions.
How can they not anticipate de-listing of EU content from news and academic article aggregators as an outcome of these new policies? (Resulting in even greater outsized impact on one possible front page that consumers can choose to consume)
For countries in the EU with less than 300 million voters, if you want:
- time for your headline: $
- time for your snippet: $$
- time for your og:description: $$
- free video hosting: $$$
- video revenue: $$$$
- < 30% American content: $$$$$
Pay your bill.
And what of academic article aggregators? Can they still index schema:ScholarlyArticle titles and provide a value-added information service for science?
Consumer science (a.k.a. home economics) as a college major
> That's why we need to bring back the old home economics class. Call it "Skills for Life" and make it mandatory in high schools. Teach basic economics along with budgeting, comparison shopping, basic cooking skills and time management.
Some Jupyter notebooks for these topics that work with https://mybinder.org could be super helpful. A self-paced edX course could also be a great intro to teaching oneself though online learning.
* Personal Finance (budgets, interest, growth, inflation, retirement)
* Food Science (nutrition, meal planning for n people, food prep safety, how long certain things can safely be left out on the counter)
* Productivity Skills (GTD, context switching overhead, calendar, email labels, memo app / shared task lists)
There were FACS (Family and Consumer Studies/Sciences) courses in our middle and high school curricula. Nutrition, cooking, sewing; family planning, carry a digital baby for awhile
Home economics https://en.wikipedia.org/wiki/Home_economics
* Family planning
https://en.wikipedia.org/wiki/Family_planning
> * Personal Finance (budgets, interest, growth, inflation, retirement)
Personal Finance https://en.wikipedia.org/wiki/Personal_finance
Khan Academy > College, careers, and more > Personal finance https://www.khanacademy.org/college-careers-more/personal-fi...
"CS 007: Personal Finance For Engineers" https://cs007.blog
https://reddit.com/r/personalfinance/wiki
> * Food Science (nutrition, meal planning for n people, food prep safety, how long certain things can safely be left out on the counter)
Food Science https://en.wikipedia.org/wiki/Food_science
Dietary management https://en.wikipedia.org/wiki/Dietary_management
Nutrition Education: https://en.wikipedia.org/wiki/Nutrition_Education
MyPlate https://en.wikipedia.org/wiki/MyPlate
Healthy Eating Plate https://www.hsph.harvard.edu/nutritionsource/healthy-eating-...
How to make salads, smoothies, sandwiches
How to compost and avoid unnecessary packaging
* School, College, Testing, "How Children Learn"
GED, SAT, ACT, MCAT, LSAT, GRE, GMAT, ASVAB
Defending a Thesis, Bar Exam, Boards
Khan Academy > College, careers, and more https://www.khanacademy.org/college-careers-more
Educational Testing https://wrdrd.github.io/docs/consulting/educational-testing
529 Plans (can be used for qualifying educational expenses for any person) https://en.wikipedia.org/wiki/529_plan
Middle School "Glimpse" project: Past, Present, Future. Present, Future: plan your 4-year highschool course plan, pick 3 careers, pick 3 colleges (and how much they cost)
High school literature: write a narrative essay for college admissions
* Health and Medicine
How to add emergency contact and health information to your phone, carseat (ICE: In Case of Emergency)
How to get health insurance ( https://healthcare.gov/ )
"What's your blood type?" (?!)
Khan Academy > Science > Health and Medicine https://www.khanacademy.org/science/health-and-medicine
Facebook vows to run on 100 percent renewable energy by 2020
Miami Will Be Underwater Soon. Its Drinking Water Could Go First
Now, now, let's focus on the positives here:
- more pollution from shipping routes through the Arctic circle (and yucky-looking icebergs that tourists don't like)
- less beachfront property
- more desalinatable water
- hotter heat
- more revulsive detestable significant others (displaced global unrest)
- costs of responding to natural disasters occurring with greater frequency due to elevated ocean temperatures
- less parking spaces (!)
What are the other costs and benefits here?
I've received a number of downvotes for this comment. I think it's misunderstood, and that's my fault: I should have included [sarcasm] around the whole comment [/sarcasm].
I've written about our need to address climate change here in past comments. I think the administration's climate change denials (see: "climate change politifact') and regulatory rollbacks are beyond despicable: they're sabotaging the United States by allowing more toxic chemicals into the environment that we all share, and allowing more sites that must be protected with tax dollars that aren't there because these industries pay far less than benchmarks in terms of effective tax rate. We know that vehicle emissions, mercury, and coal ash are toxic: why would we allow people to violate the rights of others in that way?
A person could voluntarily consume said toxic byproducts and not have violated their own rights or the rights of others, you understand. There's no medical value and low potential for abuse, so we just sit idly by while they're violating the rights of other people by dumping toxic chemicals into the environment that are both poisonous and strongly linked to climate change.
What would help us care about this? A sarcastic list of additional reasons that we should care? No! Miami underwater during tourist season is enough! I've had enough!
So, my mistake here - my downvote-earning mistake - was dropping my generally helpful, hopeful tone for cynicism and sarcasm that wasn't motivating enough.
We need people to regulate pollution in order to prevent further costs of climate change. Water in the streets holds up commerce, travel, hampers national security, and destroys the road.
We must stop rewarding pollution if we want it - and definitely resultant climate change - to stop. What motivates other people to care?
Free hosting VPS for NGO project?
The Burden: Fossil Fuel, the Military and National Security
Here's a link to the video: https://vimeo.com/194560636
Scientists Warn the UN of Capitalism's Imminent Demise
The actual document title: "Global Sustainable Development Report 2019 drafted by the Group of independent scientists: Invited background document on economic transformation, to chapter: Transformation: The Economy" (2018) https://bios.fi/bios-governance_of_economic_transition.pdf [PDF]
Why I distrust command economies (beyond just because of our experiences with violent fascism and defense overspending and the subsequent failures of various communist regimes):
We have elections today. We don't choose to elect people that regard the environment (our air, water, land, and other natural resources) as our most important focus. A command economy driven by these folks for longer than a term limit would be even more disastrous.
The market does not solve for 'externalities': things that aren't costed in. We must have regulation to counteract the blind optimization for profit (and efficiency) which capitalism rewards most.
Environmental regulation is currently insufficient; worldwide. That is the consensus from the Paris Agreement which 195 countries signed in 2015. https://en.wikipedia.org/wiki/Paris_Agreement
Maybe incentives?
We could sell tokens for how much pollutants we're allowed to f### everyone else over with and penalize exceeding the amount we've purchased. That would incentivize firms to pollute less so that they can save money by having to buy fewer tokens. (Europe does this already; and it's still not going to save the planet from industrial production externalities)
So, while I'm wary of any suggestion that a command economy would somehow bring forth talent in governance, I look to this article for actionable suggestions that penalize and/or incentivize sustainable business and living practices.
Sustainable reporting really is a must: how can I design an investment portfolio that excludes reckless, irresponsible, indifferent, and careless investments and highly values sustainability?
No one likes to be driven by harsh penalties; everyone likes to be rewarded (even with carrots as incentives).
Markets do not solve for long term outcomes. Case in point: the market has not chosen the most energy efficient cryptocurrencies. Is this an information asymmetry issue: people just don't know, or just don't care because the incentives are so alluring, the brand is so strong, or the perceived security assurances of the network outweighs the energy use (and environmental impact) in comparison to dry cleaning and fossil fuel transport.
How would a command economy respond to this? It really is denial and delusion to think that the market will cast aside less energy efficient solutions in order to save the environment all on its own.
So, what do we do?
Do we incentivize getting inefficient vehicles off of the road and into a recycling plant where they belong?
Do we shut down major sources of pollution (coal plants, vehicle emissions)?
Do we create tokens to account for pollution allowances (for carbon and other toxic f###ing chemicals)?
Do we cut irrational subsidies for industries that don't pay their taxes (even when they make money); so that we're aware of the actual costs of our behavior?
Do we grow hemp to absorb carbon, clean up the soil, replace emissions, and store energy?
Who's in the mood to dom these greedy shortsighted idiots into saving themselves and preventing the violation of our right to health (life)? No, you can't because you're busy violating your own rights and finding drugs/druggies and that's not allowed? Is that a lifetime position?
"Go burn a charcoal grill and your gas vehicle in your closed garage for awhile and come talk to me." That's really what we're dealing with here.
Anyways, this paper raises some good points; although I have my doubts about command economies.
[strikethrough] You can't do that to yourself. [/strikethrough] You can't do that to others (even if you pay for their healthcare afterwards).
Where's Captain Planet when you need 'em, anyways?
The problem may actually be solved by a free market... But it has to have different optimizations.
"Value" or "Quality" of a company has to stop being measured in unconstrained growth and instead measured in minimized externalities, while achieving your stated goal.
Accounting has to become a hell of a lot more complicated. We're talking "keep track of your industrial waste product", possibly even having to take responsibility in some manner for dealing with it directly.
There also needs to be some societal change as well. We need to Look at how we've stretched our supply and industry chains worldwide and start to figure out ways to minimize transportation and production costs.
Competition may need to fundamentally change it's nature as well. Trade secrets need to be ripped out into the light of day. It's not a contest of out doing the other guy, but to see who can find a way to make the entire industry more efficient.
Definitely a lot of change needed. That is for sure.
Firefox Nightly Secure DNS Experimental Results
> The experiment generated over a billion DoH transactions and is now closed. You can continue to manually enable DoH on your copy of Firefox Nightly if you like.
...
> Using HTTPS with a cloud service provider had only a minor performance impact on the majority of non-cached DNS queries as compared to traditional DNS. Most queries were around 6 milliseconds slower, which is an acceptable cost for the benefits of securing the data. However, the slowest DNS transactions performed much better with the new DoH based system than the traditional one – sometimes hundreds of milliseconds better.
Long-sought decay of Higgs boson observed at CERN
Furthermore, both teams measured a rate for the decay that is consistent with the Standard Model prediction, within the current precision of the measurement.
And now everyone: Noooo, not again.
(Explanation: it's well-known that the Standard Model can't be completely correct but again and again physicists fail to find an experiment contradicting its predictions, see https://en.wikipedia.org/wiki/Physics_beyond_the_Standard_Mo... for example)
Well, the standard model can be correct. It is correct until some experiment proves otherwise.
It is full of unexplained hadcoded parameters, indeed, which need an explanation from outside of the SM.
> It is full of unexplained hadcoded parameters, indeed, which need an explanation from outside of the SM.
https://en.wikipedia.org/wiki/Magic_number_(programming)#Unn...
> The term magic number or magic constant refers to the anti-pattern of using numbers directly in source code
Who is going to request changes when doing a code review on a pull request from God?
But which branch do you pull from? God has been forked so many times, it's hard to keep track. There are several distros available, so I guess you just get to pick the one that checks the most of your needs at the time.
Sen. Wyden Confirms Cell-Site Simulators Disrupt Emergency Calls
Building a Model for Retirement Savings in Python
re: pulling historical data with pandas-datareader, backtesting, algorithmic trading: https://www.reddit.com/r/Python/comments/7zxptg/pulling_stoc...
re: historical returns
- [The article uses a constant 7% annual return rate]
- "The current average annual return from 1923 (the year of the S&P’s inception) through 2016 is 12.25%." https://www.daveramsey.com/blog/the-12-reality (but that doesn't account for inflation)
- https://www.quantopian.com/posts/56b62019a4a36a79da000059 (300%+ over n years (from a down market))
Is there a Jupyter notebook with this code (with a requirements.txt for https://mybinder.org (repo2docker))?
New E.P.A. Rollback of Coal Pollution Regulations Takes a Major Step Forward
Would you move your family downwind from a coal plant? Why or why not?
Coal ash pollutes air, water, rain (acid rain), crops (our food), and soil. Which rights of victims does coal pollution infringe? Who is liable for the health effects?
Canada (2030), France (2021), and the UK (2025) are all working to entirely phase out coal-fired power plants for very good reasons (such as neonatal health).
~"They're just picking on coal": No, we're choosing renewables that are lower cost AND don't make workers and citizens sick.
If you can mine for coal, you can set up solar panels and wind turbines.
If you can run a coal mine; you can buy some cheap land, put up solar panels and wind turbines, and connect it to the grid.
Researchers Build Room-Temp Quantum Transistor Using a Single Atom
"Quasi-Solid-State Single-Atom Transistors" (2018) https://onlinelibrary.wiley.com/doi/full/10.1002/adma.201801...
New “Turning Tables” Technique Bypasses All Windows Kernel Mitigations
Um – Create your own man pages so you can remember how to do stuff
Interesting project. I believe these two shell functions in the shell's initialization file (~/.profile, ~/.bashrc, etc.) can serve as a poor man's um:
umedit() { mkdir -p ~/notes; vim ~/notes/"$1.txt"; }
um() { less ~/notes/"$1.txt"; }If you write these in .rst, you can generate actual manpages with Sphinx: http://www.sphinx-doc.org/en/master/usage/configuration.html...
sphinx.builders.manpage: http://www.sphinx-doc.org/en/master/_modules/sphinx/builders...
[deleted]
Leverage Points: Places to Intervene in a System
Extremely interesting take, with a lot of good stuff to think about.
I'm unclear about the claim that less economic growth would be better, though, and the author seems very committed to it. I wasn't able to find the article they reference as explaining how less growth is what we really need (J.W. Forrester, World Dynamics, Portland OR, Productivity Press, 1971), and it comes from almost 50 years ago, which might as well be another economic era altogether.
Does anyone know what the arguments are, what assumptions they require, and whether they still apply today? My understanding is that "less growth is better" is a distinctly minority take amongst modern economists, but the rest of this article seems very intelligently laid out, so I'd like to dig deeper.
I've always thought that for any dial we have, there's always an optimal setting, whether it's tax rates, growth rates, birth rates, etc., and blindly pushing one way or the other (like both political parties tend to do) is not helpful, or at the very least merely indicates different value systems.
The book is called "limits to growth", and it's a work of the Club of Rome. You can watch this video to have an idea of their work [1]. At first it's a very strange idea, afterall we work and consume everyday in order to grow the economy. But every healthy system as a homeostastic point, a point where it doesn't need to grow, only to be maintained. We are now a fat society and we need to get our health back, we need to degrow. We need to work much less hours and consume much less. Reducing working hours is a great leverage point. People will start to have time to care about the community, and take care of their own health. Maybe have more time to take a walk instead of using the car. This can improve health and environment but will not contribute to economic growth, healthy people that don't use cars are not good friends of economic growth based on GDP. English is not my mother language, Ursula Le Guin had an eloquent post about this on her blog, but was removed to be on the last book. The post is called "Clinging to a Metaphor" (the metaphor is economic growth) and the book is called "No Time to Spare". [1] https://youtu.be/kz9wjJjmkmc
"The Limits to Growth" (1972) https://en.wikipedia.org/wiki/The_Limits_to_Growth
"Thinking in Systems: a Primer" (2008) https://g.co/kgs/B71ebC
Glossary of systems theory https://en.wikipedia.org/wiki/Glossary_of_systems_theory
Systems Theory https://en.wikipedia.org/wiki/Systems_theory
...
Computational Thinking https://en.wikipedia.org/wiki/Computational_thinking
Which of the #GlobalGoals (UN Sustainable Development Goals) Targets and Indicators are primary leverage points for ensuring - if not growth - prosperity? https://en.wikipedia.org/wiki/Sustainable_Development_Goals
SQLite Release 3.25.0 adds support for window functions
https://www.windowfunctions.com is a good introduction to window functions.
Besides that, the comprehensive testing and evaluation of SQLite never ceases to amaze me. I'm usually hesitant to call software development "engineering", but SQLite is definitely well-engineered.
This looks great, but I couldn't get through the first question on aggregate functions. Are there any SQL books/tutorials that go over things like this?
A lot of material I've seen has been like the classic image of "How to draw an owl. First draw two circles, then draw the rest of the owl", where they tell you the super basic stuff, then assume you know everything.
Ibis uses windowing functions for aggregations if the database supports them. IDK when support for the new SQLite support will be implemented? http://docs.ibis-project.org/sql.html#window-functions
[EDIT]
I created an issue for this here: https://github.com/ibis-project/ibis/issues/1597
Update on the Distrust of Symantec TLS Certificates
Is the certifi bundle (2018.8.13) on PyPI also updated? https://pypi.org/project/certifi/
https://github.com/certifi/certifi.io/issues/18
> Are these still in the bundle?
> Should projects like requests which depend on certifi also implement this logic?
The Transport Layer Security (TLS) Protocol Version 1.3
Academic Torrents – Making 27TB of research data available
All the .torrent files are served over http so with a simple MITM attack a bad actor could swap in their own custom tweaked version of any data set here in order to achieve whatever goals that might serve for the bad actor's interests.
I really wish we could get basic security concepts added to the default curriculum for grade schoolers. You shouldn't need a PhD in computer security to know this stuff. These site creators have PhDs in other fields, but obviously no concept of security. This stuff should be basic literacy for everyone.
> This stuff should be basic literacy for everyone.
Arguably, one compromised PKI x.509 CA jeopardizes all SSL/TLS channel sec if there's no certificate pinning and an alternate channel for distributing signed cert fingerprints (cryptographically signed hashes).
We could teach blockchain and cryptocurrency principles: private/secret key, public key, hash verification; there there's money on the table.
GPG presumes secure key distribution (`gpg --verify .asc`).
TUF is designed to survive certain role key compromises. https://theupdateframework.github.io
1/0 = 0
1/0 = 1(±∞)
https://twitter.com/westurner/status/960508624849244160
> How many times does zero go into any number? Infinity. [...]
> How many times does zero go into zero? infinity^2?
Power Worth Less Than Zero Spreads as Green Energy Floods the Grid
I don't truly understand this "problem". I understand storing the energy in batteries is currently very expensive economically and materially.
However I believe there are plenty of "goods" (irrespective of if they are bulk materials, or partially processed products) which have a high processing energy per volume ratio (this does not need to be recoverable stored energy).
Allow me to give an example: currently we have a drought in Belgium (or at least Flanders). We are not landlocked, there is plenty of water in the sea. Desalination is energy intensive. Instead of only looking at energy storage, why can't we increase the processing capacity (more desalination sites capable of working in parallel), and desalinate say sea water during the energy flood? I don't expect this to be an ideal real world example, only a pattern for identifying such examples: any product (could be composite parts, or bulk material) which is relatively compact and has some high energy per product volume processinng step. Just do the process (desalination, welding some part to another part...) when the sun shines, and store them for later.
Products with very high step energy density are good candidates for storing, and could help flatten daily variations, and perhaps even seasonal variations!
Now some companies would prefer avoiding risk if they don't have guaranteed orders far enough into the future, then perhaps there should be a market for insurance or loans, so that the company is encouraged to take the risk, instead of wasting the cheap energy...
Capital costs vs. marginal costs.
You're going to build a $100M desalination plant and run it for three hours a day? That's a ton of money sitting idle most of the day, far more than what is recovered with zero operating costs.
(This is called the utilization factor -- how long a piece of equipment is used vs. staying idle)
Ideally you want useful processes with low capital costs and expensive marginal/energy costs. Desalination is not one of those.
A desal plant is a useful thing and can be run around the clock off traditional energy sources. The "free energy" hours would help lower the costs.
I can even imagine a SETI-like application where people who over-generate power are able to donate it to causes of their preference...
But if you are using traditional energy sources to run it all the time, it is no longer 'solving' the problem of burning off excess energy during peak renewable times.
Someone is having to build a lot of highly wasteful, redundant infrastructure.
Rational cryptocurrency mining firms can use the excess (unstorable) energy by converting it back to money (while the sun shines and the wind blows).
Money > Energy > Money
> Someone is having to build a lot of highly wasteful, redundant infrastructure.
We're nowhere near having the energy infrastructure necessary to support everyone having an electric vehicle yet.
Energy storage is key to maximizing returns from renewables and minimizing irreversible environmental damage.
Kernels, a free hosted Jupyter notebook environment with GPUs
Hey Ben, are these going to support arbitrary CUDA?
At the moment, we're focused on providing great support for the Python and R analytics/machine learning ecosystems. We'll likely expand this in the future, and in the meantime it's possible to hack through many other usecases we don't formally support well.
How do you handle custom environment requirements, whether it’s Python version, library version, or more complex things in the environment that some code might run on?
Basically, suppose I wanted everything that I could define in a Docker container to be available “as the environment” in which the notebook is running. How do I do that?
I ask because I’ve started to see an alarming proliferation of “notebook as a service” platforms that don’t offer that type of full environment spec, if they offer any configuration of the run time environment at all.
I’ve taught probability and data science at university level and worked in machine learning in a variety of businesses too, and I’d say for literally all use cases, from the quickest little pure-pedagogy prototype of a canned Keras model to a heavily customized use case with custom-compiled TensorFlow, different data assets for testing vs ad hoc exploration vs deployment, etc., the absolutely minimum thing needed before anything can be said to offer “reproducibility” is complete specification of the run time environment and artifacts.
The trend to convince people that a little “poke around with scripts in a managed environment” offering is value-additive is dangerous, very similar to MATLAB’s approach to entwine all data exploration with the atrocious development havits that are facilitated by the console environment (and to specifically target university students with free licenses, to use a drug dealer model to get engineers hooked on MATLAB’s workflow model and use that to leverage employers to oblige by buying and standardizing on abjectly bad MATLAB products).
Any time I meet young data scientists I always try to encourage them to avoid junk like that. It’s vital to begin experiments with fully reproducible artifacts like thick archive files or containers, and to structure code into meaningful reproducible units even for your first ad hoc explorations, and to absolutely always avoid linear scripting as an exploratory technique (it is terrible and ineffective for such a task).
Kaggle Kernels seems like a cool idea, so long as the programmer must fully define artifacts that describe the complete entirety of the run time environment, and nobody is sold on the Kool Aid of just linear scripting in some other managed environment.
Each kernel for example could have a link back to a GitHub repo containing a Dockerfile and build scripts for what defined the precise environment the notebook is running in. Now that’s reproducible.
Here are the Kaggle Kernels Dockerfiles:
- Python: https://github.com/Kaggle/docker-python/blob/master/Dockerfi...
- R: https://github.com/Kaggle/docker-rstats/blob/master/Dockerfi...
https://mybinder.org builds containers (and launches free cloud instances) on demand with repo2docker from a (commit hash, branch, or tag) repo URL: https://repo2docker.readthedocs.io/en/latest/config_files.ht...
That’s a great first step! Adding the ability to customize on a per-notebook basis would be impressive.
Solar and wind are coming. And the power sector isn’t ready
I don't know that fatalism and hopelessness are motivating for decision makers (who are seeking greater margins regardless of policy and lobbies).
Is our transformation to 100% clean energy ASAP a certain eventuality? On a long enough timescale, it would be irrational for utilities to not choose both lower cost and more sustainable environmental impact ('price-rational', 'environment-rational').
We should expect storage and generation costs to continue to fall as we realize even just the current pipeline of capitalizable [storage] research.
Solar energy is free.
Solar Just Hit a Record Low Price in the U.S
Relevant bits:
> “On their face, they’re less than a third the price of building a new coal or natural gas power plant,” Ramez Naam, an energy expert and lecturer at Singularity University, told Earther in an email. “In fact, building these plants is cheaper than just operating an existing coal or natural gas plant.”
> There’s a 30 percent federal investment tax credit for solar projects that helps drive down the cost of this and other solar projects. But Naam said even if you take away that credit, “these bids, un-subsidized, are still cheaper than any new coal or gas plants, and possibly cheaper than operating existing plants.”
(emphasis mine)
>> Relevant bits:
>> “On their face, they’re less than a third the price of building a new coal or natural gas power plant,” Ramez Naam, an energy expert and lecturer at Singularity University, told Earther in an email. “In fact, building these plants is cheaper than just operating an existing coal or natural gas plant.”
>> There’s a 30 percent federal investment tax credit for solar projects that helps drive down the cost of this and other solar projects. But Naam said even if you take away that credit, “these bids, un-subsidized, are still cheaper than any new coal or gas plants, and possibly cheaper than operating existing plants.”
I'm assuming that's without factoring in the health cost externalities.
Yes, this is solely for cost of power. The healthcare savings and quality of life improvements are an additional bonus on top of very cheap power.
https://www.sciencenews.org/article/air-pollution-triggering... (Air pollution is triggering diabetes in 3.2 million people each year)
https://www.scientificamerican.com/article/the-other-reason-... (The Other Reason to Shift away from Coal: Air Pollution That Kills Thousands Every Year)
Causal Inference Book
Causal inference (Causal reasoning) https://en.wikipedia.org/wiki/Causal_inference ( https://en.wikipedia.org/wiki/Causal_reasoning )
Tim Berners-Lee is working a platform designed to re-decentralize the web
Does anyone have some links on Solid that aren't media articles? I can't find anything, not even in Tim's homepage.
Home page: https://solid.mit.edu/
Spec: https://github.com/solid/solid-spec
Source: https://github.com/solid/solid
...
From https://news.ycombinator.com/item?id=16615679 ( https://westurner.github.io/hnlog/#comment-16615679 )
> ActivityPub (and OStatus, and ActivityStreams/Salmon, and OpenSocial) are all great specs and great ideas. Hosting and moderation cost real money (which spammers/scammers are wasting).
> Know what's also great? Learning. For learning, we have the xAPI/TinCan spec and also schema.org/Action.
Mastodon has now supplanted GNU StatusNet.
More States Opting to 'Robo-Grade' Student Essays by Computer
edX can automate short essay grading with edx/edx-ora2 "Open Response Assessment Suite" [1] and edx/ease "Enhanced AI scoring engine" [2].
1: https://github.com/edx/edx-ora2 2: https://github.com/edx/ease
... I believe there's also a tool for peer feedback.
Peer feedback/grading on MOOCs is pretty bad in my experience. There’s too much diversity of skills, language ability, etc. And too many people who bring their own biases and mostly ignore any grading instructions.
Peer discussion and feedback are useful in things like college classes. Much less so with MOOCs.
Ask HN: Looking for a simple solution for building an online course
I want to build an online course on graph algorithms for my university. I've tried to find a solution which would let submit, execute and test student's code (implement an online judge), but have had no success. There are a lot of complex LMS and none of them seem to have this feature as a basic functionality.
Are there any good out-of-box solutions? I'm sure I can build a course using Moodle or another popular LMS with some plugin, but I don't want to spend my time customizing things.
I'm interested both in platforms and self-hosted solutions. Thanks!
Maybe look at Jupyter Notebook? It does much of this out of the box, but may not be exactly what you are looking for.
nbgrader is a "A system for assigning and grading Jupyter notebooks." https://github.com/jupyter/nbgrader
jupyter-edx-grader-xblock https://github.com/ibleducation/jupyter-edx-grader-xblock
> Auto-grade a student assignment created as a Jupyter notebook, using the nbgrader Jupyter extension, and write the score in the Open edX gradebook
... networkx is a graph library written in Python which has pretty good docs: https://networkx.github.io/documentation/stable/reference/
There are a few books which feature networkx.
There is now a backprop principle for deep learning on quantum computers
"A Universal Training Algorithm for Quantum Deep Learning" https://www.arxiv-vanity.com/papers/1806.09729/
New research a ‘breakthrough for large-scale discrete optimization’
Looks like it may be this paper:
"An Exponential Speedup in Parallel Running Time for Submodular Maximization without Loss in Approximation" https://www.arxiv-vanity.com/papers/1804.06355/
The ACM STOC 2018 conference links to "The Adaptive Complexity of Maximizing a Submodular Function" http://dl.acm.org/authorize?N651970 https://scholar.harvard.edu/files/ericbalkanski/files/the-ad...
A DOI URI would be great, thanks.
Wind, solar farms produce 10% of US power in the first four months of 2018
Take note: 10% PRODUCED, not 10% CONSUMED.
This is counting all output by wind and solar regardless if it is needed and usable when the power is being produced. This is quite important because wind and solar are not on-demand sources of power.
> This is counting all output by wind and solar regardless if it is needed and usable when the power is being produced. This is quite important because wind and solar are not on-demand sources of power.
I think you have that backwards: in the US, we lack the ability to scale down coal and nuclear plants. Solar and Wind are generally the first to get pulled offline when generated capacity exceeds demand and storage.
TIL this is called "curtailment" and it's an argument that utilities have used to justify not spending on renewables that are saving the environment from global warming (which is going to require more electricity for air conditioning).
Solar energy production peaks around noon. Demand for electricity peaks in the evening. We need storage (batteries with supercapacitors out front) in order to store the difference between peak generation and peak use. Because they're unable to store this extra energy, they temporarily shut down solar and wind and leave the polluting plants online.
Consumers aren't exposed to daily price fluctuations: they get a flat rate that makes it easy to check their bill; so there's no price incentive to e.g. charge an EV at midday when energy is cheapest.
The 'Duck curve' shows this relation between peak supply and demand in electricity markets: https://en.wikipedia.org/wiki/Duck_curve
Developing energy storage capabilities (through infrastructure and open access basic research that can be capitalized by all) is likely the best solution. According to a fairly recent report, we could go 100% renewable with the energy storage tech that exists today.
But there's no money for it. There's money for subsidizing oil production (regardless of harms (!)), but not so much for wind and solar. There's money for responding to natural disasters caused by global warming, but not so much for non-carbon-based energy sources that don't cause global warming. A film called "The Burden: Fossil Fuel, the Military, and National Security" quotes the actual unsubsidized price of a gallon of gasoline.
Wouldn't it be great if there was some kind of computer workload that could be run whenever energy is cheapest ( 'energy spot instances') so that we can accelerate our migration to renewable energy sources that are saving the environment for future generations? If there were people who had strong incentives to create demand for power-efficient chips and inexpensive clean energy.
Where would be if we had continued with Jimmy Carter's solar panels on the roof of the White House (instead of constant war and meddling with competing oil production regions of the world)?
It's good to see wind and solar growing this fast this year. A chart with cost per kWhr or MWhr would be enlightening.
FDA approves first marijuana-derived drug and it may spark DEA rescheduling
Perhaps someone from the US could help me understand the federal vs state legality of cannabis in the USA?
Can a state override any federal law?
Could federal-level law enforcement theoretically charge someone in a cannabis-legal state for drug offenses?
states cannot override any federal law.
the federal government merely tolerates the semi-autonomous nature of states in order to maintain order. but the legality of the federal government's supremacy is well evolved, its power and might is extremely disproportionally higher than any collective of states, and at this point the semi-autonomous nature of states is pure fiction as the federal government can assume jurisdiction over any intra-state matter if it wanted to through its interstate commerce laws.
yes, federal level law enforcement can theoretically charge someone in a cannabis-legal state for drug offenses. this still happens. the discretion of the words of the President, the heads of the DEA and DOJ prevent the MAJORITY of it from happening and also guide the discretion of the courts and public sympathy. so right now, for the last two administrations it has not been a priority to upset the social order in cannabis-legal states. but it can still happen.
> states cannot override any federal law.
Not actually true: https://en.wikipedia.org/wiki/Nullification_(U.S._Constituti...
> not actually true
a concept which requires agreement from the federal courts themselves, who have never upheld a single argument related to this concept.
not actually what?
Selective incorporation: https://en.wikipedia.org/wiki/Incorporation_of_the_Bill_of_R...
10th Amendment: https://en.wikipedia.org/wiki/Tenth_Amendment_to_the_United_...
> The Tenth Amendment, which makes explicit the idea that the federal government is limited to only the powers granted in the Constitution, has been declared to be a truism by the Supreme Court.
Supremacy clause: https://en.wikipedia.org/wiki/Supremacy_Clause
> The Supremacy Clause of the United States Constitution (Article VI, Clause 2) establishes that the Constitution, federal laws made pursuant to it, and treaties made under its authority, constitute the supreme law of the land.[1] It provides that state courts are bound by the supreme law; in case of conflict between federal and state law, the federal law must be applied. Even state constitutions are subordinate to federal law.[2] In essence, it is a conflict-of-laws rule specifying that certain federal acts take priority over any state acts that conflict with federal law
Natural rights ('inalienable rights': Equal rights, Life, Liberty, pursuit of Happiness): https://en.wikipedia.org/wiki/Natural_and_legal_rights
9th Amendment: https://en.wikipedia.org/wiki/Ninth_Amendment_to_the_United_...
> The Ninth Amendment (Amendment IX) to the United States Constitution addresses rights, retained by the people, that are not specifically enumerated in the Constitution. It is part of the Bill of Rights.
If the 9th Amendment recognizes any unenumerated rights of the people (with Supremacy, regardless of selective incorporation), it certainly recognizes those of the Declaration of Independence ((secession from the king ('CSA')), Equality, Life, Liberty, pursuit of Happiness), our non-binding charter which frames the entirety of the Constitutional convention
All of these things have been interpreted by the courts and their conclusion was not like yours
It is nice that you are interested in these things, but they simply cannot be read verbatim and then extrapolated to other things.
This isn't educational for anybody, this is a view that lacks all consensus and all avenues to ever garner consensus in this country.
Again, I ask you to explain how the current law grants equal rights.
https://news.ycombinator.com/item?id=17401906
> We tend to have issues with Equal rights/protections: slavery, voting rights, [school] segregation. Please help us understand how to do this Equally:
>> Furthermore, (1) write a function to determine whether a given Person has a (natural inalienable) right: what information may you require? (2) write a function to determine whether any two Persons have equal rights.
Abolitionists faced similar criticism from on high.
States Can Require Internet Tax Collection, Supreme Court Rules
I have a hunch that this will, in the end, be a massive win for large retailers vs. small ones. The task of figuring out how to calculate tax for all states is more or less the same amount of work regardless of size, which means for someone like Amazon it's more or less trivial, but for a mom-and-pop store it's a major hassle.
Thankfully with Shopify it is extremely easy and straightforward to manage for my wife's small online store. Their platform does a great job properly charging taxes by state, county and city in certain situations. Then using an inexpensive plan from https://www.taxjar.com/ the entire filing and paying process is 100% automated.
In 10 minutes I was able to file and pay all the sales taxes to several state, dozens of California counties and a handful of cities that charge additional taxes on top.
So you assume. As a small business you are unlikely to be audited, but that software could easily be wrong creating a huge minefield and potential liability.
So that would be the software companies' liability. And such business practice can differentiate good ones from the bad ones. I'm seeing a new business market here even.
https://en.wikipedia.org/wiki/Parable_of_the_broken_window
There is zero economic gain from more complex tax rules. Further, the software does not absolve you of liability. At best they may agree to cover it, but that's unlikely and they can also go broke if they get it wrong.
Actually, current sales tax software provided by South Dakota and other states does absolve a merchant for liability if used to calculate sales tax due.
I agree with this approach!
Actually, the federal government should oblige each member state to provide the algorithm, and sign it cryptographically and have it expire every X fixed time interval, and have signed algorithms for the current and next time interval, so that software can automatically fetch and stay up to date.
Then the "business opportunity" of navigating FUD evaporates. Currently any such enterprise charging for such a service can spend a fraction of their budget lobbying against harmonization...
Since it would be an obligation of the states to the federal government, these algorithms (provided by each member state) should be hosted on a fixed federal government site.
Time to start a petition?
This would reduce costs of tax collection for all parties.
What is the most convenient format for this layered geographic data? Are the tax district boundary polygons already otherwise available as open data? What do localities call these? Sales tax tables, sales tax database, machine-readable flat files in an open format with a common schema?
How much tax revenue should it cost to provide such a service on a national level?
States, Counties, Cities, 'Tax Zones'(?) could be required to host tax.state.us.gov or similar with something like Project Open Data JSONLD /data.json that could be aggregated and shared by a server with a URL registry, a task queue service, and a CDN service.
While the Bitcoin tax payments bill passed the Senate and House in Arizona, it was vetoed in May 2018. Seminole County in Florida now allows tax payment with crytocurrencies such as Bitcoin:
https://cointelegraph.com/news/us-seminole-county-florida-to...
> According to a press release, the county will begin accepting Bitcoin (BTC) and Bitcoin Cash (BCH) to pay for services, including property taxes, driver license and ID card fees, as well as tags and titles. The Seminole County Tax Collector will reportedly employ blockchain payments company BitPay, which will allow the county to receive settlement the next business day directly to its bank account in US dollars.
This could also help reduce the costs of tax collection and possibly increase the likelihood of compliance with the forthcoming tax bills!
these are all very good questions, and only a community discussion of people with the right skills and interests can draft a petition, if enough people contribute to the discussion we can make the proposal more reasonable and robust against valid criticisms... but I believe we can make this happen by just starting the discussion. We can bitch on Hacker News, or we can draft a proposal for the different government levels. The more reasonable we draft it, the higher the probability the petition will be a success. I think it wouldn't be hard to argue against this proposal that a legally enforced computation should be open source, i.e. not just the algorithm for the computation but also all the data lists and boundary polygons used in the algorithm...
Ask HN: Do you consider yourself to be a good programmer?
if not, why? how do you validate your achievements?
> For identifying strengths and weaknesses: "Programmer Competency Matrix":
> - http://sijinjoseph.com/programmer-competency-matrix/
> - https://competency-checklist.appspot.com/
> - https://github.com/hltbra/programmer-competency-checklist
Don’t read too much into that. TDD for example is not leveling up it’s an opinionated approach to development.
Automated testing is not a choice in many industries.
If you're not familiar with TDD, you haven't yet achieved that level of mastery.
There's a productivity boost to being able to change quickly without breaking things.
Is all unit/functional/integration testing and continuous integrating TDD? Is it still TDD if you write the tests after you write the function (and before you commit/merge)?
I think this competency matrix is a helpful resource. And I think that learning TDD is an important thing for a good programmer.
There is absolutely no need to follow TDD to be good at testing.
This is all unfounded conjecture: it seems easier to remember which parameter combinations may exist and need to be tested when writing the function; so "let's all write tests later" becomes a black box exercise which is indeed a helpful perspective for review, but isn't the most effective use of resources.
IMHO being convinced that there's only one true and correct methodology (TDD, Scrum, etc.) or paradigm (functional, objective, reactive programming, etc.) is a sign of being a bad programmer.
A good programmer finds common attributes and behaviors and organizes them into namespaced structs/arrays/objects with functions/methods and tests. Abstractly, which terms should we use to describe hierarchical clusters of things with information and behaviors if not those from a known software development or project management methodology?
And a good programmer asks why people might have spent so much time formalizing project development methodologies. "What sorts of product (team) failures are we dealing with here?" is an expensive question to answer as a team.
By applying tenets of Named agile software development methodologies, teams and managers can feel like they're discussing past and current experiences/successes/failures with comparable implementations of approaches that were or are appropriate for different contexts.
To argue the other side, just cherry picking from different methodologies is creating a new methodology, which requires time to justify basically what we already have terms for on the wall over here.
"We just pop tasks off the queue however" is really convenient for devs but can be kept cohesive by defining sensible queues: [kanban] board columns can indicate task/issue/card states and primacy, [sprint] milestone planning meetings can yield complexity 'points' estimates for completable tasks and their subtasks. With team velocity (points/time), a manager can try to appropriately schedule optimal paths of tasks (that meet the SMART criteria (specific, measurable, achievable, relevant, and Time-bound)); instead of fretting with the team over adjusting dates on a Gantt chart (task dependency graph) deadline, the team can
What about your testing approach makes it 'NOT TDD'?
How long should the pre-release static analysis and dynamic analyses take in my fancy DevOps CI TDD with optional CD? Can we release or deploy right now? Why or why not?
'We can't release today because we spent too much time arguing about quotes like "A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines." ("Self Reliance" 1841. Emerson) and we didn't spec out the roof trusses ahead of time because we're continually developing a new meeting format, so we didn't get to that, or testing the new thing, yet.'
A good programmer can answer the three questions in a regular meeting at any time, really:
> 1. What have you completed since the last meeting?
> 2. What do you plan to complete by the next meeting?
> 3. What is getting in your way?
And:
Can we justify refactoring right now for greater efficiency or additional functionality?
The simple solution there is to simply not use specific parameters (outside ovious edge-cases, ie supplying -1 and 2^63 into your memory allocator). Writing a simple reproducible fuzzer is easy for most contained functions.
I find blackbox testing itself also fairly useful. The part where you forget which parameter combinations may occur can be useful since you now A) rely on documentation you made and B) can write your test independent of how you implemented it just like if you had written it beforehand. (Just don't forget to avoid falling into the 'write test to pass function' trap)
IMHO, it's so much easier to write good, comprehensive tests while writing the function (FUT: function under test) because that information is already in working memory.
It's also easier to adversarially write tests with a fresh perspective.
I shouldn't need to fuzz every parameter for every commit. Certainly for releases.
"Building an AppSec Pipeline: Keeping your program, and your life, sane" https://www.owasp.org/index.php/OWASP_AppSec_Pipeline
I mean, I general don't think you should write fuzzers for absolutely everything (most contained functions => doesn't touch a lot of other stuff and few parameters with a known parameter space)
The general solution is to use whatever testing methodology you are comfortable, that is very effective, very efficient and covers a lot of problem space. Of course no testing method does that so you'll have to constantly balance whatever works best (which is why I think pure TDD is overrated)
> Is all unit/functional/integration testing and continuous integrating TDD?
No. They differentiate in the matrix.
> If you're not familiar with TDD, you haven't yet achieved that level of mastery.
That's not true - I've worked on teams with far lower defect rates than the typical TDD team.
TDD can help keep a developer focused - and this can help overall productivity rates - but it doesn't directly help lower defect rates.
> TDD can help keep a developer focused - and this can help overall productivity rates - but it doesn't directly help lower defect rates.
We would need to reference some data with statistical power; though randomization and control are infeasible: no two teams are the same, no two projects are the same, no two objective evaluations of different apps' teams' defect rates are an apples to apples comparison.
Maybe it's the coverage expectation: do not add code that is not run by at least one test.
Handles are the better pointers
A few years ago I wanted to make a shoot-em-up game using C# and XNA I could play with my kids on an Xbox. It worked fine except for slight pauses for GC every once in a while which ruined the experience.
The best solution I found was to get rid of objects and allocation entirely and instead store the state of the game in per-type tables like in this article.
Then I realized that I didn't need per-type tables at all, and if I used what amounted to a union of all types I could store all the game state in a single table, excluding graphical data and textures.
Next I realized that since most objects had properties like position and velocity, I could update those with a single method by iterating through the table.
That led to each "object" being a collection of various properties acted on by independent methods which were things like gravity and collision detection. I could specify that a particular game entity was subject to gravity or not, etc.
The resulting performance was fantastic and I found I could have many thousands of on-screen objects at the time with no hiccups. The design also made it really easy to save and replay game state; just save or serialize the single state table and input for each frame.
The final optimization would have been to write a language that would define game entities in terms of the game components they were subject to and automatically generate the single class that was union of all possible types and would be a "row" in the table. I didn't get around to that because it just wasn't necessary for the simple game I made.
I was inspired to do this from various articles about "component based game design" published at the time that were variations on this technique, and the common thread was that a hierarchical OO structure ended up adding a lot of unneeded complexity for games that hindered flexibility as requirements changed or different behaviors for in-game entities were added.
Edit: This is a good article on the approach. http://cowboyprogramming.com/2007/01/05/evolve-your-heirachy...
> The final optimization would have been to write a language that would define game entities in terms of the game components they were subject to and automatically generate the single class that was union of all possible types and would be a "row" in the table
django-typed-models https://github.com/craigds/django-typed-models
> polymorphic django models using automatic type-field downcasting
> The actual type of each object is stored in the database, and when the object is retrieved it is automatically cast to the correct model class
...
> the common thread was that a hierarchical OO structure ended up adding a lot of unneeded complexity for games that hindered flexibility as requirements changed or different behaviors for in-game entities were added.
So, in order to draw a bounding box for an ensemble of hierarchically/tree/graph-linked objects (possibly modified in supersteps for reproducibility), is an array-based adjacency matrix still fastest?
Are sparse arrays any faster for this data architecture?
> django-typed-models https://github.com/craigds/django-typed-models
Interesting, never came across it. But I've got a Django GitHub library (not yet open source, because it's in a project I've been working with on and off) that does the same for managing GitHub's accounts. An Organization and User inherit the same properties and I downcast them based on their polymorphic type field.
ContentType.model_class(), models.Model.meta.abstract=True, django-reversion, django-guardian
IDK how to do partial indexes with the Django ORM? A simple filter(bool, rows) could probably significantly shrink the indexes for such a wide table.
Arrays are fast if the features/dimensions are known at compile time (if the TBox/schema is static). There's probably an intersection between object reference overhead and array copy costs.
Arrow (with e.g. parquet on disk) can help minimize data serialization/deserialization costs and maximize copy-free data interoperability (with columnar arrays that may have different performance characteristics for whole-scene transformation operations than regular arrays).
Many implementations of SQL ALTER TABLE don't have to create a full copy in order to add a column, but do require a permission that probably shouldn't be GRANTed to the application user and so online schema changes are scheduled downtime operations.
If you're not discovering new features at runtime and your access pattern is generally linear, arrays probably are the fastest data structure.
Hacker News also has a type attribute that you might say is used polymorphically: https://github.com/HackerNews/API/blob/master/README.md#item...
Types in RDF are additive: a thing may have zero or more rdf:type property instances. RDF quads can be stored in one SQL table like:
_id,g,s,p,o,xsd:datatype,xml:lang
... with a few compound indexes that are combinations of (s,p,o) so that triple pattern graph queries like (?s,?p,1) are fast. Partial indexes (SQLite, PostgreSQL,) would be faster than full-table indexes for RDF in SQL, too.
Neural scene representation and rendering
This work is a natural progression from a lot of other prior work in the literature... but that doesn't make the results any less impressive. The examples shown are amazingly, unbelievably good! Really GREAT WORK.
Based on a quick skim of the paper, here is my oversimplified description of how this works:
During training, an agent navigates an artificial 3D scene, observing multiple 2D snapshots of the scene, each snapshot from a different vantage point. The agent passes these snapshots to a deep net composed of two main parts: a representation-learning net and a scene-generation net. The representation-learning net takes as input the agent's observations and produces a scene representation (i.e., a lower-dimensional embedding which encodes information about the underlying scene). The scene-generation network then predicts the scene from three inputs: (1) an arbitrary query viewpoint, (2) the scene representation, and (3) stochastic latent variables. The two networks are trained jointly, end-to-end, to maximize the likelihood of generating the ground-truth image that would be observed from the query viewpoint. See Figure 1 on Page 15 of the Open Access version of the paper. Obviously I'm playing loose with language and leaving out numerous important details, but this is essentially how training works, as I understand it based on a first skim.
EDIT: I replaced "somewhat obvious" with "natural," which better conveys what I actually meant to write the first time around.
I, literally just 15 minutes ago, had a chat with a friend of mine exactly about how what we are doing right now with computer vision is all based on a flawed premise (supervised 2D training set). The human brain works in 3D space (or 3D+time) and then projects all this knowledge in a 2D image.
Here I was, thinking I finally had thought of a nice PhD project and then Deepmind comes along and gets the scoop! Haha.
"Spatial memory" https://en.wikipedia.org/wiki/Spatial_memory
It may be splitting hairs, but I think the mammalian brain, at least, can simulate/remember/imagine additional 'dimensions' like X/Y/Z spin, derivatives of velocity like acceleration/jerk/jounce.
Is space 11 dimensional (M string theory) or 2 dimensional (holographic principle)? What 'dimensions' does the human brain process? Is this capacity innate or learned; should we expect pilots and astronauts to have learned to more intuitively cognitively simulate gravity with their minds?
New US Solar Record – 2.155 Cents per KWh
"Cost of electricity by source" https://en.wikipedia.org/wiki/Cost_of_electricity_by_source
"Electricity pricing" https://en.wikipedia.org/wiki/Electricity_pricing
> United States 8 to 17 ; 37[c] 43[c] [cents USD/kWh]
Ask HN: Is there a taxonomy of machine learning types?
Besides classification and regression, and the unsupervised methods for principle components, clustering and frequent item-sets, what tools are there in the ML toolkit and what kinds of problems are amenable to their use?
Outline of Machine Learning https://en.wikipedia.org/wiki/Outline_of_machine_learning
Machine learning # Applications https://en.wikipedia.org/wiki/Machine_learning#Applications
"machine learning map" image search: https://www.google.com/search?q=machine+learning+map&tbm=isc...
Senator requests better https compliance at US Department of Defense [pdf]
The "Mozilla SSL Configuration Generator" has a checkbox for 'HSTS enabled?' and can generate SSL/TLS configs for Apache, Nginx, Lighttpd, HAProxy, AWS, ELB. https://mozilla.github.io/server-side-tls/ssl-config-generat...
You can select 'nginx', then 'modern', and then 'apache' for a modern Apache configuration.
Are the 'modern' configs FIPS compliant?
What browsers/tools does requiring TLS 1.3 break?
Banks Adopt Military-Style Tactics to Fight Cybercrime
> In a windowless bunker here, a wall of monitors tracked incoming attacks — 267,322 in the last 24 hours, according to one hovering dial, or about three every second — as a dozen analysts stared at screens filled with snippets of computer code.
> Cybercrime is one of the world’s fastest-growing and most lucrative industries. At least $445 billion was lost last year, up around 30 percent from just three years earlier, a global economic study found, and the Treasury Department recently designated cyberattacks as one of the greatest risks to the American financial sector.
Is this type of monitoring possible (necessary, even) with blockchains? Blockchains generally silently disregard bad/invalid transactions. Where could discarded/disregarded transactions and forks be reported to in a decentralized blockchain system? Who would pay for log storage? How redundantly replicated should which data be?
How DDOS resistant are centralized and decentralized blockchains?
Exchanges have risk. In terms of credit fraud: some crypto asset exchanges do allow margin trading, many credit card companies either refuse transactions with known exchanges or charge cash advance interest rates, and all transactions are final.
Exchanges hold private keys for customers' accounts, move a lot to offline cold storage, and maybe don't do a great job of explaining that YOU SHOULD NOT LEAVE MONEY ON AN EXCHANGE. One should transfer funds to a different account; such as a hardware or paper wallet or a custody service.
Do/can crypto asset exchanges participate in these exercises? To what extent do/can blockchains help solve for aspects of our unfortunately growing cybercrime losses?
Premined blockchains could reportedly handle card/chip/PIN transaction volumes today.
No, Section 230 Does Not Require Platforms to Be “Neutral”
> It’s foolish to suggest that web platforms should lose their Section 230 protections for failing to align their moderation policies to an imaginary standard of political neutrality. Trying to legislate such a “neutrality” requirement for online platforms—besides being unworkable—would be unconstitutional under the First Amendment.
... https://en.wikipedia.org/wiki/Section_230_of_the_Communicati...
Ask HN: Do battery costs justify “buy all sell all” over “net metering”?
Are batteries the primary justification for "buy all sell all" over "net metering"?
Are next-gen supercapacitors the solution?
> Ask HN: Do battery costs justify "buy all sell all" over "net metering"?
> Are batteries the primary justification for "buy all sell all" over "net metering"?
> Are next-gen supercapacitors the solution?
With "Net Metering", electric utilities buy consumers' excess generated energy at retail or wholesale rates. https://en.wikipedia.org/wiki/Net_metering
With "Buy All, Sell All", electric utilities require consumers to sell all of the energy they generate from e.g. solar panels (usually at wholesale prices, AFAIU) and buy all of the energy they consume at retail rates. They can't place the meter after any local batteries.
Do I have this right?
Net metering:
(used-generated) x (retail || wholesale)
Buy all, sell all:
(used x retail) - (generated x wholesale)
For the energy generating consumer, net metering is a better deal: they have power when the grid is down, and they keep or earn more for the energy generation capability they choose to invest in.
Break-even on solar panels happens sooner with net metering.
Utilities argue that: maintaining grid storage and transfer costs money, which justifies paying energy generating consumers less than they pay for more constant sources of energy like dams, wind farms, and commercial solar plants.
Building a two-way power transfer grid costs money. Batteries require replacement after a limited number of cycles. Spiky or bursting power generation is not good for batteries because they don't get a full cycle. [Hemp] supercapacitors can smooth out that load and handle many more partial charge and discharge cycles.
Is energy storage the primary justifying cost driver for "buy all, sell all"?
What investments are needed in order to more strongly incentivize clean energy generation? Do we need low cost supercapacitors to handle the spiky load?
Are these utilities granted a monopoly? Are they price fixing?
Energy demand from blockchain mining has not managed to keep demand constant so that utilities have profit to invest in clean energy generation and a two-way smart grid that accommodates spiky consumer energy generation. Demand for electricity is falling as we become less wasteful and more energy efficient. As the cost of renewable energy continues to fall (and become less expensive than nonrenewables), there should be more margin for energy utilities which cost-rationally and environmentally-rationally choose to buy renewable energy and sell it to consumers.
Please correct me with the appropriate terminology.
How can we more strongly incentivize consumer solar panel investments?
Here's a discussion about the lower costs of hemp supercapacitors as compared with graphene super capacitors: https://news.ycombinator.com/item?id=16800693
""" Hemp supercapacitors might be a good solution to the energy grid storage problem. Hemp absorbs carbon, doesn't leave unplowable roots in the fields, returns up to 70% of nutrients to the soil, and grows quickly just about anywhere. Hemp bast fiber is normally waste. Hemp anodes for supercapacitors are made from the bast fiber that is normally waste.
Graphene is very useful; but industrial production of graphene is dangerous because lungs and blood-brain barrier.
Hemp is an alternative to graphene for modern supercapacitors (which now have much greater [energy density] in wH/kg)
"Hemp Carbon Makes Supercapacitors Superfast” https://www.asme.org/engineering-topics/articles/energy/hemp...
> “Our device’s electrochemical performance is on par with or better than graphene-based devices,” Mitlin says. “The key advantage is that our electrodes are made from biowaste using a simple process, and therefore, are much cheaper than graphene.”
> Graphene is, however, expensive to manufacture, costing as much as $2,000 per gram. [...] developed a process for converting fibrous hemp waste into a unique graphene-like nanomaterial that outperforms graphene. What’s more, it can be manufactured for less than $500 per ton.
> Hemp fiber waste was pressure-cooked (hydrothermal synthesis) at 180 °C for 24 hours. The resulting carbonized material was treated with potassium hydroxide and then heated to temperatures as high as 800 °C, resulting in the formation of uniquely structured nanosheets. Testing of this material revealed that it discharged 49 kW of power per kg of material—nearly triple what standard commercial electrodes supply, 17 kW/kg.
https://scholar.google.com/scholar?hl=en&q=hemp+supercapacit....
https://en.wikipedia.org/wiki/Supercapacitor
I feel like a broken record mentioning this again and again. ""'
Portugal electricity generation temporarily reaches 100% renewable
Currently we are passing through an atypical wind and rain period. Our dams are full and pouring out water as we are producing more energy than we consume.
Despite all of this energy prices don't drop as they are bound to one of the few legal monopolies in Europe: the grid operation cartel (REN in the article, there is only one allowed for each EU country).
Also this feels as fake news since we have a few very old (and historically insecure, like sines power plant and carregado power plant) fossil fuel power plants that are still operating with no signs of slowing down while making profits through several scams in chemical engineering their way into the 0 emissions lot.
are batteries (or some sort of storage) in the infrastructure plan for the future?
Hemp supercapacitors might be a good solution to the energy grid storage problem. Hemp absorbs carbon, doesn't leave unplowable roots in the fields, returns up to 70% of nutrients to the soil, and grows quickly just about anywhere.
Hemp bast fiber is normally waste. Hemp anodes for supercapacitors are made from the bast fiber that is normally waste.
Graphene is very useful; but industrial production of graphene is dangerous because lungs and blood-brain barrier.
Hemp is an alternative to graphene for modern supercapacitors (which now have much greater power density in wH/kg)
"Hemp Carbon Makes Supercapacitors Superfast” https://www.asme.org/engineering-topics/articles/energy/hemp...
> “Our device’s electrochemical performance is on par with or better than graphene-based devices,” Mitlin says. “The key advantage is that our electrodes are made from biowaste using a simple process, and therefore, are much cheaper than graphene.”
> Graphene is, however, expensive to manufacture, costing as much as $2,000 per gram. [...] developed a process for converting fibrous hemp waste into a unique graphene-like nanomaterial that outperforms graphene. What’s more, it can be manufactured for less than $500 per ton.
> Hemp fiber waste was pressure-cooked (hydrothermal synthesis) at 180 °C for 24 hours. The resulting carbonized material was treated with potassium hydroxide and then heated to temperatures as high as 800 °C, resulting in the formation of uniquely structured nanosheets. Testing of this material revealed that it discharged 49 kW of power per kg of material—nearly triple what standard commercial electrodes supply, 17 kW/kg.
https://scholar.google.com/scholar?hl=en&q=hemp+supercapacit...
https://en.wikipedia.org/wiki/Supercapacitor
I feel like a broken record mentioning this again and again.
If you're going to be mentioning this again in the future please correct your usage of power/energy density. Power density is measured in W/kg, energy density is measured in Wh/kg. Supercapacitors tend to excel in the former but be poor in the latter. You mentioned power density but used units for energy density. This happens so often in media that I feel the need to correct it even in a comment.
> please correct your usage of power/energy density. Power density is measured in W/kg, energy density is measured in Wh/kg. Supercapacitors tend to excel in the former but be poor in the latter.
I'd update the units; good call. You may have that confused? Traditional supercapacitors have had lower power density and faster charging/discharging. Graphene and hemp somewhat change the game, AFAIU.
It makes sense to put supercapacitors in front of the battery banks because they last so many cycles and because they charge and discharge so quickly (a very helpful capability for handling spiky wind and solar loads).
I think you may still be a little confused. Power density is the rate at which energy can be added to or drawn from the the cell per unit mass. So faster charging and discharging means high power density. Energy density is the total amount of energy that can be stored per unit mass. Supercapacitors are typically higher in power density and lower in energy density than batteries[1].
You're right that it makes sense to put supercapacitors in front of the battery banks for the reasons you said.
[1] http://berc.berkeley.edu/storage-wars-batteries-vs-supercapa...
I must have logically assumed that rate of charge and discharge include time (hours) in the unit: Wh/kg.
My understanding is that there's usually a curve over time t that represents the charging rate from empty through full.
[edit]
"C rate"
Battery_(electricity)#C_rate https://en.wikipedia.org/wiki/Battery_(electricity)#C_rate
Battery_charger#C-rates https://en.wikipedia.org/wiki/Battery_charger#C-rates
> Charge and discharge rates are often denoted as C or C-rate, which is a measure of the rate at which a battery is charged or discharged relative to its capacity. As such the C-rate is defined as the charge or discharge current divided by the battery's capacity to store an electrical charge. While rarely stated explicitly, the unit of the C-rate is [h^−1], equivalent to stating the battery's capacity to store an electrical charge in unit hour times current in the same unit as the charge or discharge current.
It does sound amazing and economical, like almost too good to be true, but I very much hope it is true. What are the downsides? Is there a degradation problem or something similar? Other than the stoner connection in people's minds, what kind of resistance is there to this? Why isn't it widely known?
You know, I'm not sure. This article is from a few years ago now and there's not much uptake.
It may be that most people dismiss supercapacitors based on the stats for legacy (pre-graphene/pre-hemp) supercapacitors: large but quick and long-lasting.
It may be that hemp is taxed at up to 90% because it's a controlled substance in the US (but not in Europe, Canada, or China; where we must import shelled hemp seeds from). A historical accident?
GPU Prices Drop ~25% in March as Supply Normalizes
How do these new GPUs compare to those from 10 years ago in terms of FLOPs per Watt? https://en.wikipedia.org/wiki/Performance_per_watt
The new ASICs for Ethereum mining can't be solely responsible for this percent of the market.
(Note that NVIDIA's stock price is up over 1700% over the past 10 years. And that Bitcoin mining on CPUs and GPUs hasn't been profitable for quite awhile. In 2007, I don't think we knew that hashing could be done on GPUs; though there were SSL accelerator cards that were mighty expensive)
Apple says it’s now powered by renewable energy worldwide
As this article [1] explains, Apple does not (and cannot) actually run on 100% renewable energy globally, as any of its stores/premises/facilities that are connected to local municipal power grids will use whatever power generation method is used on that grid, and that is still likely to be fossil fuel in most locations.
But they purchase Renewable Energy Certificates to offset their use of non-renewable energy, so they can make the claim that their net consumption of non-renewable electricity is negative.
[1] https://www.theverge.com/2018/4/9/17216656/apple-renewable-e...
This is weird. So if I run a wind farm that generates 1 MW(...h? why is this an energy unit and not power? but whatever...) and I use all of that electricity myself, I can also sell 1 REC to someone else so that they claim they run on green electricity. Which means that now either (a) I have to legally claim I run on dirty electricity (which is a lie on its face and make no sense???) or (b) we both claim we run on green power, double-dipping and screwing up the accounting of greenness.
Am I misunderstanding something? How does this work?
No that's how it works. That's also the reason why Norway, despite only using hydro, has only 40-50% renewable energy in some statistics. They sell green energy certificates to consumers abroad (e.g. in Germany). Officially, Norwegians then use coal power whereas in reality it's all hydro power. There isn't even enough transmission capacity to the south to get that kind of exchange physically.
Wow! And I just realized there seems to be another loophole: that means (say) a company like Apple could start a separate power company in Norway based on hydro power, have that company completely waste 100% of the energy it produce there, and yet "buy" the equivalent REC in another jurisdiction where they run on coal and suddenly get to 100% "green" power... potentially even making more money in tax credits, if there are any, all while consuming more and more dirty power without actually helping anybody shift to renewable energy. Right?
I think it comes down to giving them credit for funding the construction of massive clean energy projects, even if they don't exclusively use the electricity from that project themselves.
Look at Microsoft. They just funded a deal to build out an absolutely massive solar farm in Virginia.
https://blogs.microsoft.com/on-the-issues/2018/03/21/new-sol...
They do have facilities in state that will only need a fraction of that power to be 100% green, and the excess will be pumped into the local Virginia power grid and used by consumers there.
Basically, Microsoft is saying that they funded the project to generate excess green energy in one place to offset the dirty energy they consume in areas where there is no local green power option available.
100% renewable energy by purchasing and funding renewable energy is an outstanding acheivement.
Is there another statistic for measuring how many KWhr or MWhr are sourced directly from renewable energy sources (or, more logically, 'directly' from batteries + hemp supercapacitors between use and generation)?
Hackers Are So Fed Up with Twitter Bots They’re Hunting Them Down Themselves
This is an interesting approach. Maybe Twitter shouldn't solve the fake accounts problem directly, maybe they should come up with an evaluation criteria and then create a market for identifying fake accounts.
If their evaluation criteria is good, they could get away with 0 cost to build the best possible system (motivated by competition on a market).
There's an open call for papers/proposals for handling the deluge. "Funding will be provided as an unrestricted gift to the proposer's organization(s)" ... "Twitter Health Metrics Proposal Submission" https://blog.twitter.com/official/en_us/topics/company/2018/...
A real hacker move would be to just leave Twitter and go to Mastodon https://joinmastodon.org
Are you suggesting that Mastodon has a better system for identifying harassment, spam, and spam accounts? Or that, given that they're mostly friendly early adopters, they haven't yet encountered the problem?
It seems to me like you don't understand the crucial difference between Twitter and Mastodon.
There's no such thing as Mastodon, a singular social network. Mastodon is a series of instances that talk to each other. A sysadmin running the instance can do whatever he pleases in his instance, including closing the registration, banning entire instances from communicating with his instance, and enforcing whichever rules he wants to enforce.
Mastodon doesn't deal with such issues at all. It's sysadmins running Mastodon instances that are supposed to deal with such issues.
It's more like reddit, where mods of subreddits have nearly complete authority over their own space on the social network, than it is like Twitter, in which a single entity is in charge.
Mastodon is a federated system like StatusNet/GNU Social.
So, in your opinion, Mastodon nodes - by virtue of being federated - would be better equipped to handle the spam and harassment volume that Twitter is subject to?
I find that hard to believe.
ActivityPub (and OStatus, and ActivityStreams/Salmon, and OpenSocial) are all great specs and great ideas. Hosting and moderation cost real money (which spammers/scammers are wasting).
Know what's also great? Learning. For learning, we have the xAPI/TinCan spec and also schema.org/Action.
“We’re committing Twitter to increase the health and civility of conversation”
First Amendment protections apply to suits brought by the government. Civil suits are required to prove damages ("quantum of loss").
There are many open platforms. (I've contributed to those as well). Some are built on open standards. None of said open platforms have procedures or resources for handling the onslaught of disrespectful trash that the people we've raised eventually use these platforms for communicating at other people who have feelings and understand the Golden Rule.
https://en.wikipedia.org/wiki/Golden_Rule
The initial early adopters (who have other better things to do) are fine: helpful, caring, critical, respectful; healthy. And then everyone else comes surging in with hate, disrespect, and vitriol; unhealthy. They don't even realize that being hateful and disrespectful is making them more depressed. They think that complaining and talking smack to people is changing the world. And then they turn off the phone or log out of the computer, and carry on with their lives.
No-one taught them to be the positive, helpful energy they want to attract from the world. No-one properly conditioned them to either respectfully disagree according to the data or sit down and listen. No-one explained to them that a well-founded argument doesn't fit in 140 or 280 characters, but a link and a headline do. No-one explained to them that what they write on the internet lasts forever and will be found by their future interviewers, investors, jurors, and voters. No-one taught them that being respectful and helpful in service of other people - of the group's success, of peaceful coexistence - is the way to get ahead AND be happy. "No-one told me that."
Shareholders of public corporations want to see growth in meaningless numbers, foreign authoritarian governments see free expression as a threat to their ever-so-fragile self-perceptions, political groups seek to frame and smear and malign and discredit (because they are so in need of group acceptance; because money still isn't making them happy), and there are children with too much free time reading all of these.
No-one is holding these people accountable: we need transparency and accountability. We need to focus on more important goals and feel good about helping; about volunteering our time to help others be happier.
Instead, now that these haters and scam artists have all self-identified, we must spend our time conditioning their communications until they learn to respectfully disagree on facts and data or go somewhere else. "That's how you feel? Great. How does that make your victim feel?" is the confrontation that some people are seeking from companies that set out to serve free speech and provide a forum for citizens to share the actual news.
Who's going to pay for that? Can they sue for their costs and losses? Advertisers do not want a spot next to hateful and disrespectful.
"How dare you speak of censorship in such veiled terms!?" Really? They're talking about taking down phrases like "kill" and "should die"; not phrases like "I disagree because:"
So, now, because there are so many hateful economically disadvantaged people in the world with nothing better to do and no idea how to run a business or keep a job with benefits, these companies need to staff 24 hour a day censors to take down the hate and terror and gang recruiting within one hour. What a distorted mirror of our divisively fractured wealth inequality, indeed.
"Ban gangs ASAP, please: they'll just go away"
How much does it cost to pay prison labor to redundantly respond to this trash? Are those the skills they need to choose a different career with benefits and savings that meet or exceed inflation when they get out?
What is the procedure for referring threats of violence to justice in your jurisdiction? Are there wealthy individuals in your community who would love to contribute resources to this effort? Maybe they have some region-specific pointers for helping the have-nots out here trolling like it's going to get them somewhere they want to be in life?
Let me share a little story with you:
A person walks into a bar/restaurant, flicks off the bartender/waiter, orders 5 glasses of free water, starts plastering ads to the walls and other peoples' tables, starts making threats to groups of people cordially conversing, and walks out.
Gitflow – Animated in React
Thanks! A command log would be really helpful too.
The HubFlow docs contain GitFlow docs and some really helpful diagrams: https://datasift.github.io/gitflow/IntroducingGitFlow.html
I change the release prefix to 'v' so that the git tags for the release look like 'v0.0.1' and 'v0.1.0':
git config --replace-all gitflow.prefix.versiontag v
git config --replace-all hubflow.prefix.versiontag v
Sure.. will add the command log
Ask HN: How feasible is it to become proficient in several disciplines?
For example to become a professional in:
- back-end api development
- DevOps
- Data Engineer (big data, data science, ML, etc)
It is feasible, though as with any type of specialization, you're then a "jack of all trades, master of none". Maybe a title like "Full Stack Data Engineer" would be descriptive.
You could write an OAuth API for accepting and performing analysis of datasets (model fitting / parameter estimation; classification or prediction), write a test suite, write Kubernetes YAML for a load-balanced geodistributed dev/test/prod architecture, and continuously deploy said application (from branch merges, optionally with a manual confirmation step; e.g. with GitLab CI) and still not be an actual Data Engineer.
After rising for 100 years, electricity demand is flat
Cryptocurrency mining is about to consume more electricity than home usage in Iceland.[1] I assume it's similar in other places that have cheaper electricity.
Seems that power companies should encourage consumers to mine Bitcoin. Problem solved.
[1] https://qz.com/1204840/iceland-will-use-more-electricity-min...
> Seems that power companies should encourage consumers to mine Bitcoin. Problem solved.
Blockchains will likely continue to generate considerable demand for electricity for the foreseeable future.
Blockchain firms can locate where energy is cheapest. Currently that's in countries where energy prices go negative due to excess capacity and insufficient energy storage resources (batteries, [hemp/graphene] supercapacitors, water towers).
With continued demand, energy companies can continue to invest in new clean energy generation alternatives.
Unfortunately, in the current administration's proposed budget, funding for ARPA-E is cancelled and allocated to clean coal; which Canada, France, and the UK are committed to phasing out entirely by ~2030.
A framework for evaluating data scientist competency
Something HTML with local state like "Programmer Competency Matrix" would be great.
http://sijinjoseph.com/programmer-competency-matrix/
Levi Strauss to use lasers instead of people to finish jeans
Chaos Engineering: the history, principles, and practice
awesome-chaos-engineering lists a bunch of chaos engineering resources and tools such as Gremlin: https://github.com/dastergon/awesome-chaos-engineering
Scientists use an atomic clock to measure the height of a mountain
Quantum_clock#More_accurate_experimental_clocks: https://en.wikipedia.org/wiki/Quantum_clock#More_accurate_ex...
> In 2015 JILA evaluated the absolute frequency uncertainty of their latest strontium-87 optical lattice clock at 2.1 × 10−18, which corresponds to a measurable gravitational time dilation for an elevation change of 2 cm (0.79 in) on planet Earth that according to JILA/NIST Fellow Jun Ye is "getting really close to being useful for relativistic geodesy".
AFAIU, this type of geodesy isn't possible with 'normal' time structs. Are nanoseconds enough?
"[Python-Dev] PEP 564: Add new time functions with nanosecond resolution" https://mail.python.org/pipermail/python-dev/2017-October/14...
The thread you linked makes a good point: there isn't really any reason to care about the actual time, only the relative time. These sorts of clocks just use 256- or 512-bit counters; it's not like they're having overflow issues.
Resources to learn project management best practices?
My side project is beginning to attract interest from a few people who would like to hop on board. At this point I am just doing what feels familiar and sensible, but the project manager perspective is new to me. Are there any sort of articles/books/podcasts/etc that could clue me into how to become better at it?
Project Management: https://wrdrd.github.io/docs/consulting/software-development... ... #requirements-traceability, #work-breakdown-structure (Mission, Project, Goal/Objective #n; Issue #n, - [ ] Task)
"Ask HN: How do you, as a developer, set measurable and actionable goals?" https://westurner.github.io/hnlog/#story-15119635
- Burndown Chart, User Stories
... GitHub and GitLab have milestones and reorderable issue boards. I still like https://waffle.io for complexity points; though you can also just create labels for e.g. complexity (Complexity-5) and priority (Priority-5).
Ask HN: Thoughts on a website-embeddable, credential validating service?
Reading Troy Hunt's password release V2 blog post [0], I came across the NIST recommendation to prevent users from creating accounts with passwords discovered in data breaches. This got me thinking: would a website admin (ex. small business owner with a custom website) benefit from a service that validates user passwords? The idea is to create a registration iframe with forms for email, password, etc., which would check hashed credentials against a database of data from breaches. Additionally, client-side validation would enforce rules recommended by the NIST's Digital Identity Guidelines [1], which would relieve admins from implementing their own rules. I'm sure there are additional security features that can be added.
1. Have you seen a need for this type of service, and could you see this being adopted at all?
2. Do you know of a service like this? I've looked, no hits so far.
3. Does the architecture seem sound?
[0]: https://www.troyhunt.com/ive-just-launched-pwned-passwords-version-2/
[1]: https://www.nist.gov/itl/tig/projects/special-publication-800-63
blockchain-certificates/cert-verifier-js: https://github.com/blockchain-certificates/cert-verifier-js
> A library to enable parsing and verifying a Blockcert. This can be used as a node package or in a browser. The browserified script is available as verifier.js.
https://github.com/blockchain-certificates/cert-issuer
> The cert-issuer project issues blockchain certificates by creating a transaction from the issuing institution to the recipient on the Bitcoin blockchain that includes the hash of the certificate itself.
... We could/should also store X.509 cert hashes in a blockchain.
Exactly what part of such a service would benefit from anything related to a blockchain?
Are you asking me why blockcerts stores certs in a blockchain?
Or whether using certs (really long passwords) is a better option than submitting unhashed passwords on a given datetime to a third-party in order to make sure they're not in the pwned passwords tables?
I was just reading about a company trying to make self-sovereign identity including actual certs (like degrees and such) an accessible and widely applicable/acceptable technology using Ethereum blockchain. I thought it showed some real practicality and promise. I believe it begins with U- forgot the name.Perhaps UPort? Anyhow, I'd be interested in hearing from anyone here about why that might be a bad or good idea. I don't personally have the skill in that tech to know.
Known Traveler Digital Identity system is a "new model for airport screening and security that uses biometrics, cryptography and distributed ledger technologies."
Blockcerts are for academic credentials, AFAIU.
[EDIT]
Existing blockchains have a limited TPS (transactions per second) for writes; but not for reads. Sharding and layer-2 (sidechains) do not have the same assurances. I'm sure we all remember how cryptokitties congested the txpool during the Bitcoin futures launch.
Thank you. I looked into "clear", one of the airport known traveler ID systems. (I'm assuming there are others) It's pretty cool/concerning. Takes ~ 8 min to load a traveler into its system. Thanks for reminding me of TPS---the info on read vs write.
Ask HN: What's the best algorithms and data structures online course?
These aren't courses, but from answers to "Ask HN: Recommended course/website/book to learn data structure and algorithms" :
Data Structure: https://en.wikipedia.org/wiki/Data_structure
Algorithm:https://en.wikipedia.org/wiki/Algorithm
Big O notation:https://en.wikipedia.org/wiki/Big_O_notation
Big-O Cheatsheet: http://bigocheatsheet.com
Coding Interview University > Data Structures: https://github.com/jwasham/coding-interview-university/blob/...
OSSU: Open Source Society University > Core CS > Core Theory > "Algorithms: Design and Analysis, Part I" [&2] https://github.com/ossu/computer-science/blob/master/README....
"Algorithms, 4th Edition" (2011; Sedgewick, Wayne): https://algs4.cs.princeton.edu/
Complexity Zoo > Petting Zoo (P, NP,): https://complexityzoo.uwaterloo.ca/Petting_Zoo
While perusing awesome-awesomeness [1], I found awesome-algorithms [2] , algovis [3], and awesome-big-o [4].
[1] https://github.com/bayandin/awesome-awesomeness
[2] https://github.com/tayllan/awesome-algorithms
Using Go as a scripting language in Linux
I, too, didn't realize that shebang parsing is implemented in the `binfmt_script` kernel module.
Does this persist across reboots?
echo ':golang:E::go::/usr/local/bin/gorun:OC' | sudo tee /proc/sys/fs/binfmt_misc/registerNo, but different init systems may autoload formats based on some configuration files. Systemd, for instance: https://www.freedesktop.org/software/systemd/man/systemd-bin...
Guidelines for enquiries regarding the regulatory framework for ICOs [pdf]
This is a helpful table indicating whether a Payment, Utility, Asset, or Hybrid coin/token: is a security, qualifies under Swiss AML payment law.
The "Minimum information requirements for ICO enquiries" appendix seems like a good set of questions for evaluating ICOs. Are there other good questions to ask when considering whether to invest in a Payment, Utility, Asset, or Hybrid ICO?
Are US regulations different from these clear and helpful regulatory guidelines for ICOs in Switzerland?
> Are there other good questions to ask when considering whether to invest in a Payment, Utility, Asset, or Hybrid ICO?
This paper doesn seem to cover that, only on how regulators should treat investments.
On investing in ICOs, the questions are the same as any other IPO. And, in most cases, just speculation.
The Benjamin Franklin method for learning more from programming books
> Read your programming book as normal. When you get to a code sample, read it over
> Then close the book.
> Then try to type it up.
According to a passage in "The Autobiography of Benjamin Franklin" (1791) regarding re-typing from "The Spectator"
https://en.wikipedia.org/wiki/The_Autobiography_of_Benjamin_...
Avoiding blackouts with 100% renewable energy
I notice that cases A and C require batteries for storage.
Should there be a separate entry for new gen supercapacitors? Supercapacitors built with both graphene and hemp have different Max Charge Rate (GW), Max Discharge Rate (GW), and Storage (TWh) capacities than even future-extrapolated batteries and current supercapacitors.
https://en.wikipedia.org/wiki/Supercapacitor
The cost and capabilities stats in this article look very promising:
"Hemp Carbon Makes Supercapacitors Superfast” https://www.asme.org/engineering-topics/articles/energy/hemp...
> “Our device’s electrochemical performance is on par with or better than graphene-based devices,” Mitlin says. “The key advantage is that our electrodes are made from biowaste using a simple process, and therefore, are much cheaper than graphene.”
> Graphene is, however, expensive to manufacture, costing as much as $2,000 per gram. [...] developed a process for converting fibrous hemp waste into a unique graphene-like nanomaterial that outperforms graphene. What’s more, it can be manufactured for less than $500 per ton.
> Hemp fiber waste was pressure-cooked (hydrothermal synthesis) at 180 °C for 24 hours. The resulting carbonized material was treated with potassium hydroxide and then heated to temperatures as high as 800 °C, resulting in the formation of uniquely structured nanosheets. Testing of this material revealed that it discharged 49 kW of power per kg of material—nearly triple what standard commercial electrodes supply, 17 kW/kg.
https://scholar.google.com/scholar?hl=en&q=hemp+supercapacit...
To be clear, supercapacitors are an alternative to li-ion batteries.
"Matching demand with supply at low cost in 139 countries among 20 world regions with 100% intermittent wind, water, and sunlight (WWS) for all purposes" (Renewable Energy, 2018) https://web.stanford.edu/group/efmh/jacobson/Articles/I/Comb...
Ask HN: What are some common abbreviations you use as a developer?
These are called 'codelabels'. They're great for prefix-tagging commit messages, pull requests, and todo lists:
BLD: build
BUG: bug
CLN: cleanup
DOC: documentation
ENH: enhancement
ETC: config
PRF: performance
REF: refactor
RLS: release
SEC: security
TST: test
UBY: usability
DAT: data
SCH: schema
REQ: requirement
REQ: request
ANN: announcement
STORY: user story
EPIC: grouping of user stories
There's a table of these codelabels here: https://wrdrd.github.io/docs/consulting/software-development...
Someday TODO FIXME XXX I'll get around to:
- [ ] DOC: create a separate site/organization for codelabels
- [ ] ENH: a tool for creating/renaming GitHub labels with unique foreground and background colors
YAGNI: Ya' ain't gonna need it
LOL, lulz
DRY: Don't Repeat Yourself
KISS: Keep It Super Simple
MVC: Model-View-Controller
MVT: Model-View-Template
MVVM: Model-View-View-Model
UI: User Interface
UX: User Experience
GUI: Graphical User Interface
CLI: Command Line Interface
CAP: Consistency, Availability, Partition tolerance
DHT: Distributed Hash Table
ETL: Extract, Transform, and Load
ESB: Enterprise Service Bus
MQ: Message Queue
VM: Virtual Machine
LXC: Linux Containers
[D]VCS, RCS: [Distributed] Version/Revision Control System
XP: Extreme Programming
CI: Continuous Integration
CD: Continuous Deployment
TDD: Test-Driven Development
BDD: Behavior-Driven Development
DFS, BFS: Depth/Breadth First Search
CRM: Customer Relationship Management
CMS: Content Management System
LMS: Learning Management System
ERP: Enterprise Resource Planning system
HTTP: Hypertext Transfer Protocol
HTTP STS: HTTP Strict Transport Security
REST: Representational State Transfer
API: Application Programming Interface
HTML: Hypertext Markup Language
DOM: Document Object Model
LD: Linked Data
LOD: Linked Open Data
URI: Uniform Resource Indicator
URN: Uniform Resource Name
URL: Uniform Resource Locator
UUID: Universally Unique Identifier
RDF: Resource Description Format
RDFS: RDF Schema
OWL: Web Ontology Language
JSON-LD: JSON Linked Data
JSON: JavaScript Object Notation
CSVW: CSV on the Web
CSV: Comma Separated Values
CIA: Confidentiality, Integrity, Availability
ACL: Access Control List
RBAC: Role-Based Access Control
MAC: Mandatory Access Control
CWE: Common Weakness Enumeration
CVE: Common Vulnerabilities and Exposures
XSS: Cross-Site Scripting
CSRF: Cross-Site Request Forgery
SQLi: SQL Injection
ORM: Object-Relational Model
AUC: Area Under Curve
ROC: Receiver Operating Characteristic
DL: Description Logic
RL: Reinforcement Learning
CNN: Convolutional Neural Network
DNN: Deep Neural Network
IS: Information Systems
ROI: Return on Investment
RPU: Revenue per User
MAU: Monthly Active Users
DAU: Daily Active Users
STEM: Science, Technology, Engineering, Mathematics/Medicine
STEAM: STEM + Arts
W3C: World-Wide-Web Consortium
GNU: GNU's not Unix
WRDRD: WRD R&D
... The Sphinx ``.. index::`` directive makes it easy to include index entries for acronym forms, too https://wrdrd.github.io/docs/genindex
There Might Be No Way to Live Comfortably Without Also Ruining the Planet
"A good life for all within planetary boundaries" (2018) https://www.nature.com/articles/s41893-018-0021-4
> Abstract: Humanity faces the challenge of how to achieve a high quality of life for over 7 billion people without destabilizing critical planetary processes. Using indicators designed to measure a ‘safe and just’ development space, we quantify the resource use associated with meeting basic human needs, and compare this to downscaled planetary boundaries for over 150 nations. We find that no country meets basic needs for its citizens at a globally sustainable level of resource use. Physical needs such as nutrition, sanitation, access to electricity and the elimination of extreme poverty could likely be met for all people without transgressing planetary boundaries. However, the universal achievement of more qualitative goals (for example, high life satisfaction) would require a level of resource use that is 2–6 times the sustainable level, based on current relationships. Strategies to improve physical and social provisioning systems, with a focus on sufficiency and equity, have the potential to move nations towards sustainability, but the challenge remains substantial.
> "Radical changes are needed if all people are to live well within the limits of the planet," [...]
> "These include moving beyond the pursuit of economic growth in wealthy nations, shifting rapidly from fossil fuels to renewable energy, and significantly reducing inequality.
> "Our physical infrastructure and the way we distribute resources are both part of what we call provisioning systems. If all people are to lead a good life within the planet's limits then these provisioning systems need to be fundamentally restructured to allow for basic needs to be met at a much lower level of resource use."
Perhaps ironically, our developments in service of sustainability (resource efficiency) needs for a civilization on Mars are directly relevant to solving these problems on Earth.
Recycle everything.
Survive without soil, steel, hydrocarbons, animals, oxygen.
Convert CO2, sunlight, H20, and geothermal energy to forms necessary for life.
https://en.wikipedia.org/wiki/Colonization_of_Mars
Algae, carbon capture, carbon sequestration, lab grown plants, water purification, solar power, [...]
Mars requires a geomagnetic field in order to sustain an atmosphere in order to [...].
"The Limits to Growth" (1972, 2004) [1] very clearly forecasts these same unsustainable patterns of resource consumption: 'needs' which exceed and transgress our planetary biophysical boundaries.
The 17 UN Sustainable Development Goals (#GlobalGoals) [2] outline our worthwhile international objectives (Goals, Targets, and Indicators). The Paris Agreement [3] sets targets and asks for commitments from nation states (and businesses) to help achieve these goals most efficiently and most sustainably.
In the US, the Clean Power Plan [4] was intended to redirect our national resources toward renewable energy with far less external costs. Direct and indirect subsidies for nonrenewables are irrational. Are subsidies helpful or necessary to reach production volumes of renewable energy products and services?
There are certainly financial incentives for anyone who chooses to invest in solving for the Global Goals; and everyone can!
[1] https://en.wikipedia.org/wiki/The_Limits_to_Growth
[2] http://www.un.org/sustainabledevelopment/sustainable-develop...
Multiple GWAS finds 187 intelligence genes and role for neurogenesis/myelination
> We found evidence that neurogenesis and myelination—as well as genes expressed in the synapse, and those involved in the regulation of the nervous system—may explain some of the biological differences in intelligence.
re: nurture, hippocampal plasticity and hippocampal neurogenesis also appear to be affected by dancing and omega-3,6 (which are transformed into endocannabinoids by the body): https://news.ycombinator.com/item?id=15109698
Could we solve blockchain scaling with terabyte-sized blocks?
These numbers in a computational model (or even Jupyter notebooks) would be useful.
We may indeed need fractional satoshis ('naks').
With terabyte blocks, lightning network would be unnecessary: at least for TPS.
There will need to be changes to account for quantum computing capabilities somewhere in the future timeline of Bitcoin (and everything else in banking and value-producing industry). Probably maybe a different hash function instead of just a routine difficulty increase (and definitely something other than ECDSA, which isn't a primary cost). $1.3m/400k a year to operate a terabyte mining rig with 50Gbps bandwidth would affect decentralization; though maybe not any more than it already is affected now.
https://en.bitcoin.it/wiki/Weaknesses#Attacker_has_a_lot_of_... (51%)
Confidence intervals for these numbers would be useful.
Casper PoS and beyond may also affect future Bitcoin volume estimates.
Ask HN: Do you have ADD/ADHD? How do you manage it?
Also, how has it affected your CS career? I feel that transitioning to management would help, as it does not require lengthy periods of concentration, but rather distributed attention for shorter periods.
Music. Headphones. Chillstep, progressive, chillout etc. from di.fm. Long mixes from SoundCloud with and without vocals. "Instrumental"
Breathe in through the nose and out through the mouth.
Less sugar and processed foods. Though everyone has a different resting glucose level.
Apparently it's called alpha-pinene.
Fidget things. Rubberband, paperclip.
The Pomodoro Technique: work 25 minutes, chill for 5 (and look at something at least 20 feet away (20-20-20 rule))
Lists. GTD. WBS.
Exercise. Short walks.
Ask HN: How to understand the large codebase of an open-source project?
Hello All!
what are techniques you all used to learn and understand a large codebase? what are the tools you use?
What is the best way to learn to code from absolute scratch?
We have been hosting a Ugandan refugee in our home in Oakland for the past 9 months and he wants to learn how to code.
Where is the best place for him to start from absolute scratch? What resources can we point him to? Who can help?
Here's an answer to a similar question: "Ask HN: How to introduce someone to programming concepts during 12-hour drive?" https://news.ycombinator.com/item?id=15454421
https://learnxinyminutes.com/docs/python3/ (Python3)
https://learnxinyminutes.com/docs/javascript/ (Javascript)
https://learnxinyminutes.com/docs/git/ (Git)
https://learnxinyminutes.com/docs/markdown/ (Markdown)
Read the docs. Read the source. Write docstrings. Write automated tests: that's the other half of the code.
Keep a journal of your knowledge as e.g. Markdown or ReStructuredText; regularly pull the good ones from bookmarks and history into an outline.
I keep a tools reference doc with links to Wikipedia, Homepage, Source, Docs: https://wrdrd.github.io/docs/tools/
And a single-page log of my comments: https://westurner.github.io/hnlog/
> To get a job, "Coding Interview University": https://github.com/jwasham/coding-interview-university
Wow, thank you so much. This is incredibly useful.
Tesla racing series: Electric cars get the green light – Roadshow
Tesla Racing Circuit ideas for increasing power discharge rate, reducing heat, and reducing build weight:
Hemp supercapacitors (similar power density as graphene supercapacitors and li-ion, lower cost than graphene)
Active cooling. Modified passive cooling.
Biocomposite frame and panels (stronger and lighter than steel and aluminum (George Washington Carver))
> Biocomposite frame and panels (stronger and lighter than steel and aluminum (George Washington Carver))
"Soybean Car" (1941) https://en.wikipedia.org/wiki/Soybean_car
What happens if you have too many jupyter notebooks?
These days there is a tendency in data analysis to use Jupyter Notebooks. But what happens if you have too many jupyter notebooks? For example, there are more than a hundred.
Actually, you start creating some modules. However, it is less convenient to work with them compared to what was before. It happens that you should code in web interface, somewhere in similar to the notepad++ form or you should change your IDLE.
Personally, I work in Pycharm and so far I couldn't assess remote interpreter or VCS. It is because pickle files or word2vec weighs too much (3gb+) and so I don't want to download/upload them. Also Jupyter is't cool in pycharm.
Do you have better practices in your companies? How to correctly adjust IDLE? Do you know about any possible substitution for the IPython notebook in the world of data analysis?
> what happens if you have too many jupyter notebooks? For example, there are more than a hundred.
Like anything else, Jupyter Notebook is limited by the CPU and RAM of the system hosting the Tornado server and Jupyter kernels.
At 100 notebooks (or even just one), it may be a good time to factor common routines into a packaged module with tests and documentation.
It's actually possible (though inefficient) to import code from Jupyter notebooks with ipython/ipynb (pypi:ipynb): https://github.com/ipython/ipynb ( https://jupyter-notebook.readthedocs.io/en/stable/examples/N... )
> Actually, you start creating some modules. However, it is less convenient to work with them compared to what was before. It happens that you should code in web interface, somewhere in similar to the notepad++ form or you should change your IDLE.
The Spyder IDE has support for .ipynb notebooks converted to .py (which have the IPython prompt markers in them). Spyder can connect an interpreter prompt to a running IPython/Jupyter kennel. There's also a Spyder plugin for Jupyter Notebook: https://github.com/spyder-ide/spyder-notebook
> Personally, I work in Pycharm and so far I couldn't assess remote interpreter or VCS. It is because pickle files or word2vec weighs too much (3gb+) and so I don't want to download/upload them.
Remote data access times can be made faster by increasing the space efficiency of the storage format, increasing the bandwidth of the connection, moving the data to the code, or moving the code to the data.
> Do you have better practices in your companies?
There are a number of [Reproducible] Data Science cookiecutter templates which have a directory for notebooks, module packaging, and Sphinx docs: https://cookiecutter.readthedocs.io/en/latest/readme.html#da...
Refactoring increases testability and code reuse.
> How to correctly adjust IDLE?
I don't think I understand the question?
"Configuring IPython" https://ipython.readthedocs.io/en/stable/config/index.html
Jupyter > "Installation, Configuration, and Usage" https://jupyter.readthedocs.io/en/latest/projects/content-pr...
> Do you know about any possible substitution for the IPython notebook in the world of data analysis?
From https://en.wikipedia.org/wiki/Notebook_interface :
> > "Examples of the notebook interface include the Mathematica notebook, Maple worksheet, MATLAB notebook, IPython/Jupyter, R Markdown, Apache Zeppelin, Apache Spark Notebook, and the Databricks cloud."
There are lots of Jupyter kernels for different tools and languages (over 100; including for other 'notebook interfaces'): https://github.com/jupyter/jupyter/wiki/Jupyter-kernels
And there are lots of Jupyter integrations and extensions: https://github.com/quobit/awesome-python-in-education/blob/m...
Cancer ‘vaccine’ eliminates tumors in mice
The article is about this study:
"Eradication of spontaneous malignancy by local immunotherapy" http://stm.sciencemag.org/content/10/426/eaan4488
> In situ vaccination with low doses of TLR ligands and anti-OX40 antibodies can cure widespread cancers in preclinical models.
Boosting teeth’s healing ability by mobilizing stem cells in dental pulp
Tideglusib
https://en.wikipedia.org/wiki/Tideglusib
> "Promotion of natural tooth repair by small molecule GSK3 antagonists" https://www.nature.com/articles/srep39654
> [...] Here we describe a novel, biological approach to dentine restoration that stimulates the natural formation of reparative dentine via the mobilisation of resident stem cells in the tooth pulp.
This Biodegradable Paper Donut Could Let Us Reforest the Planet
"These drones can plant 100,000 trees a day" https://news.ycombinator.com/item?id=16260892
Drones that can plant 100k trees a day
> It’s simple maths. We are chopping down about 15 billion trees a year and planting about 9 billion. So there’s a net loss of 6 billion trees a year.
This is a regional thing [0] though. We need to plant the trees in Latin America, Caribbean, and Sub-Saharan Africa. The rest of the world is gaining forests.
[0] http://www.telegraph.co.uk/news/2016/03/23/deforestation-whe...
Now, the question is, which industry sector cuts the largest percentage of trees. Now, answering that question, is there a way to do the same thing without cutting trees?
Answering that question and then executing a business plan is probably worth billions.
Planting trees to combat 6 billion trees lost every year is a pure expense, no profit to be made at all. At least not if you won't cut them down a few decades later.
A very good way to absorb atmospheric carbon is to plant new trees and cut down and use old ones for anything else but burning it.
"This Biodegradable Paper Donut Could Let Us Reforest The Planet" https://news.ycombinator.com/item?id=16261101
I think this kind of thing is funny for us westerners. I recently found this tech has been used in arid countries for possibly hundreds or thousands of years.
https://duckduckgo.com/?q=ollas+gardening&t=lm&iax=images&ia...
I do realize the ollas are for more permanent gardens, but it's the same concept. I do hope the paper version gets used, there are plenty of places that could benefit from it.
What are some YouTube channels to progress into advanced levels of programming?
There are some cool YouTube channel suggestions on https://news.ycombinator.com/item?id=16224165 But I wanted to know which of those are great to progress into advanced level of programming? Which of the channels teach advanced techniques?
These aren't channels, but a number of the links are marked with "(video)":
https://github.com/jwasham/coding-interview-university
https://github.com/jwasham/coding-interview-university/blob/...
Thanks
Multiple issue and pull request templates
+1
Default: /ISSUE_TEMPLATE.md
/ISSUE_TEMPLATE/<name>.md</name>
Default: /PULL_REQUEST_TEMPLATE.md
/PULL_REQUEST_TEMPLATE/<name>.md</name>
You can leave the last S for off (for savings) if you want /ISSUE_TEMPLATE(S)/ or /PULL_REQUEST_TEMPLATE(S)/
Five myths about Bitcoin’s energy use
Regarding the proof-of-stake part: Ethereum devs are also working on sharding which will make scaling on the Ethereum Blockchain way easier. I really think that Ethereum will be the No.1 cryptocurrency in the near future. Bitcoin devs showed plenty of times, that they are not capable of keeping pace with demand (couldn't even get the block size thing right). Bitcoin is virtually dead. No one can really use it for real world transactions. Plus Bitcoin in reality is a really central coin, completely in the hands of the miners. Even if the Bitcoin devs decided to go with PoS, the miners wouldn't agree.
Proof of Work (Bitcoin*, ...), Proof of Stake (Ethereum Casper), Proof of Space, Proof of Research (GridCoin, CureCoin,)
Plasma (Ethereum) and Lightning Network (BitCoin (SHA256), Litecoin (scrypt),) will likely offload a significant amount of transaction volume and thereby reduce the kWh/transaction metrics.
> But electricity costs matter even more to a Bitcoin miner than typical heavy industry. Electricity costs can be 30-70% of their total costs of operation.
> [...] If Bitcoin mining really does begin to consume vast quantities of the global electricity supply it will, it follows, spur massive growth in efficient electricity production—i.e. the green energy revolution. Moore’s Law was partially a story about incredible advances in materials science, but it was also a story about incredible demand for computing that drove those advances and made semiconductor research and development profitable. If you want to see a Moore’s-Law-like revolution in energy, then you should be rooting for, and not against, Bitcoin. The fact is that the Bitcoin network, right now, is providing a $200,000 bounty every 10 minutes (the mining reward) to the person who can find the cheapest energy on the planet.
This is ridiculous. The economy is already incentivized to find cheaper electricity by forces far more powerful than Bitcoin. By this logic, you would defend a craze for trying to boil the sea.
If the market had internalized the external health, environmental, and defense costs of nonrenewable energy, we would already have cheap, plentiful renewable energy. But we don't: the market is failing to optimize for factors other than margin. (New Keynesian economics admits market failure, but not non-rationality.)
So, (speculative_valuation - cost) is the margin. Whereas with a stock in a leveraged high-frequency market with shorting, (shareholder_equity - market_cap) is explainable in terms of the market information that is shared.
So, it's actually (~$200K-(n_kwhrs*cost_kwhr)) for whoever wins the block mining lottery (which is about every 10 minutes and can be anyone who's mining).
But the point about Bitcoin maintaining demand for and while we move to competitive lower cost renewable energy and greater efficiency is good.
What we should hope to see is the blockchain industry directly investing in clean energy capacity development in order to rationally minimize their primary costs and maximize environmental sustainability.
It is kind of missing the point though. If everyone was competing to make electrical single-person planes to help people commute to work it would also increase demand for renewable/electrical energy - but can we agree, that just using electrical cars instead of planes is going to take a lot less electricity overall?
> The U.S. Digital Service isn’t perfect, but it is clearly working. The team estimates that for every $1 million invested in USDS that the government will avoid spending $5 million and save thousands of labor hours. Over a five-year period, the team’s efforts will save $1.1 billion, redirect almost 2,000 labor years towards higher value work, and generate over 400 percent return on investment. Most importantly, USDS will continue to deliver better government services for the American people, including Veterans who deserve better.
> In the private sector, these kinds of numbers would not lead to a 50 percent cut in budget. Instead, you’d clearly invest further with that kind of return. Considering the ambitious goals set out in the President’s Management Agenda, the Trump Administration should double down on better support for the public, our troops, and our veterans. The best way to do that is clearly through investments like USDS.
Why would you halve the budget of a team that's yielding a more than 400% ROI (in terms of cost savings)?
https://en.wikipedia.org/wiki/United_States_Digital_Service
400% ROI to whom?
More data makes hiding corruption harder, so I'm not sure if it really gives 400% to the decision makers.
USDS reports 400% ROI in savings to the taxpayers who fund the government with tax revenue (instead of kicking the can down the road with debt financing) and improvements in customer service quality.
https://www.usaspending.gov (Federal Funding Accountability and Transparency Act of 2006 (Obama, McCain, Carper, Coburn)) has more fine-grained spending data, but not credit-free immutable distributed ledger transaction IDs, quantitative ROI stats, or performance.gov and #globalgoals goal alignment. We'd need a metadata field on spending bills to link to performance.gov and SDG goals, targets, and indicators.
"Transparency and Accountability"
IIRC, here on HN, I've mentioned a number of times -- and quoted in the full from -- the 13 plays of the USDS Digital Services Playbook; all of which are applicable to and should probably be required reading for all government IT and govtech: https://playbook.cio.gov/
There are forms with workflow states that need human review sometimes. USDS helps with getting those processes online in order to reduce costs, increase cost-efficiency, and increase quality of service.