Contents^
Items^
Ask HN: Relationship between set theory and category theory
I have an idea about the relationship between set theory and category theory and I would like some feedback. I would like others to see it too, and I don't know how to do it. I think it's at least interesting to look at as a slightly crazy collage, but I was a bit more excited than normal when the idea hit, so I just had to dump it all at once in this image: https://twitter.com/FamilialRhino/status/1101777965724168193 (You will have to zoom the picture in order to be able to read the scribbles.)
It has to do with resonance in the energy flowing in emergent networks. Can't quite put my finger on it, so I'll be here to answer any questions.
Thanks for reading.
"Categorical set theory" > "References" https://en.wikipedia.org/wiki/Categorical_set_theory#Referen...
From "Homotopy category" > "Concrete categories" https://en.wikipedia.org/wiki/Homotopy_category#Concrete_cat... :
> While the objects of a homotopy category are sets (with additional structure), the morphisms are not actual functions between them, but rather a classes of functions (in the naive homotopy category) or "zigzags" of functions (in the homotopy category). Indeed, Freyd showed that neither the naive homotopy category of pointed spaces nor the homotopy category of pointed spaces is a concrete category. That is, there is no faithful functor from these categories to the category of sets.
My "understanding" of category theory is extremely shallow, but that's exactly why I think my proposal makes sense. It is the kind of thing that everybody ignores for decades precisely because it's transparently obvious, like a fish that doesn't understand water.
Here is the statement:
The meaning of no category is every category.
reference: https://terrytao.wordpress.com/2008/02/05/the-blue-eyed-isla...
This was already understood by everybody in the field, no doubt. It's just that somebody has to actually say it to someone else in order for the symmetry to break. The link above has the exact description of this, from Terence Tao.
The most popular docker images each contain at least 30 vulnerabilities
Although vulnerability scanners can be a useful tool, I find it very troublesome that you can utter the sentence "this package contains XX vulnerabilities, and that package contains YY vulnerabilities" and then stop talking. You've provided barely any useful information!
The quantity of vulnerabilities in an image is not really all that useful information. A large amount of vulnerabilities in a Docker image does not necessarily imply that there's anything insecure going on. Many people don't realize that a vulnerability is usually defined as "has a CVE security advisory", and that CVEs get assigned based on a worst-case evaluation of the bug. As a result, having a CVE in your container barely tells you anything about your actual vulnerability position. In fact, most of the time you will find that having a CVE in some random utility doesn't matter. Most CVEs in system packages don't apply to most of your containers' threat models.
Why not? Because an attacker is very unlikely to be able to use vulnerabilities in these system libraries or utilities. Those utilities are usually not in active use in the first place. Even if they are used, you are not usually in a position to exploit these vulnerabilities as an attacker.
Just as an example, a hypothetical outdated version of grep in one of these containers can hypothetically contain many CVEs. But if your Docker service doesn't use grep, then you would need to manually run grep to be vulnerable. And an attacker that is able to run grep in your Docker container has already owned you - it doesn't make a difference that your grep is vulnerable! This hypothetical vulnerable version of grep therefore makes no difference in the security of your container, despite containing many CVEs.
It's the quality of these vulnerabilities that matters. Can an attacker actually exploit the vulnerabilities to do bad things? The answer for almost all of these CVEs is "no". But that's not really the product that Snyk sells - Snyk sells a product to show you as many vulnerabilities as possible. Any vulnerability scanner company thinks it can provide most business value (and make the most money) by reporting as many vulnerabilities as it can. For sure it can help you to pinpoint those few vulnerabilities that are exploitable, but that's where your own analysis comes in.
I'm not saying there's not a lot to improve in terms of container security. There's a whole bunch to improve there. But focusing on quantities like "amount of CVEs in an image" is not the solution - it's marketing.
This entire depending on a base container for setup and then essentially throwing away everything you get from the package manager is part of the issue. There is no real package management for Docker. Hell, there isn't even an official way to determine if your image needs an upgrade (I wrote a ruby script that gets the latest tags from a docker repo, extracts the ones with numbers and sorts them in order and compares them to what I have running).
Relying on an Alpine/Debian/Ubuntu base helps to get dependencies installed quickly. Docker could have just created their own base distro and some mechanism to track package updates across images, but they did not.
There are guides for making bare containers, they contain nothing .. no ip, grep, bash .. only the bare minimum libraries and requirements to run your service. They are minimal, but incredibly difficult to debug (sysdig still sucks unless you shell out money for enterprise).
I feel like containers are alright, but Docker is a partial dumpster fire. cgroup isolation is good, the crazy way we deal with packages in container systems is not so good.
Sure if you're just checking for base-distro packages for security vulnerabilities, you're going to find security issues that don't apply (e.g. an exploit in libpng even though your container runs nothing that even links to libpng), but it does excuse the whole issue with the way containers are constructed.
I think this space is really open too, for people to find better systems that are also portable: image formats that are easy to build, easy to maintain dependencies for, and easy to run as FreeBSD jails OR Linux cgroup managed containers (Docker for FreeBSD translated images to jails, but it's been unmaintained for years).
I agree the tooling is a tire fire :(
> e.g. an exploit in libpng even though your container runs nothing that even links to libpng
it's a problem because some services or api's could be abused giving the attacker a path to these vulnerable resources and then use that vulnerability regardless if it is used currently by your service.
I like my images to only contain what's absolutely needed to do that job only. It's not so difficult to do, provided people would be willing to architect systems from the ground up, instead of pulling in a complete debian or fedora installation and then removing things (that should be outlawed imho lol). Not only do I get less attack surface but also smaller updates (which again then is incentive to do more often), less complexity, less logs, easier auditabile (now every log file or even log line might give valid clues), faster incident response, easier troubleshooting, sorry for going on and on ...
It's a cultural problem too: where people working in an environment where it's normal to have every command available on a production system (really?), and where there is no barrier to install anything new that is "required" without discussion & peer review (what are we pair programming for?) or where nobody even tracks the dead weight in production or whether configs are locked down?.
I sometimes think many companies lost control over this long ago. [citation needed] :(
I'm not familiar with Docker infrastructure but what is the alternative to "pulling in a complete debian or fedora installation and then removing things"? Compiling your own kernel and doing the whole "Linux From Scratch" thing? Isn't that incredibly time-intensive to do for every single container?
Just have an image with very a minimal user land. Compiling your own kernel is irrelevant because you need the host kernel to run the container, and container images don't contain a kernel.
The busybox image is a good starting point. Take that, then copy your executables and libraries. If you are willing to go further, you can rather easily compile your own busybox with most utilities stripped out. It's not time intensive because you need to do it just once, and it takes just an afternoon to figure out how.
I don't think this is a tooling problem at all.
"The tooling makes it too easy to do it wrong." Compared to shell scripts with package manager invocations? Nobody configures a system with just packages: there are always scripts to call, chroots to create, users and groups to create, passwords to set, firewall policies to update, etc.
There are a bunch of ways to create LXC containers: shell scripts, Docker, ansible. Shell scripts preceded Docker: you can write a function to stop, create an intermediate tarball, and then proceed (so that you don't have to run e.g. debootstrap without a mirror every time you manually test your system build script; so that you can cache build steps that completed successfully).
With Docker images, the correct thing to do is to extend FROM the image you want to use, build the whole thing yourself, and then tag and store your image in a container repository. Neither should you rely upon months-old liveCD images.
"You should just build containers on busybox." So, no package management? A whole ensemble of custom builds to manually maintain (with no AppArmor or SELinux labels)? Maintainers may prefer for distros to field bug reports for their own common build configurations and known-good package sets. Please don't run as root in a container ("because it's only a container that'll get restarted someday"). Busybox is not a sufficient OS distribution.
It's not the tools, it's how people are choosing to use them. They can, could, and should try and use idempotent package management tasks within their container build scripts; but they don't and that's not Bash/Ash/POSIX's fault either.
> With Docker images, the correct thing to do is to extend FROM the image you want to use, build the whole thing yourself, and then tag and store your image in a container repository. Neither should you rely upon months-old liveCD images.
This should rebuild all. There should be an e.g. `apt-get upgrade -y && rm -rf /var/lib/apt/lists` in there somewhere (because base images are usually not totally current (and neither are install ISOs)).
`docker build --no-cache --pull`
You should check that each Dockerfile extends FROM `tag:latest` or the latest version of the tag that you support. Its' not magical, you do have to work it.
Also, IMHO, Docker SHOULD NOT create another Linux distribution.
Tinycoin: A small, horrible cryptocurrency in Python for educational purposes
The 'dumbcoin' jupyter notebook is also a good reference: "Dumbcoin - An educational python implementation of a bitcoin-like blockchain" https://nbviewer.jupyter.org/github/julienr/ipynb_playground...
When does the concept of equilibrium work in economics?
"Modeling stock return distributions with a quantum harmonic oscillator" (2018) https://iopscience.iop.org/article/10.1209/0295-5075/120/380...
> We propose a quantum harmonic oscillator as a model for the market force which draws a stock return from short-run fluctuations to the long-run equilibrium. The stochastic equation governing our model is transformed into a Schrödinger equation, the solution of which features "quantized" eigenfunctions. Consequently, stock returns follow a mixed χ distribution, which describes Gaussian and non-Gaussian features. Analyzing the Financial Times Stock Exchange (FTSE) All Share Index, we demonstrate that our model outperforms traditional stochastic process models, e.g., the geometric Brownian motion and the Heston model, with smaller fitting errors and better goodness-of-fit statistics. In addition, making use of analogy, we provide an economic rationale of the physics concepts such as the eigenstate, eigenenergy, and angular frequency, which sheds light on the relationship between finance and econophysics literature.
"Quantum harmonic oscillator" https://en.wikipedia.org/wiki/Quantum_harmonic_oscillator
The QuantEcon lectures have a few different multiple agent models:
"Rational Expectations Equilibrium" https://lectures.quantecon.org/py/rational_expectations.html
"Markov Perfect Equilibrium" https://lectures.quantecon.org/py/markov_perf.html
"Robust Markov Perfect Equilibrium" https://lectures.quantecon.org/py/rob_markov_perf.html
"Competitive Equilibria of Chang Model" https://lectures.quantecon.org/py/chang_ramsey.html
... "Lectures in Quantitative Economics as Python and Julia Notebooks" https://news.ycombinator.com/item?id=19083479 (data sources (pandas-datareader, pandaSDMX), tools, latex2sympy)
"Econophysics" https://en.wikipedia.org/wiki/Econophysics
> Indeed, as shown by Bruna Ingrao and Giorgio Israel, general equilibrium theory in economics is based on the physical concept of mechanical equilibrium.
Simdjson – Parsing Gigabytes of JSON per Second
> Requirements: […] A processor with AVX2 (i.e., Intel processors starting with the Haswell microarchitecture released 2013, and processors from AMD starting with the Rizen)
Also noteworthy that on Intel at least, using AVX/AVX2 reduces the frequency of the CPU for a while. It can even go below base clock.
iirc, it's complicated. Some instructions don't reduce the frequency; some reduce it a little; some reduce it a lot.
I'm not sure AVX2 is as ubiquitous as the README says: "We assume AVX2 support which is available in all recent mainstream x86 processors produced by AMD and Intel."
I guess "mainstream" is somewhat subjective, but some recent Chromebooks have Celeron processors with no AVX2:
https://us-store.acer.com/chromebook-14-cb3-431-c5fm
https://ark.intel.com/products/91831/Intel-Celeron-Processor...
Because someone wanting 2.2GB/s JSON parsing is deploying to a chromebook...
It doesn't seem that laughable to me to want faster JSON parsing on a Chromebook, given how heavily JSON is used to communicate between webservers and client-side Javascript.
"Faster" meaning faster than Chromebooks do now; 2.2 GB/s may simply be unachievable hardware-wise with these cheap processors. They're kinda slow, so any speed increase would be welcome.
AVX2 also incurs some pretty large penalties for switching between SSE and AVX2. Depending on the amount of time taken in the library between calls, it could be problematic.
This looks mostly applicable to server scenarios where the runtime environment is highly controlled.
There is no real penalty for switching between SSE and AVX2, unless you do it wrong. What are you referring to specifically?
Are you talking about state transition penalties that can occur if you forget a vzeroupper? That's the only thing I'm aware of which kind of matches that.
A faster, more efficient cryptocurrency
Full disclosure: I work on the cryptocurrency in this article, Algorand.
There are a lot of questions and speculation here about this paper and Algorand. I would be happy to try an answer them to your satisfaction. Some context may be helpful first, though. This paper is an innovation about one aspect of our technology. Algorand has a very fast consensus mechanism and can add blocks as quickly as the network can deliver them. We become a victim of our success. The blockchain will grow very rapidly. A terabyte a month is possible. The storage issue associated with our performance can quickly become an issue. The Vault paper is focused on solving this and other storage scaling problems.
The Algorand pure proof-of-stake blockchain and associated cryptocurrency has many novel innovations aside from Vault. It possesses security and scalability properties beyond what any other blockchain technology allows while still being completely decentralized. Our website, algorand.com, and whitepaper are great places to start to learn more.
If you learn best from videos then I suggest you watch Turing award winner and cryptographic pioneer, Silvio Micali, talk about Algorand: https://youtu.be/NykZ-ZSKkxM. He is a captivating speaker and the founder of Algorand.
Are there reasons that e.g. Bitcoin and Ethereum and Stellar could not implement some of these more performant approaches that Algorand [1] and Vault [2] have developed, published, and implemented? Which would require a hard fork?
[2] https://dspace.mit.edu/handle/1721.1/117821
My understanding is that PoS approaches follow normal byzantine agreement theory which states that adversaries cannot control more than 1/3rd of the accounts (or money in the case of algorand). You can also delay new blocks more easily.
Ethereum is scared or that so they are implementing some hybrid form.
Bitcoin is doomed from my perspective, because of the focus on proof of work and the confirmation times. When you realize that algorand is super fast, there is no "confirmation time", and there is no waste in energy to mine, then it is hard to back up any cryptocurrency focusing on proof of work.
And what of decentralized premined chains (with no PoW, no PoS, and far less energy use) that release coins with escrow smart contracts over time such as Ripple and Stellar (and close a new ledger every few seconds)?
> Algorand has a very fast consensus mechanism and can add blocks as quickly as the network can deliver them. We become a victim of our success. The blockchain will grow very rapidly. A terabyte a month is possible. The storage issue associated with our performance can quickly become an issue. The Vault paper is focused on solving this and other storage scaling problems.
What prevents a person from using a chain like IPFS?
Ethereum Casper PoS has been under review for quite some time.
Why isn't all Bitcoin on Lightning Network?
Bitcoin could make bootstrapping faster by choosing a considered-good blockhash and balances, but AFAIU, re-verifying transactions like Bitcoin and derivatives do prevents hash collision attacks that are currently considered infeasible for SHA-256 (especially given a low block size).
There was an analysis somewhere where they calculated the cloud server instance costs of mounting a ~51% attack (which applies to PoW chains) for various blockchains.
Bitcoin is not profitable to mine in places without heavily subsidized dirty/clean energy anymore: energy and Bitcoin commodity costs and prices have intersected. They'll need any of: inexpensive clean energy, more efficient chips, higher speculative value.
Energy arbitrage (grid-scale energy storage) may be more profitable now. We need energy storage in order to reach 100% renewable energy (regardless of floundering policy support).
Ripple is not decentralized. I don't know enough about Stellar to answer.
Bitcoin is software and can easily implement these features but the community is divided and can't reach consensus on anything. Lightning Network as layer two solution is pretty good from what I know.
Ethereum improvements are coming along very slowly and that's good. They're the only blockchain with active engagement by thousands of multiple parties.
Aragaon and Vault's papers might sound good, but who knows how they'll turn out in production.
People argue this all day. There's a lot of FUD.
Ripple only runs ~7% of validator nodes; which is far less centralized control than major Bitcoin mining pools and businesses (who do the deciding in regards to the many Bitcoin hard forks); that's one form of decentralization.
Ripple clients can use their own UNL or use the Ripple-approved UNL.
Ripple is traded on a number of exchanges (though fewer than Bitcoin for certain); that's another form of decentralization.
As an open standard, ILP will further reduce vendor lock in (and increase interoperability between) networks that choose to implement it.
There are forks of Ripple (e.g. Stellar) just like there are forks of Bitcoin and Ethereum.
From https://ripple.com/insights/the-inherently-decentralized-nat... :
> In contrast, the XRP Ledger requires 80 percent of validators on the entire network, over a two-week period, to continuously support a change before it is applied. Of the approximately 150 validators today, Ripple runs only 10. Unlike Bitcoin and Ethereum — where one miner could have 51 percent of the hashing power — each Ripple validator only has one vote in support of an exchange or ordering a transaction.
How does your definition of 'decentralized' differ?
[deleted]
Git-signatures – Multiple PGP signatures for your commits
Is there anything out there that doesn't need GPG? Having a working GPG install is a huge lift for developers.
I take this to mean: apart from the barnacles on GPG, could there be a system which does what GPG does for software development (signing), without the non-functioning web-of-trust of GPG, or the hierarchical system of x509 signing? Something that deals with lost keys, compromised keys/accounts, loss of DNS control, MitMing, MitBing, etc?
I think it is probably in the class of problems where there are no great foolproof solutions. However, I can imagine that techniques like certificate transparency (all signed x509 certificates pushed to a shared log) would be quite useful. Even blockchain techniques. Maybe send someone to check on me, I'm feeling unwell having written that.
> I think it is probably in the class of problems where there are no great foolproof solutions. However, I can imagine that techniques like certificate transparency (all signed x509 certificates pushed to a shared log) would be quite useful.
Securing DNS: "https://news.ycombinator.com/item?id=19181362"
> Certs on the Blockchain: "Can we merge Certificate Transparency with blockchain?" https://news.ycombinator.com/item?id=18961724
> Namecoin (decentralized blockchain DNS): https://en.wikipedia.org/wiki/Namecoin
(Your first link is broken.)
My main problem with blockchain is the excessive energy consumption of PoW. I know there are PoS efforts, but they seem problematical.
I like the recent CertLedger paper: https://eprint.iacr.org/2018/1071.pdf
My mistake. How ironic. Everything depends upon the red wheelbarrow. Here's that link without the trailing ": https://news.ycombinator.com/item?id=19181362
> My main problem with blockchain is the excessive energy consumption of PoW. I know there are PoS efforts, but they seem problematical.
One report said that 78% of Bitcoin energy usage is from renewable sources (many of which would otherwise be curtailed and otherwise unfunded due to flat-to-falling demand for electricity). But PoW really is expensive and hopefully the market will choose less energy-inefficient solutions from the existing and future blockchain solutions while keeping equal or better security assurances.
>> Proof of Work (Bitcoin, ...), Proof of Stake (Ethereum Casper), Proof of Space, Proof of Research (GridCoin, CureCoin,)
The spec should be: DDOS resiliant (without a SPOF), no one entity with control over API and/or database credentials and database backups and the clock, and immutable.
Immutability really cannot be ensured with hashed records that incorporate the previous record's hash as a salt in a blocking centralized database because someone ultimately has root and the clock and all the backups and code vulnerable to e.g. [No]SQL injection; though distributed 'replication' and detection of record modification could be implemented. git push -f may be detected if it's on an already-replicated branch; but git depends upon local timestamps. google/trillian does Merkle trees in a centralized database (for Certificate Transparency).
In quickly reading the git-signatures shell script sources, I wasn't certain whether the git-notes branch with the .gitsigners that are fetched from all n keyservers (with DNS) is also signed?
I also like the "Table 1: Security comparison of Log Based Approaches to Certificate Management" in the CertLedger paper. Others are far more qualified to compare implementations.
You read my mind. I'd love if it could be rooted in a Yubikey.
Decoupling the "signing" and "verifying" parts seem like a good idea. As random Person signs something, how someone else figures out how to go trust that signature is a separate problem.
> I'd love if it could be rooted in a Yubikey.
FIDO2 and Yubico helped develop the new W3C WebAuthn standard: https://en.wikipedia.org/wiki/WebAuthn
But WebAuthn does not solve for WoT or PKI or certificate pinning.
> Decoupling the "signing" and "verifying" parts seem like a good idea. As random Person signs something, how someone else figures out how to go trust that signature is a separate problem.
Someone can probably help with terminology here. There's identification (proving that a person has the key AND that it's their key (biometrics, challenge-response)), signing (using a key to create a cryptographic signature – for the actual data or a reasonably secure cryptographic hash of said data – that could only could have been created with the given key), signature verification (checking that the signature was created by the claimed key for the given data), and then there's trusting that the given key is authorized for a specific purpose (Web of Trust (key-signing parties), PKI, ACME, exchange of symmetric keys over a different channel such as QKD) by e.g. signing a structured document that links cryptographic keys with keys for specific authorized functions and trusting the key(s) used to sign said authorizing document.
Private (e.g. Zero Knowledge) blockchains can be used for key exchange and key rotation. Public blockchains can be used for sharing (high-entropy) key components; also with an optional exchange of money to increase the cost of key compromise attempts.
There's also WKD: "Web Key Directory"; which hosts GPG keys over HTTPS from a .well-known URL for a given user@domain identifier: https://wiki.gnupg.org/WKD
Compared to existing PGP/GPG keyservers, WKD does rely upon HTTPS.
TUF is based on Thandy. TUF: "The Update Framework" does not presume channel security (is designed to withstand channel compromise) https://en.wikipedia.org/wiki/The_Update_Framework_(TUF)
The TUF spec doesn't mention PGP/GPG: https://github.com/theupdateframework/specification/blob/mas...
There's a derivative of TUF for automotive applications called Uptane: https://uptane.github.io
The Bitcoin article on multisignature; 1-of-2, 2-of-2, 2-of-3, 3-of-5, etc.: https://en.bitcoin.it/wiki/Multisignature
Running an LED in reverse could cool future computers
"Near-field photonic cooling through control of the chemical potential of photons" (2019) https://www.nature.com/articles/s41586-019-0918-8
Compounding Knowledge
Buffet’s approach to life is interesting for the same reason an Olympic gymnast is interesting. He has specialized to an extreme and is taking advantage of the rewards of that specialization and natural talent in a unique way.
It’s easy for me to feel shame that I don’t read 8 hours per day, as Warren and Charlie do. Buffett is a phenomenal investor but by all accounts, rather odd. He eats like crap, doesn’t exercise, had a profoundly weird relationship with his wife, and seems addicted to his work at the expense of everything else.
My point is this: his practice of reading income statements and business reports 8 hours per day for 60 years doesn’t make him the kind of person I wish to emulate.
Don’t get me wrong, I respect his levelheadedness toward money, lack of polish wrt PR, and generous philanthropic efforts. There’s a lot of good. But it’s easy to idolize the guy.
Controversial take: Buffett is actually not a great investor in the way people think. Via the float in his insurance companies, he receives a 0% infinite maturity loan to plow into the market. That financial leverage gives him the ability to beat the market year after year - not his own stockpicking prowess.
If you were to start with $1B, then get an extra $2B that you never had to pay back, you too would do quite well holding Coca-Cola and other safe companies with moats. Your returns would be 3x everyone else's.
Buffett himself has tried to explain the impact of float on Berkshire Hathaway but it never seems to sink in with people.
Ill take it even farther: Buffet isn't statistically different than average. He just found a strategy that happened to work, was stubborn enough to stick to it through bad times, and used copious amounts of leverage to juice returns.
A really interesting paper called "Buffet's Alpha" talks about this, and was able to replicate his performance by following a few simple rules. They found that he produced very little actual alpha. To his credit, he seems to have been observant enough to stumble into factors(value, quality, and low beta) before anyone else knew they existed, which is his real strength and contribution.
A summary of that paper from the authors is here: https://www.aqr.com/Insights/Research/Journal-Article/Buffet....
They work at AQR, a firm which is notable for being one of the only successful hedge funds to actually publish meaningful research.
The paper's conclusion is noteworthy:
> The efficient market counterargument is that Buffett was simply lucky. Our findings suggest that Buffett’s success is neither luck nor magic but is a reward for a successful implementation of value and quality exposures that have historically produced high returns. Second, we illustrated how Buffett’s record can be viewed as an expression of the practical implementability of academic factor returns after transaction costs and financing costs. We simulated how investors can try to take advan- tage of similar investment principles. Buffett’s success shows that the high returns of these academic factors are not simply “paper” returns; these returns can be realized in the real world after transaction costs and funding costs, at least by Warren Buffett. Furthermore, Buffett’s exposure to the BAB factor and his unique access to leverage are consistent with the idea that the BAB factor represents reward to the use of leverage.
BTW, AQR funded the initial development of pandas; which now powers tools like alphalens (predictive factor analysis) and pyfolio.
There's your 'compounding knowledge'.
(Days later)
"7 Best Community-Built Value Investing Algorithms Using Fundamentals" https://blog.quantopian.com/fundamentals-contest-winners/
(The Zipline backtesting library also builds upon Pandas)
How can we factor ESG/sustainability reporting into these fundamentals-driven algorithms in order to save the world?
I wonder to what extent Warren Buffet is Warren Buffet because of how he thinks and acts (like all of these non-fiction authors selling books by using his name would have us believe), and to what extent he is the product of media selection bias. -- If you take a large enough group of people who take risky stakes that are large enough (like the world of financial asset management), then one of them is bound to be as successful as Warren Buffet, even if they all behave randomly.
Funnily enough, Buffett actually calculated this in his essay "The Super Investors of Graham and Doddsville"
Basically, advocates of the Free Market Hypotheses believed that it was impossible for anyone to deliberately, repeatedly generate alpha. The market was rational, and success was probabilistically distributed. With a large enough population, you will get Buffett level returns - therefor Buffett is a fluke.
However, Buffett calculated that there weren't enough investors for his success to be a factor of random distribution, plus there were 2 dozen others who followed a similar strategy who also consistently generate alpha
"The Superinvestors of Graham and Doddsville" (1984) https://scholar.google.com/scholar?cluster=17265410477248371...
From https://en.wikipedia.org/wiki/The_Superinvestors_of_Graham-a... :
> The speech and article challenged the idea that equity markets are efficient through a study of nine successful investment funds generating long-term returns above the market index.
This book probably doesn't mention that he's given away over 71% to charity since Y2K. Or that it's really cold and windy and snowy in Omaha; which makes for lots of reading time.
"Warren Buffett and the Interpretation of Financial Statements: The Search for the Company with a Durable Competitive Advantage" (2008) [1], "Buffetology" (1999) [2], and "The Intelligent Investor" (1949, 2009) [3] are more investment-strategy-focused texts.
[1] https://smile.amazon.com/Warren-Buffett-Interpretation-Finan...
[2] https://smile.amazon.com/Buffettology-Previously-Unexplained...
[3] https://smile.amazon.com/Intelligent-Investor-Definitive-Inv...
Value Investing: https://en.wikipedia.org/wiki/Value_investing https://www.investopedia.com/terms/v/valueinvesting.asp
Why CISA Issued Our First Emergency Directive
There are a number of efforts to secure DNS (and SSL/TLS which generally depends upon DNS; and upon which DNS-over-HTTPS depends) and the identity proof systems which are used for record-change authentication and authorization.
Domain registrars can and SHOULD implement multi-factor authentication. https://en.wikipedia.org/wiki/Multi-factor_authentication
Are there domain registrars that support FIDO/U2F or the new W3C WebAuthn spec? https://en.wikipedia.org/wiki/WebAuthn
Credentials and blockchains (and biometrics): https://gist.github.com/westurner/4345987bb29fca700f52163c33...
DNSSEC: https://en.wikipedia.org/wiki/Domain_Name_System_Security_Ex...
ACME / LetsEncrypt certs expire after 3 months (*) and require various proofs of domain ownership: https://en.wikipedia.org/wiki/Automated_Certificate_Manageme...
Certificate Transparency: https://en.wikipedia.org/wiki/Certificate_Transparency
Certs on the Blockchain: "Can we merge Certificate Transparency with blockchain?" https://news.ycombinator.com/item?id=18961724
Namecoin (decentralized blockchain DNS): https://en.wikipedia.org/wiki/Namecoin
DNSCrypt: https://en.wikipedia.org/wiki/DNSCrypt
DNS over HTTPS: https://en.wikipedia.org/wiki/DNS_over_HTTPS
DNS over TLS: https://en.wikipedia.org/wiki/DNS_over_TLS
Chrome will Soon Let You Share Links to a Specific Word or Sentence on a Page
I would like to urge the browser developers/makers to adopt existing proposals which came through open consensus which do precisely cover the same use cases (and more!)
W3C Reference Note on Selectors and States: https://www.w3.org/TR/selectors-states/
It is part of the suite of specs that came through the W3C Web Annotation Working Group: https://www.w3.org/annotation/
More examples in W3C Note Embedding Web Annotations in HTML: https://www.w3.org/TR/annotation-html/
Different kinds of Web resources can combine multiple selectors and states. Here is a simple one using `TextQuoteSelector` handled by the https://dokie.li/ clientside application:
http://csarven.ca/dokieli-rww#selector(type=TextQuoteSelecto...
A screenshot/how-to: https://twitter.com/csarven/status/981924087843950595
"Integration with W3C Web Annotations" https://github.com/bokand/ScrollToTextFragment/issues/4
> It would be great to be able to comment on the linked resource text fragment. W3C Web Annotations [implementations] don't recognize the targetText parameter, so AFAIU comments are then added to the document#fragment and not the specified text fragment. [...]
> Is there a simplified mapping of W3C Web Annotations to URI fragment parameters?
Guidelines for keeping a laboratory notebook
My paper notebooks were always horrific. Writing has always been painful and awkward for me. But I could type like nobody's business thanks to programming. I survived college in the early 80s by being one of the first students to get a word processor.
By the time I was keeping a notebook, my work was generating mountains of computer readable data, source code, and so forth. We managed by agreeing on a format for data files, where the filename referenced a notebook page, and it worked OK.
Today, it's unavoidable that people are going to keep their notes electronically, and there are no perfect solutions for doing this. Wet chemists still like paper notebooks, since it's hard to get a computer close to the bench, and to type while wearing rubber gloves. Academic workers are expected to supply their own computers, and are nervous about getting them damaged or contaminated. Plus, drawing pictures and writing equations on a computer are both awkward.
Computation related fields lend themselves well to purely electronic notebooks, no surprise. Today, a lot of my work fits perfectly in a Jupyter notebook.
Commercial notebook software exists, but it tends to be sold largely for enterprise use, i.e., the solution it solves is how to control lab workers and secure their results, not how to enable independent, creative work.
> Computation related fields lend themselves well to purely electronic notebooks, no surprise. Today, a lot of my work fits perfectly in a Jupyter notebook.
Some notes and ideas regarding Jupyter notebooks as lab notebooks from "Keeping a Lab Notebook [pdf]": https://news.ycombinator.com/item?id=15710815
Superalgos and the Trading Singularity
Though others didn't, you might find this interesting: "Ask HN: Why would anyone share trading algorithms and compare by performance?" https://news.ycombinator.com/item?id=15802785 ( https://westurner.github.io/hnlog/#story-15802785 )
I think there is value in a back-testing module, however, sharing an algo doesn't make sense to me, until unless someone wants to buy mine for an absurd amount.
I think part of the value of sharing knowledge and algorithmic implementations comes from getting feedback from other experts; like peer review and open science and teaching.
Case in point: the first algorithm on this list [1] of community contributed algorithms that were migrated to their new platform is "minimum variance w/ constraint" [2]. Said algorithm showed returns of over 200% as compared with 77% returns from the SPY S&P 500 ETF over the same period, ceteris paribus. In the 69 replies, there are modifications by community members and the original author that exceed 300%.
Working together on open algorithms has positive returns that may exceed advantages of closed algorithmic development without peer review.
[1] https://www.quantopian.com/posts/community-algorithms-migrat...
[2] https://www.quantopian.com/posts/56b6021b3f3b36b519000924
How well does it do in production though and what happens when multiple algos execute the same trades? Does it cause the rest of the algos to adapt and change results? It makes sense to back-test together and work on it, but if it's proven to work, someone will create something to monitor volume on those trades and work against it. I'd be curious to see the same algo do 300% in production, and if so, then my bias would be uncalled for.
> How well does it do in production though and what happens when multiple algos execute the same trades?
Price inflation.
> Does it cause the rest of the algos to adapt and change results?
Trading index ETFs? IDK
> It makes sense to back-test together and work on it, but if it's proven to work, someone will create something to monitor volume on those trades and work against it.
Why does it need to do lots of trades? Is it possible for anyone other than e.g. SEC to review trades by buyer or seller?
> I'd be curious to see the same algo do 300% in production, and if so, then my bias would be uncalled for.
pyfolio does tear sheets with Zipline algos: pyfolio/examples/zipline_algo_example.ipynb https://nbviewer.jupyter.org/github/quantopian/pyfolio/blob/...
alphalens does performance analysis of predictive factors: alphalens/examples/pyfolio_integration.ipynb https://nbviewer.jupyter.org/github/quantopian/alphalens/blo...
awesome-quant lists a bunch of other tools for algos and superalgos: https://github.com/wilsonfreitas/awesome-quant
What's a good platform for paper trading (with e.g. zipline or moonshot algorithms)?
I disagree with price inflation just because everything is hedged, but it may be true.
The too many trades is if there are 300 algos, and I look in the order book and see different orders from different exchanges at the same price point, then I would be adapting to see what's happening, not myself, but there are people who watch order flows.
I don't paper trade, either it works in production with real money or not. Have to get a feel for spreads, commissions, and so on.
Also, in my case, I am hesitant to even use paid services as someone can be watching it, so most my tools are made by me. Good luck with your trading though, if it works out, let me know, I'd pay to use it along side my other trades.
Crunching 200 years of stock, bond, currency and commodity data
Coming from a CS/Econ/Finance background....
Efficiency as “the market fully processes all information” is decidedly untrue.
Efficiency as “Its very hard to arb markets without risk better than a passive strategy” is very true.
Most professional money managers don’t beat the market once you factor in fees. Many private equity funds produce outsize returns, but it’s based on higher risk and taking advantage of tax laws.
Over time, positive performance for mutual funds don’t persist. (If you’re in the top decile performance one year, you’re no more likely to be there next year)
Despite all of this, there are ways people make money. But it’s a small subset of professionals. It’s high frequency traders who find ways to cut the line on mounds of pennies. It inside information from hedge funds. Or earlier access to non public information. But it’s generally not in areas that normal people like you and me can access.
It inside information from hedge funds. Or earlier access to non public information. But it’s generally not in areas that normal people like you and me can access.
Asymmetric information is pretty far from what used to be said about the perfect market and rational actors. It's "there's a sucker born every minute" and "if it seems too good to be true it probably is" economics.
I might be misunderstanding what you're saying here, but are you sure you're right? Fama originally predicated the model of the efficient market (the efficient market hypothesis) on the idea of informational efficiency. Information asymmetry is a fundamental measure involved in the idealized model of an efficient market.
What you're mentioning about rational actors is actually a different topic altogether in economics.
Or have I misunderstood what you're getting at?
I was interested, so I did some research here.
Rational Choice Theory https://en.wikipedia.org/wiki/Rational_choice_theory
Rational Behavior https://www.investopedia.com/terms/r/rational-behavior.asp
> Most mainstream academic economics theories are based on rational choice theory.
> While most conventional economic theories assume rational behavior on the part of consumers and investors, behavioral finance is a field of study that substitutes the idea of “normal” people for perfectly rational ones. It allows for issues of psychology and emotion to enter the equation, understanding that these factors alter the actions of investors, and can lead to decisions that may not appear to be entirely rational or logical in nature. This can include making decisions based primarily on emotion, such as investing in a company for which the investor has positive feelings, even if financial models suggest the investment is not wise.
Behavioral finance https://www.investopedia.com/terms/b/behavioralfinance.asp
Bounded rationality > Relationship to behavioral economics https://en.wikipedia.org/wiki/Bounded_rationality
Perfectly rational decisions can be and are made without perfect information; bounded by the information available at the time. If we all had perfect information, there would be no entropy and no advantage; just lag and delay between credible reports and order entry.
Information asymmetry https://en.wikipedia.org/wiki/Information_asymmetry
Heed these words wisely: What foolish games! Always breaking my heart.
https://deepmind.com/blog/game-theory-insights-asymmetric-mu...
> Asymmetric games also naturally model certain real-world scenarios such as automated auctions where buyers and sellers operate with different motivations. Our results give us new insights into these situations and reveal a surprisingly simple way to analyse them. While our interest is in how this theory applies to the interaction of multiple AI systems, we believe the results could also be of use in economics, evolutionary biology and empirical game theory among others.
https://en.wikipedia.org/wiki/Pareto_efficiency
> A Pareto improvement is a change to a different allocation that makes at least one individual or preference criterion better off without making any other individual or preference criterion worse off, given a certain initial allocation of goods among a set of individuals. An allocation is defined as "Pareto efficient" or "Pareto optimal" when no further Pareto improvements can be made, in which case we are assumed to have reached Pareto optimality.
Which, I think, brings me to equitable availability of maximum superalgo efficiency and limits of real value creation in capital and commodities markets; which'll have to be a topic for a different day.
Show HN: React-Schemaorg: Strongly-Typed Schema.org JSON-LD for React
I have a slightly longer post describing this work and the reasoning behind it on dev.to[1].
[1]: https://dev.to/eyassh/react-schemaorg-strongly-typed-schemaorg-json-ld-for-react-4lhdhttps://dev.to/eyassh/react-schemaorg-strongly-typed-schemao...
Is there a good way to generate JSONschema and thus forms from schema.org RDFS classes and (nested, repeatable) properties?
By JSONschema do you mean [this standard](https://json-schema.org/)? I don't know of a tool that does that yet, the JSON Schema is general enough with allOf/anyOf that it should express schema.org schema as well.
Depends on the purpose here. With this, my goal was to speed up the Write-Update-Debug development loop. Depends on the use case, simply using Google's Structured Data Testing Tool [1] might be a better way to verify schema than JSON-schema?
[1]: https://search.google.com/structured-data/testing-toolThere are a number of tools for generating forms and requisite client and serverside data validations from JSONschema; but I'm not aware of any for RDFS (and thus the schema.org schema [1]). A different use case, for certain.
https://schema.org/docs/developers.html#defs
Definitely a cool area for exploration. I'm not aware of JSON Schema generators from RDFS either.
It should be possible to model the basics (nested structure, repeated structure, defining the "domain" and "range" of a value).
Schema.org definitions however have no conception of "required" values[], however, so some of the cool form validation we see in some of these tools might not apply here.
[*] _Consumers_ of Schema.org JSON-LD or microdata, however, might define their own data requirements. E.g. Google has some concept of required fields, which you can see when using the Structured Data Testing Tool.Consumer Protection Bureau Aims to Roll Back Rules for Payday Lending
From the article:
> The way payday loans work is that payday lenders typically offer small loans to borrowers who promise to pay the loans back by their next paycheck. Interest on the loans can have an annual percentage rate of 390 percent or more, according to a 2013 report by the CFPB. Another bureau report from the following year found that most payday loans — as many as 80 percent — are rolled over into another loan within two weeks. Borrowers often take out eight or more loans a year.
390%
From https://www.npr.org/2019/02/06/691944789/consumer-protection... :
> TARP recovered funds totalling $441.7 billion from $426.4 billion invested, earning a $15.3 billion profit or an annualized rate of return of 0.6% and perhaps a loss when adjusted for inflation.[2][3]
0.6%
Is your point that the US government should get into the payday lending business?
Lectures in Quantitative Economics as Python and Julia Notebooks
It's amazing how we are watching use cases for notebooks and spreadsheets converging. I wonder what the killer feature will be to bring a bigger chunk of the Excel world into a programmatic mindset... Or alternatively, whether we will see notebook UIs embedded in Excel in the future in place of e.g. VBA.
That’s not a bad idea. Spreadsheets are pure functional languages built that use literal spaces instead of namespaces.
Notebooks are cells of logic. You could conceivably change the idea of notebook cells to be an instance of a function that points to raw data and returns raw data.
Perhaps this just Alteryx though
This is brilliant.
I'm picturing the ability to write a Python function with the parameters being just like the parameters in an Excel function. You can drag the cell and have it duplicated throughout a row, updating the parameters to correspond to the rows next to it.
It would exponentially expand the power of excel. I wouldn't be limited to horribly unmaintainable little Excel functions.
VBA can't be used to do that, can it? As far as I understand (and I haven't investigated VBA too much) VBA works on entire spreadsheets.
Essentially, replace the excel formula `=B3-B4` with a Python function `subtract(b3, b4)` where Subtract is defined somewhere more conveniently (in a worksheet wide function definition list?).
This would require a reactive recomputing of cells to be anything like a spreadsheet. > Essentially, replace the excel formula `=B3-B4` with a Python function `subtract(b3, b4)`
as of now jupyter/ipython would not recompute `subtract(b3, b4)` if you change b3 or b4, this has positive and negative (reliance on hidden state and order of execution) effects.
I too would really like something like this, but I think it is pretty far away from where jupiter is now.
You can build something like this with Jupyter today.
> Traitlets is a framework that lets Python classes have attributes with type checking, dynamically calculated default values, and ‘on change’ callbacks. https://traitlets.readthedocs.io/en/stable/
> Traitlet events. Widget properties are IPython traitlets and traitlets are eventful. To handle changes, the observe method of the widget can be used to register a callback https://ipywidgets.readthedocs.io/en/stable/examples/Widget%...
You can definitely build interactive notebooks with Jupyter Notebook and JupyterLab (and ipywidgets or Altair or HoloViews and Bokeh or Plotly for interactive data visualization).
> Qgrid is a Jupyter notebook widget which uses SlickGrid to render pandas DataFrames within a Jupyter notebook. This allows you to explore your DataFrames with intuitive scrolling, sorting, and filtering controls, as well as edit your DataFrames by double clicking cells. https://github.com/quantopian/qgrid
Qgrid's API includes event handler registration: https://qgrid.readthedocs.io/en/latest/
> neuron is a robust application that seamlessly combines the power of Visual Studio Code with the interactivity of Jupyter Notebook. https://marketplace.visualstudio.com/items?itemName=neuron.n...
"Excel team considering Python as scripting language: asking for feedback" (2017) https://news.ycombinator.com/item?id=15927132
OpenOffice Calc ships with Python 2.7 support: https://wiki.openoffice.org/wiki/Python
Procedural scripts written in a general purpose language with named variables (with no UI input except for chart design and persisted parameter changes) are reproducible.
What's a good way to review all of the formulas and VBA and/or Python and data ETL in a spreadsheet?
Is there a way to record a reproducible data transformation script from a sequence of GUI interactions in e.g. OpenRefine or similar?
OpenRefine/OpenRefine/wiki/Jupyter
"Within the Python context, a Python OpenRefine client allows a user to script interactions within a Jupyter notebook against an OpenRefine application instance, essentially as a headless service (although workflows are possible where both notebook-scripted and live interactions take place. https://github.com/OpenRefine/OpenRefine/wiki/Jupyter
Are there data wrangling workflows that are supported by OpenRefine but not Pandas, Dask, or Vaex?
This interesting need to have a closer look, possibly refine can be more efficient? But haven't used it enough to know, just payed around with it a bit. Didn't realise you could combine it with jupyter.
There are undergraduate and graduate courses in each language:
Python version: https://lectures.quantecon.org/py/
Julia version: https://lectures.quantecon.org/jl/
Does anyone else find it strange that there is no real-world data in these notebooks? It's all simulations or abstract problems.
This gives me the sense, personally, that economists aren't interested in making accurate predictions about the world. Other fields would, I think, test their theories against observations.
pandas-datareader can pull data from e.g. FRED, Eurostat, Quandl, World Bank: https://pandas-datareader.readthedocs.io/en/latest/remote_da...
pandaSDMX can pull SDMX data from e.g. ECB, Eurostat, ILO, IMF, OECD, UNSD, UNESCO, World Bank; with requests-cache for caching data requests: https://pandasdmx.readthedocs.io/en/latest/#supported-data-p...
The scikit-learn estimator interface includes a .score() method. "3.3. Model evaluation: quantifying the quality of predictions" https://scikit-learn.org/stable/modules/model_evaluation.htm...
statsmodels also has various functions for statistically testing models: https://www.statsmodels.org/stable/
"latex2sympy parses LaTeX math expressions and converts it into the equivalent SymPy form" and is now merged into SymPy master and callable with sympy.parsing.latex.parse_latex(). It requires antlr-python-runtime to be installed. https://github.com/augustt198/latex2sympy https://github.com/sympy/sympy/pull/13706
IDK what Julia has for economic data retrieval and model scoring / cost functions?
If Software Is Funded from a Public Source, Its Code Should Be Open Source
From the US Digital Services Playbook [1]:
> PLAY 13
> Default to open
> When we collaborate in the open and publish our data publicly, we can improve Government together. By building services more openly and publishing open data, we simplify the public’s access to government services and information, allow the public to contribute easily, and enable reuse by entrepreneurs, nonprofits, other agencies, and the public.
> Checklist
> - Offer users a mechanism to report bugs and issues, and be responsive to these reports
> [...]
> - Ensure that we maintain contractual rights to all custom software developed by third parties in a manner that is publishable and reusable at no cost
> [...]
> - When appropriate, publish source code of projects or components online
> [...]
> Key Questions
> [...]
> - If the codebase has not been released under an open source license, explain why.
> - What components are made available to the public as open source?
> [...]
Apache Arrow 0.12.0
> Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. It also provides computational libraries and zero-copy streaming messaging and interprocess communication. Languages currently supported include C, C++, C#, Go, Java, JavaScript, MATLAB, Python, R, Ruby, and Rust.
Statement on Status of the Consolidated Audit Trail (2018)
U.S. Federal District Court Declared Bitcoin as Legal Money
"Application of FinCEN's Regulations to Persons Administering, Exchanging, or Using Virtual Currencies" (2013) https://www.fincen.gov/resources/statutes-regulations/guidan...
"Legality of bitcoin by country or territory" https://en.wikipedia.org/wiki/Legality_of_bitcoin_by_country...
"Know your customer" https://en.wikipedia.org/wiki/Know_your_customer
"Anti-money-laundering measures by region" https://en.wikipedia.org/wiki/Money_laundering#Anti-money-la...
"Anti-money-laundering measures by region > United States" https://en.wikipedia.org/wiki/Money_laundering#United_States
Post Quantum Crypto Standardization Process – Second Round Candidates Announced
> As the latest step in its program to develop effective defenses, the National Institute of Standards and Technology (NIST) has winnowed the group of potential encryption tools—known as cryptographic algorithms—down to a bracket of 26. These algorithms are the ones NIST mathematicians and computer scientists consider to be the strongest candidates submitted to its Post-Quantum Cryptography Standardization project, whose goal is to create a set of standards for protecting electronic information from attack by the computers of both tomorrow and today.
> “These 26 algorithms are the ones we are considering for potential standardization, and for the next 12 months we are requesting that the cryptography community focus on analyzing their performance,”
Links to the 17 public-key encryption and key-establishment algorithms and 9 digital signature algorithms are here: "Round 2 Submissions" https://csrc.nist.gov/Projects/Post-Quantum-Cryptography/Rou...
"Quantum Algorithm Zoo" has moved to https://quantumalgorithmzoo.org .
Ask HN: How do you evaluate security of OSS before importing?
What tools can I use to evaluate the security posture of an OSS project before I approve its usage with high confidence?
Oddly, whether a project has at least one CVE reported could be interpreted in favor of the project. https://www.cvedetails.com
Do they have a security disclosure policy? A dedicated security mailing list?
Do they pay bounties or participate in e.g Pwn2own?
Do they cryptographically sign releases?
Do they cryptographically sign VCS tags (~releases)? commits? `git tag -s` / `git commit/merge -S` https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work
Downstream packagers do sometimes/often apply additional patches and then sign their release with the repo (and thus system global) GPG key.
Whether they require "Signed-off-by" may indicate that the project has mature controls and possibly a formal code review process requirement. (Look for "Signed-off-by:" in the release branch (`git commit/merge -s/--signoff`)
How have they integrated security review into their [iterative] release workflow?
Is the software formally verified? Are parts of the software implementation or spec formally verified?
Does the system trust the channel? The host? Is it a 'trustless' system?
What are the single points of failure?
How is logging configured? To syslog?
Do they run the app as root in a Docker container? Does it require privileged containers?
If it has to run as root, does it drop privileges at startup?
Does the package have an SELinux or AppArmor policy? (Or does it say e.g. "just set SELinux to permissive mode)
Is there someone you can pay to support the software in an enterprise environment? Open or closed, such contacts basically never accept liability; but if there is an SLA, do you get a pro-rated bill?
As far as indicators of actual software quality:
How much test coverage is there? Line coverage or statement coverage?
Do they run static analysis tools for all pull requests and releases? Dynamic analysis? Fuzzing?
Of course, closed or open source projects may do none or all of these and still be totally secure, insecure, or unsecure.
This is a pretty extensive list. Thanks for sharing!
Ask HN: How can I use my programming skills to support nonprofit organizations?
Lately I've been thinking about doing programming for nonprofits, both because I want to help out with what I'm good at but also to hone my skills and potentially get some open source credit.
So far I've had a hard time finding nonprofit projects where I can just pick up something and start programming. I know about freecodecamp.org, but they force you to go through their courses, and as I already have multiple years of experience as a developer, I feel like that would be a waste of time.
Isn't there a way to contribute to nonprofit organization in a more direct and simple manner like how you would contribute to an open source project on GitHub?
There are lots of project management systems with issue tracking and kanban boards with swimlanes. Because it's unreasonable to expect all volunteers to have a GH account or even understand what GH is for, support for external identity management and SSO may be essential to getting people to actually log in and change their password regularly.
Sidling a nonprofit with custom built software with no other maintainers is not what they need. Build (and pay for development, maintenance, timely security upgrades and security review) or Buy (where is our data? who backs it up? how much does it cost for a month or a few years? Is it open source with a hosted option; so that we can pay a developer to add or fix what we need?)
"Solutions architect" may be a more helpful objective title for what's needed. https://en.wikipedia.org/wiki/Solution_architecture
What are their needs? Marketing, accounting, operations, HR
Marketing: web site, maps service, directions, active social media presence that speaks to their defined audience
Accounting: Revenue and expenses, payroll/benefits/HR, projections, "How can we afford to do more?", handle donations and send receipts for tax purposes, reports to e.g. https://charitynavigator.org/ and infographics for wealth-savvy donors
Operations: Asset inventory, project management, volunteer scheduling
HR: payroll, benefits, volunteer scheduling, training, turnover, retaining selfless and enlightenedly-self-interested volunteers
Create a spreadsheet. Rows: needs/features/business processes. Columns: essential, nice to have, software products and services.
Create another spreadsheet. Rows: APIs. Columns: APIs.
Training: what are the [information systems] processes/workflows/checklists? How can I suggest a change? How do we reach consensus that there's a better way to do this? Is there a wiki? Is there a Q&A system?
"How much did you sink on that? Probably seemed like the best option according to the information available at the time, huh? Do you have a formal systems acquisition process? Who votes according to what type of prepared analysis? How much would it cost to switch? What do we need to do to ETL (extract, transform, and load) into a newer better system?"
When estimating TCO for a nonprofit, turnover is a very real consideration. People move. Chances are, as with most organizations TBH, there's a patchwork of partially-integrated and maybe-integrable systems that it may or may not be more cost-effective and maintainable to replace with a cloud ERP specifically designed for nonprofits.
Who has access rights to manually update which parts of the website? How can we include dynamic ([other] database-backed) content in our website? What is a CMS? What is an ERP? What is a CRM? Are these customers, constituents, or both? When did we last speak with those guys? How can people share our asks with social media networks?
If you're not willing or able to make a long-term commitment, the more responsible thing to do is probably to disclose any conflicts of interest recommend a SaaS solution hosted in a compliant data center.
q="nonprofit erp"
q="nonprofit crm"
q="nonprofit cms" + donation campaign visibility
What time of day are social media posts most likely to get maximum engagement from which segments of our audience? What is our ~ARPU "average revenue per user/follower"?
... As a volunteer and not a FTE, it may be a worthwhile exercise to build a prototype of the new functionality with whatever tools you happen to be familiar with with the expectation that they'll figure out a way to accomplish the same objectives with their existing systems. If that's not possible, there may be a business opportunity: are there other organizations with the same need? Is there a sustainable market for such a solution? You may be building to be acquired.
Ask HN: Steps to forming a company?
Hey guys, I'm leaving my firm very shortly to form a startup.
Does why have a checklist of proper ways to do things?
Ie. 1. Form Chapter C Delaware company with Clerky 2. Hire payroll company x 3. use this company for patents.
any info there?
From "Ask HN: What are your favorite entrepreneurship resources" https://news.ycombinator.com/item?id=15021659 :
> USA Small Business Administration: "10 steps to start your business." https://www.sba.gov/starting-business/how-start-business/10-...
> "Startup Incorporation Checklist: How to bootstrap a Delaware C-corp (or S-corp) with employee(s) in California" https://github.com/leonar15/startup-checklist
> FounderKit has reviews for Products, Services, and Software for founders: https://founderkit.com
... I've heard good things about Gusto for payroll, HR, and benefits through Guideline: https://gusto.com/product/pricing
A Self-Learning, Modern Computer Science Curriculum
Outstanding resource.
jwasham/coding-interview-university also links to a number of also helpful OER resources: https://github.com/jwasham/coding-interview-university
MVP Spec
> The criticism of the MVP approach has led to several new approaches, e.g. the Minimum Viable Experiment MVE[19] or the Minimum Awesome Product MAP[20].
https://en.wikipedia.org/wiki/Minimum_viable_product#Critici...
Can we merge Certificate Transparency with blockchain?
From "REMME – A blockchain-based protocol for issuing X.509 client certificates" https://news.ycombinator.com/item?id=18868540 :
""" In no particular order, there are a number of blockchain PKI (and DNS (!)) proposals and proofs of concept.
"CertLedger: A New PKI Model with Certificate Transparency Based on Blockchain" (2018) https://arxiv.org/pdf/1806.03914 https://scholar.google.com/scholar?q=related:LF9PMeqNOLsJ:sc...
"TABLE 1: Security comparison of Log Based Approaches to Certificate Management" (p.12) lists a number of criteria for blockchain-based PKI implementations:
- Resilient to split-world/MITM attack
- Provides revocation transparency
- Eliminates client certificate validation process
- Eliminates trusted key management
- Preserves client privacy
- Require external auditing
- Monitoring promptness
... These papers also clarify why a highly-replicated decentralized trustless datastore — such as a blockchain — is advantageous for PKI. WoT is not mentioned.
"Blockchain-based Certificate Transparency and Revocation Transparency" (2018) https://fc18.ifca.ai/bitcoin/papers/bitcoin18-final29.pdf
https://scholar.google.com/scholar?q=related:oEsKmJvdn-MJ:sc...
Who can update and revoke which records in a permissioned blockchain (or a plain old database, for that matter)?
Letsencrypt has a model for proving domain control with ACME; which AFAIU depends upon DNS, too. """
TLA references "Certificate Transparency Using Blockchain" (2018) https://eprint.iacr.org/2018/1232.pdf https://scholar.google.com/scholar?q="Certificate+Transparen...
Thanks for the references! The main issue isn't the support and maintenance of a such distributed network, but its integration with current solutions and avoiding centralized middleware services that will weaken the schema described in the documents.
> The main issue isn't the support and maintenance of a such distributed network,
Running a permissioned blockchain is nontrivial. "Just fork XYZ and call it a day" doesn't quite describe the amount of work involved. There's read latency at scale. There's merging things to maintain vendor strings,
> but its integration with current solutions
- Verify issuee identity
- Update (domain/CN/subjectAltName, date) index
- Update cached cert and CRL bundles
- Propagate changes to all clients
> and avoiding centralized middleware services that will weaken the schema described in the documents.
Eventually, a CDN will look desireable. IPFS may fit the bill, IDK?
> Running a permissioned blockchain is nontrivial.
You are right. It's needed a relevant BFT protocol, a lot of work with masternodes community and smart economic system inside. You can look at an example of a such protocol: https://github.com/Remmeauth/remme-core/tree/dev
google/trillian https://github.com/google/trillian
> Trillian is an implementation of the concepts described in the Verifiable Data Structures white paper, which in turn is an extension and generalisation of the ideas which underpin Certificate Transparency.
> Trillian implements a Merkle tree whose contents are served from a data storage layer, to allow scalability to extremely large trees.
Why Don't People Use Formal Methods?
Which universities teach formal methods?
- q=formal+verification https://www.class-central.com/search?q=formal+verification
- q=formal-methods https://www.class-central.com/search?q=formal+methods
Is formal verification a required course or curriculum competency for any Computer Science or Software Engineering / Computer Engineering degree programs?
Is there a certification for formal methods? Something like for engineer-status in other industries?
What are some examples of tools and [OER] resources for teaching and learning formal methods?
- JsCoq
- Jupyter kernel for Coq + nbgrader
- "Inconsistencies, rolling back edits, and keeping track of the document's global state" https://github.com/jupyter/jupyter/issues/333 (jsCoq + hott [+ IJavascript Jupyter kernel], STLC: Simply-Typed Lambda Calculus)
- TDD tests that run FV tools on the spec and the implementation
What are some examples of open source tools for formal verification (that can be integrated with CI to verify the spec AND the implementation)?
What are some examples of formally-proven open source projects?
- "Quark : A Web Browser with a Formally Verified Kernel" (2012) (Coq, Haskell) http://goto.ucsd.edu/quark/
What are some examples of projects using narrow and strong AI to generate perfectly verified software from bad specs that make the customers and stakeholders happy?
From reading though comments here, people don't use formal methods because: cost-prohibitive, inflexibile, perceived as incompatible with agile / iterative methods that are more likely to keep customers who don't know what formal methods are happy, lack of industry-appropriate regulation, and cognitive burden of often-incompatible shorthand notations.
Almost University of New South Wales (Sydney, Australia) alum, and it's the biggest difference between the Software Engineering course and Comp Sci. There are two mandatory formal methods and more additional ones to take. They are really hard, and mean a lot of the SENG cohort ends up graduating as COMPSCI, since they can't hack it and don't want to repeat formal methods.
Steps to a clean dataset with Pandas
To add to the three points in the article:
Data quality https://en.wikipedia.org/wiki/Data_quality
Imputation https://en.wikipedia.org/wiki/Imputation_(statistics)
Feature selection https://en.wikipedia.org/wiki/Feature_selection
datacleaner can drop NaNs, do imputation with "the mode (for categorical variables) or median (for continuous variables) on a column-by-column basis", and encode "non-numerical variables (e.g., categorical variables with strings) with numerical equivalents" with Pandas DataFrames and scikit-learn. https://github.com/rhiever/datacleaner
sklearn-pandas "[maps] DataFrame columns to transformations, which are later recombined into features", and provides "A couple of special transformers that work well with pandas inputs: CategoricalImputer and FunctionTransformer" https://github.com/scikit-learn-contrib/sklearn-pandas
Featuretools https://github.com/Featuretools/featuretools
> Featuretools is a python library for automated feature engineering. [using DFS: Deep Feature Synthesis]
auto-sklearn does feature selection (with e.g. PCA) in a "preprocessing" step; as well as "One-Hot encoding of categorical features, imputation of missing values and the normalization of features or samples" https://automl.github.io/auto-sklearn/master/manual.html#tur...
auto_ml uses "Deep Learning [with Keras and TensorFlow] to learn features for us, and Gradient Boosting [with XGBoost] to turn those features into accurate predictions" https://auto-ml.readthedocs.io/en/latest/deep_learning.html#...
Reahl – A Python-only web framework
I feel bad about projects like these, I really do. I'm sure a lot of effort have been put in and it probably satisfies what OP wants to do. But in the end, no one serious would ever consider using it.
>But in the end, no one serious would ever consider using it.
Why not?
I would say it is the other way around. For any web application project of decent size there is almost always a transpilation step that converts whatever programming language your source code is in, into JavaScript (usually ES5). We also had very successful and widely used projects like GWT for years (GWT was first released in 2006!!!).
Before GWT, there was Wt framework (C++); and then JWt (Java), which do the server and clientsides (with widgets in a tree).
Wt: https://en.wikipedia.org/wiki/Wt_(web_toolkit)
JWt: https://en.m.wikipedia.org/wiki/JWt_(Java_web_toolkit)
GWT: https://en.wikipedia.org/wiki/Google_Web_Toolkit
Now we have Babel, ES YYYY, and faster browser release cycles.
[deleted]
Ask HN: How can you save money while living on poverty level?
I freelance remotely, making roughly $1200 a month as a programmer because I only work 10 hours maximum each week (limited by my contract). I share the apartment with my mom, and It's a section 8 so our rent contributions are based on the income we make. My contribution towards rent is $400 a month.
Although I make more money than my mom (she's of retirement age and only works 1-2 days a week), while I'm looking for more work I want to figure out how to move out and live more independently on only $1200 a month.
I need to live frugally and want to know what I can cut more easily. I own a used car (already paid in full), and pay my own car insurance, electricity, phone and internet. After all that I have about $400 left each month which can be eaten up by going out or some emergency funds.
More recently I had to pay for my new city parking sticker so that's $100 more in expenses this particular month. I would be satisfied just living in a far off town paying the same $400 a month, I feel my dollars would stretch further since I now get 100% more privacy for the same price.
On top of that this job is a contract job so I need to put money aside to pay my own taxes. This $1200 is basically living on poverty level. Any ideas to make saving work? Is it very possible for people in the US to still save while on poverty?
That's not a living wage (or a full time job). There are lots of job search sites.
Spending some time on a good resume / CV / portfolio would probably be a good investment with positive ROI.
Is there a nonprofit that you could volunteer with to increase your hireability during the other 158 hours of the week?
Or an online course with a credential that may or may not have positive ROI as a resume item?
Is there a code school in your city with a "you don't pay unless you land a full time job with a living wage and benefits" guarantee?
What is your strategy for business and career networking?
From https://westurner.github.io/hnlog/#comment-17894632 :
> Personal Finance (budgets, interest, growth, inflation, retirement)
Personal Finance https://en.wikipedia.org/wiki/Personal_finance
Khan Academy > College, careers, and more > Personal finance https://www.khanacademy.org/college-careers-more/personal-fi...
"CS 007: Personal Finance For Engineers" https://cs007.blog
A DNS hijacking wave is targeting companies at an almost unprecedented scale
> The National Cybersecurity and Communications Integration Center issued a statement [1] that encouraged administrators to read the FireEye report. [2]
[1] https://www.us-cert.gov/ncas/current-activity/2019/01/10/DNS...
[2] https://www.fireeye.com/blog/threat-research/2019/01/global-...
Show HN: Generate dank mnemonic seed phrases in the terminal
From https://github.com/lukechilds/doge-seed :
> The first four words will be a randomly generated Doge-like sentence.
The seed phrases are fully valid checksummed BIP39 seeds. They can be used with any cryptocurrency and can be imported into any BIP39 compliant wallet.
> […] However there is a slight reduction in entropy due to the introduction of the doge-isms. A doge seed has about 19.415 fewer bits of entropy than a standard BIP39 seed of equivalent length.
Can you sign a quantum state?
> Abstract. Cryptography with quantum states exhibits a number of surprising and counterintuitive features. In a 2002 work, Barnum et al. argued informally that these strange features should imply that digital signatures for quantum states are impossible [6].
> In this work, we perform the first rigorous study of the problem of signing quantum states. We first show that the intuition of [6] was correct, by proving an impossibility result which rules out even very weak forms of signing quantum states. Essentially, we show that any non-trivial combination of correctness and security requirements results in negligible security.
> This rules out all quantum signature schemes except those which simply measure the state and then sign the outcome using a classical scheme. In other words, only classical signature schemes exist.
> We then show a positive result: it is possible to sign quantum states, provided that they are also encrypted with the public key of the intended recipient. Following classical nomenclature, we call this notion quantum signcryption. Classically, signcryption is only interesting if it provides superior efficiency to simultaneous encryption and signing. Our results imply that, quantumly, it is far more interesting: by the laws of quantum mechanics, it is the only signing method available.
> We develop security definitions for quantum signcryption, ranging from a simple one-time two-user setting, to a chosen-ciphertext-secure many-time multi-user setting. We also give secure constructions based on post-quantum public-key primitives. Along the way, we show that a natural hybrid method of combining classical and quantum schemes can be used to “upgrade” a secure classical scheme to the fully-quantum setting, in a wide range of cryptographic settings including signcryption, authenticated encryption, and chosen-ciphertext security.
"Quantum signcryption"
Lattice Attacks Against Weak ECDSA Signatures in Cryptocurrencies [pdf]
From the paper:
Abstract. In this paper, we compute hundreds of Bitcoin private keys and dozens of Ethereum, Ripple, SSH, and HTTPS private keys by carrying out cryptanalytic attacks against digital signatures contained in public blockchains and Internet-wide scans.
REMME – A blockchain-based protocol for issuing X.509 client certificates
pity it's blockchain based
It's unclear to me why you would want a distributed PKI to authenticate a centralized app. Or maybe it's only for dapps?
In no particular order, there are a number of blockchain PKI (and DNS (!)) proposals and proofs of concept.
"CertLedger: A New PKI Model with Certificate Transparency Based on Blockchain" (2018) https://arxiv.org/pdf/1806.03914 https://scholar.google.com/scholar?q=related:LF9PMeqNOLsJ:sc...
"TABLE 1: Security comparison of Log Based Approaches to Certificate Management" (p.12) lists a number of criteria for blockchain-based PKI implementations:
- Resilient to split-world/MITM attack
- Provides revocation transparency
- Eliminates client certificate validation process
- Eliminates trusted key management
- Preserves client privacy
- Require external auditing
- Monitoring promptness
... These papers also clarify why a highly-replicated decentralized trustless datastore — such as a blockchain — is advantageous for PKI. WoT is not mentioned.
"Blockchain-based Certificate Transparency and Revocation Transparency" (2018) https://fc18.ifca.ai/bitcoin/papers/bitcoin18-final29.pdf
https://scholar.google.com/scholar?q=related:oEsKmJvdn-MJ:sc...
Who can update and revoke which records in a permissioned blockchain (or a plain old database, for that matter)?
Letsencrypt has a model for proving domain control with ACME; which AFAIU depends upon DNS, too.
California grid data is live – solar developers take note
> It looks like California is at least two generations of technology ahead of other states. Let’s hope the rest of us catch up, so that we have a grid that can make an asset out of every building, every battery, and every solar system.
+1. Are there any other states with similar grid data available for optimization; or any plans to require or voluntarily offer such a useful capability?
Why attend predatory colleges in the US?
> Why would people attend predatory colleges?
Why would people make an investment with insufficient ROI (Return on Investment)?
Insufficient information.
College Scorecard [1] is a database with a web interface for finding and comparing schools according to a number of objective criteria. CollegeScorecard launched in 2015. It lists "Average Annual Cost", "Graduation Rate", and "Salary After Attending" on the search results pages. When you review a detail page for an institution, there are many additional statistics; things like: "Typical Total Debt After Graduation" and "Typical Monthly Loan Payment".
The raw data behind CollegeScorecard can be downloaded from [2]. The "data_dictionary" tab of the "Data Dictionary" spreadsheet describes the data schema.
[1] https://collegescorecard.ed.gov
[2] https://collegescorecard.ed.gov/data/
Khan Academy > "College, careers, and more" [3] may be a helpful supplement for funding a full-time college admissions counselor in a secondary education institution
[3] https://www.khanacademy.org/college-careers-more
(I haven't the time to earn 10 academia.stackexchange points in order to earn the prestigious opportunity to contribute this answer to such a forum with threaded comments. In the academic journal system, journals sell academics' work (i.e. schema.org/ScholarlyArticle PDFs, mobile-compatible responsive HTML 5, RDFa, JSON-LD structured data) and keep all of the revenue).
"Because I need money for school! Next question. CPU: College Textbook costs and CPI: All over time t?!"
Ask HN: Data analysis workflow?
What kind of workflow do you employ when designing a data-flow or analyzing data?
Let me give a concrete example. For the past year, I have been selling stuff on the interwebs through two payment processors one of them being PayPal.
The selling process was put together with a bunch of SaaS hooking everything together through webhooks and notifications.
Now I need to step it that control and produce a proper flow to handle sign up, subscription and payment.
Before doing that I'm analyzing and trying to conciliate all transactions to make sure the books are OK and nothing went unseen. There lies the problem. I have data coming from different sources such as databases, excel files, CSV exports and some JSON files.
At first, I started dealing with it by having all the data in CSV files and trying to make sense of them using code and running queries within the code.
As I found holes in the data I had to dig up more data from different sources and it became a pain to continue with code. I now imported everything into Postgres and have been "debugging" with SQL.
As I advanced through the process I had to generate a lot of routines to collect and match data. I also have to keep all the data files around and organized which is very hard to do because I'm all over the place trying to find where the problem is.
How do you handle with it? What kind of workflow? Any best practices or recommendations from people who do this for a living?
Pachyderm may be basically what you're looking for. It does data version control with/for language-agnostic pipelines that don't need to always redo the ETL phase. https://www.pachyderm.io
Dask-ML works with {scikit-learn, xgboost, tensorflow, TPOT,}. ETL is your responsibility. Loading things into parquet format affords a lot of flexibility in terms of (non-SQL) datastores or just efficiently packed files on disk that need to be paged into/over in RAM. http://ml.dask.org/examples/scale-scikit-learn.html
Sklearn.pipeline.Pipeline API: {fit(), transform(), predict(), score(),} https://scikit-learn.org/stable/modules/generated/sklearn.pi...
https://docs.featuretools.com can also minimize ad-hoc boilerplate ETL / feature engineering :
> Featuretools is a framework to perform automated feature engineering. It excels at transforming temporal and relational datasets into feature matrices for machine learning.
The PLoS 10 Simple Rules papers distill a number of best practices:
"Ten Simple Rules for Reproducible Computational Research" http://www.ploscompbiol.org/article/info%3Adoi%2F10.1371%2Fj...
“Ten Simple Rules for Creating a Good Data Management Plan” http://journals.plos.org/ploscompbiol/article?id=10.1371/jou...
In terms of the scientific method, a null hypothesis like "there is no significant relation between the [independent and dependent] variables" may be dangerously unprofessional p-hacking and data dredging; and may result in an overfit model that seems to predict or classify the training and test data (when split with e.g. sklearn.model_selection.train_test_split and a given random seed).
One of these days (in the happy new year!) I'll get around to updating these notes with the aforementioned tools and docs: https://wrdrd.github.io/docs/consulting/data-science#scienti...
IDK what https://kaggle.com/learn has specifically in terms of analysis workflow? Their docker containers have very many tools configured in a reproducible way: https://github.com/Kaggle/docker-python/blob/master/Dockerfi...
Ask HN: What is your favorite open-source job scheduler
Too many business scripts rely on cron(8) to run. Classic cron cannot handle task duration, fail (only with email), same-task piling, linting, ...
So what is your favorite open-source, easy to bundle/deploy job scheduler, that is easy to use, has logging capacity, config file linting, and can handle common use-cases : kill if longer than, limit resources, prevent launching when previous one is nor finished, ...
systemd-crontab-generator may be usable for something like linting classic crontabs? https://github.com/systemd-cron/systemd-cron
Systemd/Timers as a cron replacement: https://wiki.archlinux.org/index.php/Systemd/Timers#As_a_cro...
Celery supports periodic tasks:
> Like with cron, the tasks may overlap if the first task doesn’t complete before the next. If that’s a concern you should use a locking strategy to ensure only one instance can run at a time (see for example Ensuring a task is only executed one at a time).
http://docs.celeryproject.org/en/latest/userguide/periodic-t...
How to Version-Control Jupyter Notebooks
Mentioned in the article: manual nbconvert, nbdime, ReviewNB (currently GitHub only), jupytext.
Jupytext includes a bit of YAML in the e.g. Python/R/Julia/Markdown header. https://github.com/mwouts/jupytext
Huge +1s for both nbdime and jupytext. Excellent tools both.
Really enjoying jupytext. I do a bunch of my training from Jupyter and it has made my workflow better.
Teaching and Learning with Jupyter (A book by Jupyter for Education)
Still under heavy development, but already useful. Add we accept Pull Requests to add items or fix issues. Join us!
A small suggestion.
Since Jupyter Notebook can be used for both programming and documentation, why don't you use Jupyter Notebook itself as the source of your document?
It is actually very easy to setup a Jupyter Notebook driven .ipynb -> .html publishing pipeline with github + a local Jupyter instance
Here is a toy example (for my own github page)
https://ontouchstart.github.io/
https://github.com/ontouchstart/ontouchstart.github.io
The convert script is here (also a Jupyter Notebook)
https://github.com/ontouchstart/ontouchstart.github.io/blob/...
You got the ideas.
BTW, to make the system fully replicable, I use docker for the local Jupyter instance, which can be launched via the Makefile
https://github.com/ontouchstart/ontouchstart.github.io/blob/...
Here is the custom Dockerfile:
https://github.com/ontouchstart/ontouchstart.github.io/blob/...
Margin Notes: Automatic code documentation with recorded examples from runtime
Slightly related are Go examples - they're tests, and documentation at the same time. It'd be nice if someone hooks in a listener to automatically collect examples tho
And Elixir's doctests https://elixir-lang.org/getting-started/mix-otp/docs-tests-a...
And Python's Axe!
I mean doctest: https://docs.python.org/3/library/doctest.html
1. sys.settrace() for {call, return, exception, c_call, c_return, and c_exception}
2. Serialize as/to doctests. Is there a good way to serialize Python objects as Python code?
3. Add doctests to callables' docstrings with AST
Mutation testing tools may have already implemented serialization to doctests but IDK about docstring modification.
... MOSES is an evolutionary algorithm that mutates and simplifies a combo tree until it has built a function with less error for the given input/output pairs.
Time to break academic publishing's stranglehold on research
Add in that the quality of the system is massively massively broken...peer review is about as accurate as the flip of a coin. It does not promote gorund breaking or novel research, it barely (arguably doesn't) even contribute to quality research. I had a colleague recently be told by a journal editor 'we don't publish critiques from junior scholars.' So much for the nature of peer review being entirely driven by the quality of the work.
As one of those academics...I keep getting requests to peer review, I respectfully make clear I don't review for non open source journals anymore. Same with publishing. I'm not tenure-track so am not primarily evaluated based on output.
Publishing is broken, but it is really just part of the broader and even more broken nature of academic research.
What would be a good alternative to peer-review though? Genuinely interested.
Open publishing and commenting would be a good start---having a dialog, like is done at conferences. Older academic journal articles (pre 1900) read much more like discussions than like the hundred dollar word vomits of modern academic publishing. The broken incentives are at the core of this rotten fruit, though. Just making journals open isn't enough.
We have (almost) open publishing and open commenting. Did that improve anything?
There's open commenting? I've never seen the back and forth of the review process be published. It should be published.
https://hypothes.is supports threaded comments on anything with a URI; including PDFs and specific sentences or figures thereof. All you have to do is register an account and install the browser extension or include the JS in the HTML.
It's based on open standards and an open platform.
W3C Web Annotations: http://w3.org/annotation
About Hypothesis: https://web.hypothes.is/about/
Ask HN: How can I learn to read mathematical notation?
There are a lot of fields I'm interested in, such as machine learning, but I struggle to understand how they work as most resources I come across are full of complex mathematical notation that I never learned how to read in school or University.
How do you learn to read this stuff? I'm frequently stumped by an academic paper or book that I just can't understand due to mathematical notation that I simply cannot read.
https://en.wikipedia.org/wiki/List_of_logic_symbols
https://en.wikipedia.org/wiki/Table_of_mathematical_symbols
These might help a bit.
But as someone with similar problems, I'm beginning to think there's no real solution other than thousands of hours of studying.
There are a number of Wikipedia pages which catalog various uses of symbols for various disciplines:
Outline_of_mathematics#Mathematical_notation https://en.wikipedia.org/wiki/Outline_of_mathematics#Mathema...
List_of_mathematical_symbols https://en.wikipedia.org/wiki/List_of_mathematical_symbols
List_of_mathematical_symbols_by_subject https://en.wikipedia.org/wiki/List_of_mathematical_symbols_b...
Greek_letters_used_in_mathematics,_science,_and_engineering https://en.wikipedia.org/wiki/Greek_letters_used_in_mathemat...
Latin_letters_used_in_mathematics https://en.wikipedia.org/wiki/Latin_letters_used_in_mathemat...
For learning the names of symbols (and maybe also their meaning as conventially utilized in a particular field at a particular time in history), spaced repetition with flashcards with a tool like Anki may be helpful.
For typesetting, e.g. Jupyter Notebook uses MathJax to render LaTeX with JS.
latex2sympy may also be helpful for learning notation.
… data-science#mathematical-notation https://wrdrd.github.io/docs/consulting/data-science#mathema...
New law lets you defer capital gains taxes by investing in opportunity zones
I believe the way that the program works is you can defer taxes from your original capital gains (and the cost basis gets increased so your deferred taxes are less than you would pay otherwise), reinvest them in an "opportunity zone", and not pay capital gains on your investment on the opportunity zone, if you hold it long enough
e.g. Bob bought Apple stock for $50 a share back in the day, and sells it for $200/share. He defers his taxes until 2026. Instead of paying capital gains tax on $150/share, the cost basis is adjusted by 15% so in addition to benefiting from the time value of money, future Bob will only be taxed on $142.50 of capital gains. Bob can buy a house in an "opportunity zone" (from scrolling around the embedded map, there's plenty of million dollar+ houses in these areas. There's also lots of sports teams and stadiums in these areas, so maybe Bob buys an NFL team or a parking lot next to their stadium), rent it out for 10 years, sell it, and not have to pay any capital gains tax on the appreciation. Definitely not a bad deal for him!
Yes its more akin to a 1031 exchange, but opened up so that non-real estate capital gains are eligible.
Is it just capital gains? Wondering if it applies to any other forms of active or passive income.
How are profits from these investments treated?
Can you "swap til you drop" like with a 1031 exchange?
> Is it just capital gains? Wondering if it applies to any other forms of active or passive income.
I would also like some information about this.
+1 for investing in distressed areas; self-nominated with intent or otherwise.
If it's capital gains only, -1 on requiring sale of capital assets in order to be sufficiently incentivized. (Because then the opportunity to tax-advantagedly invest in Opportunity Zones is denied to persons without assets to liquidate; i.e. unequal opportunity).
How to Write a Technical Paper [pdf]
A few years ago i found a great version of a similar piece that proposed something like a 'sandwich' model for each section, and the work as a whole. A sandwich layer was a hook (something interesting), the meat (what you actually did), and a summary.
I failed to save it and i haven't been able to dig it up again, but I liked the idea. The paper was written in the style it described, as well.
5 paragraph essay? https://en.wikipedia.org/wiki/Five-paragraph_essay
> The five-paragraph essay is a format of essay having five paragraphs: one introductory paragraph, three body paragraphs with support and development, and one concluding paragraph. Because of this structure, it is also known as a hamburger essay, one three one, or a three-tier essay.
The digraph presented in the OP really is a great approach, IMHO:
## Introduction
## Related Work, System Model, Problem Statement
## Your Solution
## Analysis
## Simulation, Experimentation
## Conclusion
... "Elements of the scientific method" https://en.wikipedia.org/wiki/Scientific_method#Elements_of_...
no, it was on arxiv somewhere. It was written as a journal article.
edit: AH! it was linked upthread on bioarxiv.
JSON-LD 1.0: A JSON-Based Serialization for Linked Data
JSON-LD 1.1 https://www.w3.org/TR/json-ld11/
"Changes since 1.0 Recommendation of 16 January 2014" https://www.w3.org/TR/json-ld11/#changes-from-10
Jeff Hawkins Is Finally Ready to Explain His Brain Research
Cortical column: https://en.wikipedia.org/wiki/Cortical_column
> In the neocortex 6 layers can be recognized although many regions lack one or more layers, fewer layers are present in the archipallium and the paleopallium.
What this means in terms of optimal artificial neural network architecture and parameters will be interesting to learn about; in regards to logic, reasoning, and inference.
According to "Cliques of Neurons Bound into Cavities Provide a Missing Link between Structure and Function" https://www.frontiersin.org/articles/10.3389/fncom.2017.0004... , the human brain appears to be [at most] 11-dimensional (11D); in terms of algebraic topology https://en.wikipedia.org/wiki/Algebraic_topology
Relatedly,
"Study shows how memories ripple through the brain" https://www.ninds.nih.gov/News-Events/News-and-Press-Release...
> The [NeuroGrid] team was also surprised to find that the ripples in the association neocortex and hippocampus occurred at the same time, suggesting the two regions were communicating as the rats slept. Because the association neocortex is thought to be a storage location for memories, the researchers theorized that this neural dialogue could help the brain retain information.
Re: Topological graph theory [1], is it possible to embed a graph on a space filling curve [2] (such as a Hilbert R-tree [3])?
[1] https://en.wikipedia.org/wiki/Topological_graph_theory
[2] https://en.wikipedia.org/wiki/Space-filling_curve
[3] https://en.wikipedia.org/wiki/Hilbert_R-tree
[4] https://github.com/bup/bup (git packfiles)
Interstellar Visitor Found to Be Unlike a Comet or an Asteroid
It's too bad we're not ready to launch probes to visit and explore such transient objects at a moment's notice.
I think it would make sense to plan a probe mission where the probe would be put into storage to stand by for these kind of events. It would still be a challenge and might be impossible to find a launch window and reach the passing object in time, but it would be worth trying.
Just imagine how tragic it would be, if an ancient artifact of an alien civilization drifts by earth and we don't manage to have at least a look at it.
Wouldn’t such an object be emitting radio signals that earth could receive?
Not if it's something like another civilization's Tesla Roadster.
> Not if it's something like another civilization's Tesla Roadster.
'Oumuamua is red and headed toward Pegasus (the winged horse) after a very long journey starting longtime in spacetime ago. It is wildly tumbling off-kilter and potentially creating a magnetic field that would be useful for interplanetary spacetravel.
They're probably pointing us to somewhere else from somewhere else.
If this is any indication of the state of another civilization's advanced physics, and it missed us by a wide margin, they're probably laughing at our energy and water markets; and indicating that we should be focused on asteroid impact avoidance (and then we will really laugh about rockets and red electromagnetic kinetic energy machines and asteroid mining). https://en.wikipedia.org/wiki/Asteroid_impact_avoidance
"Amateurs"
[We watch it fly by, heads all turning]
Maybe it would've been better to have put alone starman in the passenger seat or two starpeoples total?
Given the skull shape of October 2015 TB145 [1] (due to return in November 2018), maybe 'Oumuamua [2] is a pathology of Mars and an acknowledgement of our spacefaring intentions? Red, subsurface water, disrupted magnetic field.
[1] https://en.wikipedia.org/wiki/2015_TB145
[2] https://en.wikipedia.org/wiki/%CA%BBOumuamua
In regards to a red, unshielded, earth vehicle floating in solar orbit with a suited anthropomorphic creature whose head is too big for the windshield:
"What happened here?"
Publishing more data behind our reporting
Publishing raw data itself is definitely a good start but there also needs to be a push towards a standardized way of sharing data along with it's lineage (dependent sources, experimental design/generation process, metadata, graph relationship of other uses, etc.).
> Publishing raw data itself is definitely a good start but there also needs to be a push towards a standardized way of sharing data along with it's lineage (dependent sources, experimental design/generation process, metadata, graph relationship of other uses, etc.).
Linked Data based on URIs is reusable. ( https://5stardata.info )
The Schema.org Health and Life Sciences extension is ahead of the game here, IMHO. MedicalObservationalStudy and MedicalTrial are subclasses of https://schema.org/MedicalStudy . {DoubleBlindedTrial, InternationalTrial, MultiCenterTrial, OpenTrial, PlaceboControlledTrial, RandomizedTrial, SingleBlindedTrial, SingleCenterTrial, and TripleBlindedTrial} are subclasses of schema.org/MedicalTrial.
A schema.org/MedicalScholarlyArticle (a subclass of https://schema.org/ScholarlyArticle ) can have a https://schema.org/Dataset. https://schema.org/hasPart is the inverse of https://schema.org/isPartOf .
More structured predicates which indicate the degree to which evidence supports/confirms or disproves current and other hypotheses (according to a particular Person or Persons on a given date and time; given a level of scrutiny of the given information) are needed.
In regards to epistemology, there was some work on Fact Checking ( e.g. https://schema.org/ClaimReview ) in recent times. To quote myself here, from https://news.ycombinator.com/item?id=15528824 :
> In terms of verifying (or validating) subjective opinions, correlational observations, and inferences of causal relations; #LinkedMetaAnalyses of documents (notebooks) containing structured links to their data as premises would be ideal. Unfortunately, PDF is not very helpful in accomplishing that objective (in addition to being a terrible format for review with screen reader and mobile devices): I think HTML with RDFa (and/or CSVW JSONLD) is our best hope of making at least partially automated verification of meta analyses a reality.
"#LinkedReproducibility"; "#LinkedMetaAnalyses", "#StudyGraph"
CSV 1.1 – CSV Evolved (for Humans)
Well, if you want to improve tabular data formats:
1. Add a version identifier / content-type on the first line!
2. Create a formal grammar for this CSV format
3. Specify preferred character-encoding
4. Provide some tooling (validation, CSV 1.1 => HTML, CSV => Excel)
5. Add the option to specify column type (string, int, date)
6. Specify ISO-8601 as the preferred date format
7. Allow 'reheading' the columns in the file itself. This is useful in streaming data.
8. Specify the format of the newlines.
CSVW: CSV on the Web https://w3c.github.io/csvw/
"CSV on the Web: A Primer" http://www.w3.org/TR/tabular-data-primer/
"Model for Tabular Data and Metadata on the Web" http://www.w3.org/TR/tabular-data-model/
"Generating JSON from Tabular Data on the Web" (csv2json) http://www.w3.org/TR/csv2json/
"Generating RDF from Tabular Data on the Web" (csv2rdf) http://www.w3.org/TR/csv2rdf/
...
N. Allow authors to (1) specify how many header rows are metadata and (2) what each row is. For example: 7 metadata header rows: {column label, property URI [path], datatype URI, unit URI, accuracy, precision, significant figures}
With URIs, we can merge, join, and concatenate data (when e.g. study control URIs for e.g. single/double/triple blinding/masking indicate that the https://schema.org/Dataset meets meta-analysis inclusion criteria).
"#LinkedReproducibility"; "#LinkedMetaAnalyses"
Ask HN: Which plants can be planted indoors and easily maintained?
Chlorophytum comosum (spider plants) are good air-filtering houseplants that are also easy to take starts of: https://en.wikipedia.org/wiki/Chlorophytum_comosum
Houseplant: https://en.wikipedia.org/wiki/Houseplant
Graduate Student Solves Quantum Verification Problem
"Classical Verification of Quantum Computations" Mahadev. (2018)
https://arxiv.org/abs/1804.01082
https://www.arxiv-vanity.com/papers/1804.01082/
https://scholar.google.com/scholar?cluster=10138991277567750...
The down side to wind power
>To estimate the impacts of wind power, Keith and Miller established a baseline for the 2012‒2014 U.S. climate using a standard weather-forecasting model. Then, they covered one-third of the continental U.S. with enough wind turbines to meet present-day U.S. electricity demand. The researchers found this scenario would warm the surface temperature of the continental U.S. by 0.24 degrees Celsius, with the largest changes occurring at night when surface temperatures increased by up to 1.5 degrees. This warming is the result of wind turbines actively mixing the atmosphere near the ground and aloft while simultaneously extracting from the atmosphere’s motion.
I am confused: How does the warming work exactly and is this actually a global climate effect? Because this part of the article makes it sound to me as if it's just a very localised change of temperature caused by the exchange of different air layers, which can't be right? Because you couldn't really compare that to climate change on a global scale.
The example is clearly hypothetical only. We're never going to cover one third of the continental US with wind turbines.
The more important information to me is that neither wind nor solar have the power density that has been claimed.
For wind, we found that the average power density — meaning the rate of energy generation divided by the encompassing area of the wind plant — was up to 100 times lower than estimates by some leading energy experts
...
For solar energy, the average power density (measured in watts per meter squared) is 10 times higher than wind power, but also much lower than estimates by leading energy experts.
Then you have the separate problem that the wind doesn't always blow and the sun doesn't always shine, so you need a huge storage infrastructure (batteries, presumably) alongside the wind and solar generating infrastructure.
IMO nuclear is the only realistic alternative to coal to provide reliable, zero-emission "base load" power generation. Wind and solar could make sense in some use cases but not in general.
> IMO nuclear is the only realistic alternative to coal to provide reliable, zero-emission "base load" power generation. Wind and solar could make sense in some use cases but not in general.
How much heat energy does a reactor with n meters of concrete around it, located on a water supply in order to use water in an open closed loop, protected with national security resources, waste into the environment?
I'd be interested to see which power sources the authors of this study would choose as a control for these just sensational stats.
From https://news.ycombinator.com/item?id=17806589 :
> Canada (2030), France (2021), and the UK (2025) are all working to entirely phase out coal-fired power plants for very good reasons (such as neonatal health).
Would you burn a charcoal grill in an enclosed space like a garage? No.
Thermodynamics of Computation Wiki
"Quantum knowledge cools computers: New understanding of entropy" (2011) https://www.sciencedaily.com/releases/2011/06/110601134300.h...
> The new study revisits Landauer's principle for cases when the values of the bits to be deleted may be known. When the memory content is known, it should be possible to delete the bits in such a manner that it is theoretically possible to re-create them. It has previously been shown that such reversible deletion would generate no heat. In the new paper, the researchers go a step further. They show that when the bits to be deleted are quantum-mechanically entangled with the state of an observer, then the observer could even withdraw heat from the system while deleting the bits. Entanglement links the observer's state to that of the computer in such a way that they know more about the memory than is possible in classical physics.
"The thermodynamic meaning of negative entropy" (2011) https://www.nature.com/articles/nature10123
Landauer's principle: https://en.wikipedia.org/wiki/Landauer%27s_principle
"Thin film converts heat from electronics into energy" (2018) http://news.berkeley.edu/2018/04/16/thin-film-converts-heat-...
> This study reports new records for pyroelectric energy conversion energy density (1.06 Joules per cubic centimeter), power density (526 Watts per cubic centimeter) and efficiency (19 percent of Carnot efficiency, which is the standard unit of measurement for the efficiency of a heat engine).
"Pyroelectric energy conversion with large energy and power density in relaxor ferroelectric thin films" (2018) https://www.nature.com/articles/s41563-018-0059-8
Carnot heat engine > Carnot cycle, Carnot's theorem, "Real heat engines": https://en.wikipedia.org/wiki/Carnot_heat_engine
Carnot's theorem > Applicability to fuel cells and batteries: https://en.wikipedia.org/wiki/Carnot%27s_theorem_(thermodyna...
> Since fuel cells and batteries can generate useful power when all components of the system are at the same temperature [...], they are clearly not limited by Carnot's theorem, which states that no power can be generated when [...]. This is because Carnot's theorem applies to engines converting thermal energy to work, whereas fuel cells and batteries instead convert chemical energy to work.[6] Nevertheless, the second law of thermodynamics still provides restrictions on fuel cell and battery energy conversion
[deleted]
Is there enough heat energy from a datacenter to -- rather than heating oceans (which can result in tropical storms) -- turn a turbine (to convert heat energy back into electrical energy)?
Is there a statistic which captures the amount of heat energy discharged into ocean/river/lake water? "100% clean energy with PPAs (Power Purchase Agreements)" while bleeding energy into the oceans isn't quite representative of the total system.
"How to Reuse Waste Heat from Data Centers Intelligently" (2016) https://www.datacenterknowledge.com/archives/2016/05/10/how-...
> There are two big issues with data center waste heat reuse: the relatively low temperatures involved and the difficulty of transporting heat. Many of the reuse applications to date have used the low-grade server exhaust heat in an application physically adjacent to the data center, such as a greenhouse or swimming pool in the building next door. This is reasonable given the relatively low temperatures of data center return air, usually between 28° and 35°C (80-95°F), and the difficulty in moving heat around. Moving heat energy frequently requires insulated ducting or plumbing instead of cheap, convenient electrical cables. Trenching and installation to run a hot water pipe from a data center to a heat user may cost as much as $600 per linear foot. Just the piping to share heat with a facility one-quarter mile away might add $750,000 or more to a data center construction project. There’s currently not much that can be done to reduce this cost.
> To address the low-temperature issue, some data center operators have started using heat pumps to increase the temperature of waste heat, making the thermal energy much more valuable, and marketable. Waste heat coming out of heat pumps at temperatures in the range of 55° to 70°C (130-160°F) can be transferred to a liquid medium for easier transport and can be used in district heating, commercial laundry, industrial process heat, and many more. There are even High Temperature (HT) and Very High Temperature (VHT) heat pumps capable of moving low-grade data center heat up to 140°C.
Heat Pump: https://en.wikipedia.org/wiki/Heat_pump
"Data Centers That Recycle Waste Heat" https://www.datacenterknowledge.com/data-centers-that-recycl...
Why Do Computers Use So Much Energy?
> Also, to foster research on this topic we have built a wiki, combining lists of papers, websites, events pages, etc. We highly encourage people to visit it, sign up, and start improving it; the more scientists get involved, from the more fields, the better!
Thermodynamics of Computation Wiki https://centre.santafe.edu/thermocomp/Santa_Fe_Institute_Col...
Justice Department Sues to Stop California Net Neutrality Law
Sorry if I get some basic understanding of the law wrong but...
Isn't this the same thing as regulating car emissions? Doesn't 822 only apply to providers in the state itself? Wouldn't it be that the telecoms are welcome to engage in another method of end-customer billing in other states?
What am I missing?
> Like California’s auto emissions laws that forced automakers to adopt the standards for all production, the state’s new net neutrality rules could push broadband providers to apply the same rules to other states.
I think you're right, the motives are very similar.
> Attorney General Jeff Sessions said that California’s net neutrality law was illegal because Congress granted the federal government, through the F.C.C., the sole authority to create rules for broadband internet providers. “States do not regulate interstate commerce — the federal government does,” Mr. Sessions said in a statement.
I thought Republicans were pro-states rights and limited government? How does their position on this jive with their ideology?
Expansion of federal jurisdiction under the Commerce Clause is an egregious violation of Constitutional law.
Does the federal government have the enumerated right under the Commerce Clause to, for example, ban football for anyone that doesn't have a disability? No!
Was the Commerce Clause sufficient authorization for Federal prohibition of alcohol? No! An Amendment to the Constitution was necessary. And, Federal Alcohol and the unequal necessary State Alcohol prohibitions miserably failed to achieve the intended outcomes.
Where is the limit? How can they claim to support a states' rights, limited government position while expanding jurisdiction under the Interstate Commerce Clause? "Substantially affecting" interstate commerce is a very slippery slope.
Furthermore, de-classification from Title II did effectively - as the current administration's FCC very clearly argued (in favor of special interests over those of the majority) - relieve the FCC of authority to regulate ISPs: they claimed that it's FTC's job and now they're claiming it's their job.
Without Title II classification, FCC has no authority to preempt state net neutrality regulation. California and Washington have the right to regulate ISPs within their respective states.
Outrageous!
Limited government: https://en.wikipedia.org/wiki/Limited_government
States' rights: https://en.wikipedia.org/wiki/States%27_rights
[Interstate] Commerce Clause: https://en.wikipedia.org/wiki/Commerce_Clause
Net neutrality in the United States > Repeal of net neutrality policy: https://en.m.wikipedia.org/wiki/Net_neutrality_in_the_United...
The limit is established, and constantly reevaulated by the Supreme Court. For example it was held that gun control cannot be done through the Commerce Clause.
Car emissions and ISPs are different. As ISPs are very much perfect examples of truly local things (they need to reach your devices with EM signals either via cables or air radio), the Federal government might try to argue that the net neutrality regulation of California affects the whole economy substantially, because it allows too much interstate competition due to the lack of bundling/throttling by ISPs.
Similarly, the problem with car emissions might be that requiring thing at the time of sale affects what kind of cars are sold to CA.
Is the Commerce Clause too vague? Yes. Is there a quick and sane way to fix it? I see none. Is it at least applied consistently? Well, sort of. But we shall see.
ISPs are the very opposite of local, as the only reason I have an ISP is to deliver bits from the rest of the world. Of course, the FCC doesn't seem to understand that...
To summarize the points made in [1]: products can be sold across state lines, internet service sold in one state cannot be sold across state lines.
[1] https://news.ycombinator.com/item?id=18111651
In my opinion, the court has significantly erred in redefining interstate commerce to include (1) intrastate-only-commerce; and (2) non-commerce (i.e. locally grown and unsold wheat)
Furthermore - and this is a bit off topic - unalienable natural rights (Equality, Life, Liberty, and pursuit of Happiness) are of higher precedence. I mention this because this is yet another case where the court will be interpreting the boundary between State and Federal rights; and it's very clear that the founders intended for the powers of the federal government to be limited -- certainly not something that the Commerce Clause should be interpreted to supersede.
What penalties and civil fines are appropriate for States or executive branch departments that violate the Constitution; for failure to uphold Oaths to uphold the Constitution?
The problem is, someone has to interpret what kind of economy the Founders intended.
Is it okay if a State opts to withdraw from the interstate market for wheat? Because without power to meddle with intra-state production, consumption and transactions, it's entirely possible.
White House Drafts Order to Probe Google, Facebook Practices
This sounds like they want the equal representation policies that the Republican Party got rolled back in the 80s (ruled unconstitutional iirc). It’s what allowed the rise of partisan “news”. It seems like any “equal exposure” policies would hit the same issues.
That said the primary “imbalanced exposure” seems to be due to evicting people who simply spend their time attacking minorities, attack equal rights, and promoting violence towards anyone that they dislike. For whatever reason the Republican Party seems to have decided that those people represent “conservative” views that private companies should have to support.
i. e. : People who are exercising their right to free speech.
'Right to free speech' does not exist outside the government. It never has, unless there's an amendment to the first amendment that no one is telling us about..
Repeating what I said in an earlier comment....they were able to grow to the size they have become because they are exempted from liable laws under safe harbor. The argument for that was that they were neutral platforms. They no longer are so they either need to remove the protections or be subject to the first amendment but they should not be able to have it both ways.....and I don't think there is case law to back this one way or the other yet...
> they were able to grow to the size they have become because they are exempted from liable laws under safe harbor
This was not a selective protection. When the government grants limited resources like electromagnetic spectrum and right of way, they're not directly making a monopoly, but the FCC does then claim right to regulate speech.
In the interest of fairness, the FCC classed telecommunication service providers as common carriers; thus authorizing FCC to pass net neutrality protections which require equal prioritization of internet traffic. (No blocking, No throttling, No paid prioritization). The current administration doesn't feel that that's fair, and so they've moved to dismantle said "burdensome regulations".
The current administration is now apparently attempting to argue that information service providers - which are all equally granted safe harbor and obligated to comply with DMCA - have no right to take down abuse and harassment because anti-trust monopoly therefore Freedom of Speech doesn't apply to these corporation persons.
Selective bias, indeed! Broadcast TV and Radio are subject to different rules than Cable (non-broadcast) TV.
Other regimes have attempted to argue that the government has the right to dictate the media as well.
Taking down abuse and harassment is necessary and well within the rights of a person and a corporation in the United States. Taking down certain content is now legally required within 24 hours of notice from the government in the EU.
Where is the line between a media conglomerate that produces news entertainment and an information service provider? If there is none, and the government has the right to regulate "equal time" on non-granted-spectrum media outlets, future administrations could force ConservativeNewsOutletZ and LiberalNewsOutletZ to carry specific non-emergency content, to host abusive and offense rhetoric, and to be sued for being forced to do so because no safe harbor.
Can anyone find the story of how the GOP strongarmed and intimidated Facebook into "equal time" (and then we were all shoved full of apparently Russian conservative "fake news" propaganda) before the most recent election where the GOP won older radio, TV, and print voters and young people didn't vote because it appeared to be unnecessary?
Meanwhile, the current administration rolled back the "burdensome regulation" that was to prevent ISPs from selling complete internet usage history; regardless of age.
Maybe there's an exercise that would be helpful for understanding the "corporate media filter" and the "social media filter"?
You, having no money -- while watching corporate profits soar and income inequality grow to unprecedented heights -- will choose to take a job that requires you to judge whether thousands of reported pieces of content a day are abusive, harassing, making specific threats, inciting specific destructive acts, recruiting for hate groups, depicting abuse; or just good 'ol political disagreement over issues, values, and the appropriate role of the punishing and/or nurturing state. You will do this for weeks or months, because that's your best option, because nobody else is standing in the mirror behind these people who haven't learned to respectfully disagree over facts and data (evidence).
Next, you will plan segments of content time interspersed with ads paid for by people who are trying to sell their products, grow their businesses, and reach people. You will use a limited amount of our limited electromagnetic spectrum which the government has sold your corporate overlords for a limited period of time, contingent upon your adherence to specific and subjective standards of decency as codified in the stated regulations.
In both cases, your objective is to maximize profit for shareholders.
Your target audiences may vary from undefined (everyone watching), to people who only want to review fun things that they agree with in their safe little microcosm of the world, to people who know how to find statistics like corporate profits, personal savings rate, infant morality, healthcare costs per capita, and other Indicators identified as relevant to the Targets and Goals found in the UN Sustainable Development Goals (Global Goals Indicators).
Do you control what the audience shares?
Ask HN: Books about applying the open source model to society
I've been thinking for some time now that as productivity keeps growing, not all people will need to work any more. Society will eventually start to resemble an open source project where a few core contributors do the real work (and get to decide the direction), some others help around, and the majority of people just benefit without having to do anything. I'm wondering if any books have been written to explore this concept further?
> I've been thinking for some time now that as productivity keeps growing, not all people will need to work any more.
How much energy do autotrophs and heterotrophs need to thrive?
"But then we'll be rewarding laziness!"
Some people do enjoy the work they've chosen to do. We enjoy the benefits of upward mobility here in the US; the land of opportunity.
Why would I fully retire at 65 (especially if lifespan extension really is in reach)?
> Society will eventually start to resemble an open source project where a few core contributors do the real work (and get to decide the direction), some others help around, and the majority of people just benefit without having to do anything.
Open-source governance https://en.wikipedia.org/wiki/Open-source_governance
Free-rider problem https://en.wikipedia.org/wiki/Free-rider_problem
As we continue to reward work, the people who are investing in the means of production (energy, labor, automation, raw materials) and science (research and development; education) continue to amass wealth and influence.
This concentration of wealth -- wealth inequality -- has historically presaged and portended unrest.
How contributions to open source projects are reinforced, what motivates people who choose to contribute (altruism, enlightened self interest, compassion, acceptance,), and what makes a competitive and thus sustainable open source project is an interesting study.
... Business models for open-source software: https://en.wikipedia.org/wiki/Business_models_for_open-sourc...
... Political Science: https://en.wikipedia.org/wiki/Political_science
... National currencies are valued in FOREX markets: https://en.wikipedia.org/wiki/Foreign_exchange_market
> I'm wondering if any books have been written to explore this concept further?
"The Singularity is Near: When Humans Transcend Biology" (2005) contains a number of extrapolated predictions; chief among these is that there will continue to be exponential growth in technological change https://en.wikipedia.org/wiki/The_Singularity_Is_Near
... Until we reach limits; e.g. the carrying capacity of our ecosystem, the edge of the universe.
"The Limits to Growth" (1972, 2004) https://en.wikipedia.org/wiki/The_Limits_to_Growth
"Leverage Points: Places to Intervene in a System" (2010) https://news.ycombinator.com/item?id=17781927
Who owns what and who 'gets to' just chill while the solar robots brush their teeth? Heady questions. "Tired yet?"
The Aragon Project has a really interesting take on open source governance:
""" IMAGINE A NATION WITHOUT LAND AND BORDERS
A digital jurisdiction
> Aragon Network will be the first community governed decentralized organization whose goal is to act as a digital jurisdiction, an online decentralized court system that isn’t bound by traditional artificial barriers such as national jurisdictions or the borders of a single country.
Aragon organizations can be upgraded seamlessly using our aragonOS architecture. They can solve disputes between two parties by using the decentralized court system, a digital jurisdiction that operates only online and utilizes your peers to resolve issues.
The Aragon Network Token, ANT, puts the power into the hands of the people participating in the operation of the Network. Every single aspect of the Network will be governed by those willing to make an effort for a better future. """
Today, Europe Lost The Internet. Now, We Fight Back
Here's a quote from this excellent article:
> An error rate of even one percent will still mean tens of millions of acts of arbitrary censorship, every day.
And a redundant -- positively defiant -- link and page title:
"Today, Europe Lost The Internet. Now, We Fight Back." https://www.eff.org/deeplinks/2018/09/today-europe-lost-inte...
Firms with 50 or less employees should stay that small, really.
VPN providers in North and South America FTW.
> VPN providers in North and South America FTW.
Article 13 will affect the entire Internet, not just people in Europe. Most people on the Internet use large, multinational platforms. Those platforms will set rules according to the lowest common denominator, because it's the easiest to implement.
This means that people all over the world are going to have a much more difficult time with any user-generated content. That's true even for user-created content, even if there is no other copyright holder involved (look at how badly Content ID has played out).
Or they just stop doing business in Europe.
Since europe is a quite big market, maybe not an option and easier to geolocate and restrict just EU-Traffic
Or just not, if there's no nexus to europe. I suspect for a small business, the right thing to do would be to simply ignore EU directives. I would not be surprised if the US passes a law to make judgements against US companies without a European nexus (users in Europe would not count) unenforceable in the US. That was done with the SPEECH act to stop libel tourism.
Technically, the phrase "Useful Arts and Sciences" in the Copyright Clause of the US Constitution applies to just that; the definitions of which have coincidentally changed over the years.
The harms to Freedom of Speech -- i.e. impossible 99% accuracy in content filtering still results in far too much censorship -- so significantly outweigh the benefits for a limited number of special interests intending to thwart inferior American information services which also currently host "art" and content pertaining to the "useful arts"; that it's hard to believe this new policy will have it's intended effects.
Haven't there been multiple studies which show that free marketing from e.g. content piracy -- people who experience and recommended said goods at $0 -- is actually a net positive for the large corporate entertainment industry? That, unimpeded, content spreads like the common cold through word of mouth; resulting in greater number of artful impressions.
How can they not anticipate de-listing of EU content from news and academic article aggregators as an outcome of these new policies? (Resulting in even greater outsized impact on one possible front page that consumers can choose to consume)
For countries in the EU with less than 300 million voters, if you want:
- time for your headline: $
- time for your snippet: $$
- time for your og:description: $$
- free video hosting: $$$
- video revenue: $$$$
- < 30% American content: $$$$$
Pay your bill.
And what of academic article aggregators? Can they still index schema:ScholarlyArticle titles and provide a value-added information service for science?
Consumer science (a.k.a. home economics) as a college major
> That's why we need to bring back the old home economics class. Call it "Skills for Life" and make it mandatory in high schools. Teach basic economics along with budgeting, comparison shopping, basic cooking skills and time management.
Some Jupyter notebooks for these topics that work with https://mybinder.org could be super helpful. A self-paced edX course could also be a great intro to teaching oneself though online learning.
* Personal Finance (budgets, interest, growth, inflation, retirement)
* Food Science (nutrition, meal planning for n people, food prep safety, how long certain things can safely be left out on the counter)
* Productivity Skills (GTD, context switching overhead, calendar, email labels, memo app / shared task lists)
There were FACS (Family and Consumer Studies/Sciences) courses in our middle and high school curricula. Nutrition, cooking, sewing; family planning, carry a digital baby for awhile
Home economics https://en.wikipedia.org/wiki/Home_economics
* Family planning
https://en.wikipedia.org/wiki/Family_planning
> * Personal Finance (budgets, interest, growth, inflation, retirement)
Personal Finance https://en.wikipedia.org/wiki/Personal_finance
Khan Academy > College, careers, and more > Personal finance https://www.khanacademy.org/college-careers-more/personal-fi...
"CS 007: Personal Finance For Engineers" https://cs007.blog
https://reddit.com/r/personalfinance/wiki
> * Food Science (nutrition, meal planning for n people, food prep safety, how long certain things can safely be left out on the counter)
Food Science https://en.wikipedia.org/wiki/Food_science
Dietary management https://en.wikipedia.org/wiki/Dietary_management
Nutrition Education: https://en.wikipedia.org/wiki/Nutrition_Education
MyPlate https://en.wikipedia.org/wiki/MyPlate
Healthy Eating Plate https://www.hsph.harvard.edu/nutritionsource/healthy-eating-...
How to make salads, smoothies, sandwiches
How to compost and avoid unnecessary packaging
* School, College, Testing, "How Children Learn"
GED, SAT, ACT, MCAT, LSAT, GRE, GMAT, ASVAB
Defending a Thesis, Bar Exam, Boards
Khan Academy > College, careers, and more https://www.khanacademy.org/college-careers-more
Educational Testing https://wrdrd.github.io/docs/consulting/educational-testing
529 Plans (can be used for qualifying educational expenses for any person) https://en.wikipedia.org/wiki/529_plan
Middle School "Glimpse" project: Past, Present, Future. Present, Future: plan your 4-year highschool course plan, pick 3 careers, pick 3 colleges (and how much they cost)
High school literature: write a narrative essay for college admissions
* Health and Medicine
How to add emergency contact and health information to your phone, carseat (ICE: In Case of Emergency)
How to get health insurance ( https://healthcare.gov/ )
"What's your blood type?" (?!)
Khan Academy > Science > Health and Medicine https://www.khanacademy.org/science/health-and-medicine
Facebook vows to run on 100 percent renewable energy by 2020
Miami Will Be Underwater Soon. Its Drinking Water Could Go First
Now, now, let's focus on the positives here:
- more pollution from shipping routes through the Arctic circle (and yucky-looking icebergs that tourists don't like)
- less beachfront property
- more desalinatable water
- hotter heat
- more revulsive detestable significant others (displaced global unrest)
- costs of responding to natural disasters occurring with greater frequency due to elevated ocean temperatures
- less parking spaces (!)
What are the other costs and benefits here?
I've received a number of downvotes for this comment. I think it's misunderstood, and that's my fault: I should have included [sarcasm] around the whole comment [/sarcasm].
I've written about our need to address climate change here in past comments. I think the administration's climate change denials (see: "climate change politifact') and regulatory rollbacks are beyond despicable: they're sabotaging the United States by allowing more toxic chemicals into the environment that we all share, and allowing more sites that must be protected with tax dollars that aren't there because these industries pay far less than benchmarks in terms of effective tax rate. We know that vehicle emissions, mercury, and coal ash are toxic: why would we allow people to violate the rights of others in that way?
A person could voluntarily consume said toxic byproducts and not have violated their own rights or the rights of others, you understand. There's no medical value and low potential for abuse, so we just sit idly by while they're violating the rights of other people by dumping toxic chemicals into the environment that are both poisonous and strongly linked to climate change.
What would help us care about this? A sarcastic list of additional reasons that we should care? No! Miami underwater during tourist season is enough! I've had enough!
So, my mistake here - my downvote-earning mistake - was dropping my generally helpful, hopeful tone for cynicism and sarcasm that wasn't motivating enough.
We need people to regulate pollution in order to prevent further costs of climate change. Water in the streets holds up commerce, travel, hampers national security, and destroys the road.
We must stop rewarding pollution if we want it - and definitely resultant climate change - to stop. What motivates other people to care?
Free hosting VPS for NGO project?
The Burden: Fossil Fuel, the Military and National Security
Here's a link to the video: https://vimeo.com/194560636
Scientists Warn the UN of Capitalism's Imminent Demise
The actual document title: "Global Sustainable Development Report 2019 drafted by the Group of independent scientists: Invited background document on economic transformation, to chapter: Transformation: The Economy" (2018) https://bios.fi/bios-governance_of_economic_transition.pdf [PDF]
Why I distrust command economies (beyond just because of our experiences with violent fascism and defense overspending and the subsequent failures of various communist regimes):
We have elections today. We don't choose to elect people that regard the environment (our air, water, land, and other natural resources) as our most important focus. A command economy driven by these folks for longer than a term limit would be even more disastrous.
The market does not solve for 'externalities': things that aren't costed in. We must have regulation to counteract the blind optimization for profit (and efficiency) which capitalism rewards most.
Environmental regulation is currently insufficient; worldwide. That is the consensus from the Paris Agreement which 195 countries signed in 2015. https://en.wikipedia.org/wiki/Paris_Agreement
Maybe incentives?
We could sell tokens for how much pollutants we're allowed to f### everyone else over with and penalize exceeding the amount we've purchased. That would incentivize firms to pollute less so that they can save money by having to buy fewer tokens. (Europe does this already; and it's still not going to save the planet from industrial production externalities)
So, while I'm wary of any suggestion that a command economy would somehow bring forth talent in governance, I look to this article for actionable suggestions that penalize and/or incentivize sustainable business and living practices.
Sustainable reporting really is a must: how can I design an investment portfolio that excludes reckless, irresponsible, indifferent, and careless investments and highly values sustainability?
No one likes to be driven by harsh penalties; everyone likes to be rewarded (even with carrots as incentives).
Markets do not solve for long term outcomes. Case in point: the market has not chosen the most energy efficient cryptocurrencies. Is this an information asymmetry issue: people just don't know, or just don't care because the incentives are so alluring, the brand is so strong, or the perceived security assurances of the network outweighs the energy use (and environmental impact) in comparison to dry cleaning and fossil fuel transport.
How would a command economy respond to this? It really is denial and delusion to think that the market will cast aside less energy efficient solutions in order to save the environment all on its own.
So, what do we do?
Do we incentivize getting inefficient vehicles off of the road and into a recycling plant where they belong?
Do we shut down major sources of pollution (coal plants, vehicle emissions)?
Do we create tokens to account for pollution allowances (for carbon and other toxic f###ing chemicals)?
Do we cut irrational subsidies for industries that don't pay their taxes (even when they make money); so that we're aware of the actual costs of our behavior?
Do we grow hemp to absorb carbon, clean up the soil, replace emissions, and store energy?
Who's in the mood to dom these greedy shortsighted idiots into saving themselves and preventing the violation of our right to health (life)? No, you can't because you're busy violating your own rights and finding drugs/druggies and that's not allowed? Is that a lifetime position?
"Go burn a charcoal grill and your gas vehicle in your closed garage for awhile and come talk to me." That's really what we're dealing with here.
Anyways, this paper raises some good points; although I have my doubts about command economies.
[strikethrough] You can't do that to yourself. [/strikethrough] You can't do that to others (even if you pay for their healthcare afterwards).
Where's Captain Planet when you need 'em, anyways?
The problem may actually be solved by a free market... But it has to have different optimizations.
"Value" or "Quality" of a company has to stop being measured in unconstrained growth and instead measured in minimized externalities, while achieving your stated goal.
Accounting has to become a hell of a lot more complicated. We're talking "keep track of your industrial waste product", possibly even having to take responsibility in some manner for dealing with it directly.
There also needs to be some societal change as well. We need to Look at how we've stretched our supply and industry chains worldwide and start to figure out ways to minimize transportation and production costs.
Competition may need to fundamentally change it's nature as well. Trade secrets need to be ripped out into the light of day. It's not a contest of out doing the other guy, but to see who can find a way to make the entire industry more efficient.
Definitely a lot of change needed. That is for sure.
Firefox Nightly Secure DNS Experimental Results
> The experiment generated over a billion DoH transactions and is now closed. You can continue to manually enable DoH on your copy of Firefox Nightly if you like.
...
> Using HTTPS with a cloud service provider had only a minor performance impact on the majority of non-cached DNS queries as compared to traditional DNS. Most queries were around 6 milliseconds slower, which is an acceptable cost for the benefits of securing the data. However, the slowest DNS transactions performed much better with the new DoH based system than the traditional one – sometimes hundreds of milliseconds better.
Long-sought decay of Higgs boson observed at CERN
Furthermore, both teams measured a rate for the decay that is consistent with the Standard Model prediction, within the current precision of the measurement.
And now everyone: Noooo, not again.
(Explanation: it's well-known that the Standard Model can't be completely correct but again and again physicists fail to find an experiment contradicting its predictions, see https://en.wikipedia.org/wiki/Physics_beyond_the_Standard_Mo... for example)
Well, the standard model can be correct. It is correct until some experiment proves otherwise.
It is full of unexplained hadcoded parameters, indeed, which need an explanation from outside of the SM.
> It is full of unexplained hadcoded parameters, indeed, which need an explanation from outside of the SM.
https://en.wikipedia.org/wiki/Magic_number_(programming)#Unn...
> The term magic number or magic constant refers to the anti-pattern of using numbers directly in source code
Who is going to request changes when doing a code review on a pull request from God?
But which branch do you pull from? God has been forked so many times, it's hard to keep track. There are several distros available, so I guess you just get to pick the one that checks the most of your needs at the time.
Sen. Wyden Confirms Cell-Site Simulators Disrupt Emergency Calls
Building a Model for Retirement Savings in Python
re: pulling historical data with pandas-datareader, backtesting, algorithmic trading: https://www.reddit.com/r/Python/comments/7zxptg/pulling_stoc...
re: historical returns
- [The article uses a constant 7% annual return rate]
- "The current average annual return from 1923 (the year of the S&P’s inception) through 2016 is 12.25%." https://www.daveramsey.com/blog/the-12-reality (but that doesn't account for inflation)
- https://www.quantopian.com/posts/56b62019a4a36a79da000059 (300%+ over n years (from a down market))
Is there a Jupyter notebook with this code (with a requirements.txt for https://mybinder.org (repo2docker))?
New E.P.A. Rollback of Coal Pollution Regulations Takes a Major Step Forward
Would you move your family downwind from a coal plant? Why or why not?
Coal ash pollutes air, water, rain (acid rain), crops (our food), and soil. Which rights of victims does coal pollution infringe? Who is liable for the health effects?
Canada (2030), France (2021), and the UK (2025) are all working to entirely phase out coal-fired power plants for very good reasons (such as neonatal health).
~"They're just picking on coal": No, we're choosing renewables that are lower cost AND don't make workers and citizens sick.
If you can mine for coal, you can set up solar panels and wind turbines.
If you can run a coal mine; you can buy some cheap land, put up solar panels and wind turbines, and connect it to the grid.
Researchers Build Room-Temp Quantum Transistor Using a Single Atom
"Quasi-Solid-State Single-Atom Transistors" (2018) https://onlinelibrary.wiley.com/doi/full/10.1002/adma.201801...
New “Turning Tables” Technique Bypasses All Windows Kernel Mitigations
Um – Create your own man pages so you can remember how to do stuff
Interesting project. I believe these two shell functions in the shell's initialization file (~/.profile, ~/.bashrc, etc.) can serve as a poor man's um:
umedit() { mkdir -p ~/notes; vim ~/notes/"$1.txt"; }
um() { less ~/notes/"$1.txt"; }If you write these in .rst, you can generate actual manpages with Sphinx: http://www.sphinx-doc.org/en/master/usage/configuration.html...
sphinx.builders.manpage: http://www.sphinx-doc.org/en/master/_modules/sphinx/builders...
[deleted]
Leverage Points: Places to Intervene in a System
Extremely interesting take, with a lot of good stuff to think about.
I'm unclear about the claim that less economic growth would be better, though, and the author seems very committed to it. I wasn't able to find the article they reference as explaining how less growth is what we really need (J.W. Forrester, World Dynamics, Portland OR, Productivity Press, 1971), and it comes from almost 50 years ago, which might as well be another economic era altogether.
Does anyone know what the arguments are, what assumptions they require, and whether they still apply today? My understanding is that "less growth is better" is a distinctly minority take amongst modern economists, but the rest of this article seems very intelligently laid out, so I'd like to dig deeper.
I've always thought that for any dial we have, there's always an optimal setting, whether it's tax rates, growth rates, birth rates, etc., and blindly pushing one way or the other (like both political parties tend to do) is not helpful, or at the very least merely indicates different value systems.
The book is called "limits to growth", and it's a work of the Club of Rome. You can watch this video to have an idea of their work [1]. At first it's a very strange idea, afterall we work and consume everyday in order to grow the economy. But every healthy system as a homeostastic point, a point where it doesn't need to grow, only to be maintained. We are now a fat society and we need to get our health back, we need to degrow. We need to work much less hours and consume much less. Reducing working hours is a great leverage point. People will start to have time to care about the community, and take care of their own health. Maybe have more time to take a walk instead of using the car. This can improve health and environment but will not contribute to economic growth, healthy people that don't use cars are not good friends of economic growth based on GDP. English is not my mother language, Ursula Le Guin had an eloquent post about this on her blog, but was removed to be on the last book. The post is called "Clinging to a Metaphor" (the metaphor is economic growth) and the book is called "No Time to Spare". [1] https://youtu.be/kz9wjJjmkmc
"The Limits to Growth" (1972) https://en.wikipedia.org/wiki/The_Limits_to_Growth
"Thinking in Systems: a Primer" (2008) https://g.co/kgs/B71ebC
Glossary of systems theory https://en.wikipedia.org/wiki/Glossary_of_systems_theory
Systems Theory https://en.wikipedia.org/wiki/Systems_theory
...
Computational Thinking https://en.wikipedia.org/wiki/Computational_thinking
Which of the #GlobalGoals (UN Sustainable Development Goals) Targets and Indicators are primary leverage points for ensuring - if not growth - prosperity? https://en.wikipedia.org/wiki/Sustainable_Development_Goals
SQLite Release 3.25.0 adds support for window functions
https://www.windowfunctions.com is a good introduction to window functions.
Besides that, the comprehensive testing and evaluation of SQLite never ceases to amaze me. I'm usually hesitant to call software development "engineering", but SQLite is definitely well-engineered.
This looks great, but I couldn't get through the first question on aggregate functions. Are there any SQL books/tutorials that go over things like this?
A lot of material I've seen has been like the classic image of "How to draw an owl. First draw two circles, then draw the rest of the owl", where they tell you the super basic stuff, then assume you know everything.
Ibis uses windowing functions for aggregations if the database supports them. IDK when support for the new SQLite support will be implemented? http://docs.ibis-project.org/sql.html#window-functions
[EDIT]
I created an issue for this here: https://github.com/ibis-project/ibis/issues/1597
Update on the Distrust of Symantec TLS Certificates
Is the certifi bundle (2018.8.13) on PyPI also updated? https://pypi.org/project/certifi/
https://github.com/certifi/certifi.io/issues/18
> Are these still in the bundle?
> Should projects like requests which depend on certifi also implement this logic?
The Transport Layer Security (TLS) Protocol Version 1.3
Academic Torrents – Making 27TB of research data available
All the .torrent files are served over http so with a simple MITM attack a bad actor could swap in their own custom tweaked version of any data set here in order to achieve whatever goals that might serve for the bad actor's interests.
I really wish we could get basic security concepts added to the default curriculum for grade schoolers. You shouldn't need a PhD in computer security to know this stuff. These site creators have PhDs in other fields, but obviously no concept of security. This stuff should be basic literacy for everyone.
> This stuff should be basic literacy for everyone.
Arguably, one compromised PKI x.509 CA jeopardizes all SSL/TLS channel sec if there's no certificate pinning and an alternate channel for distributing signed cert fingerprints (cryptographically signed hashes).
We could teach blockchain and cryptocurrency principles: private/secret key, public key, hash verification; there there's money on the table.
GPG presumes secure key distribution (`gpg --verify .asc`).
TUF is designed to survive certain role key compromises. https://theupdateframework.github.io
1/0 = 0
1/0 = 1(±∞)
https://twitter.com/westurner/status/960508624849244160
> How many times does zero go into any number? Infinity. [...]
> How many times does zero go into zero? infinity^2?
Power Worth Less Than Zero Spreads as Green Energy Floods the Grid
I don't truly understand this "problem". I understand storing the energy in batteries is currently very expensive economically and materially.
However I believe there are plenty of "goods" (irrespective of if they are bulk materials, or partially processed products) which have a high processing energy per volume ratio (this does not need to be recoverable stored energy).
Allow me to give an example: currently we have a drought in Belgium (or at least Flanders). We are not landlocked, there is plenty of water in the sea. Desalination is energy intensive. Instead of only looking at energy storage, why can't we increase the processing capacity (more desalination sites capable of working in parallel), and desalinate say sea water during the energy flood? I don't expect this to be an ideal real world example, only a pattern for identifying such examples: any product (could be composite parts, or bulk material) which is relatively compact and has some high energy per product volume processinng step. Just do the process (desalination, welding some part to another part...) when the sun shines, and store them for later.
Products with very high step energy density are good candidates for storing, and could help flatten daily variations, and perhaps even seasonal variations!
Now some companies would prefer avoiding risk if they don't have guaranteed orders far enough into the future, then perhaps there should be a market for insurance or loans, so that the company is encouraged to take the risk, instead of wasting the cheap energy...
Capital costs vs. marginal costs.
You're going to build a $100M desalination plant and run it for three hours a day? That's a ton of money sitting idle most of the day, far more than what is recovered with zero operating costs.
(This is called the utilization factor -- how long a piece of equipment is used vs. staying idle)
Ideally you want useful processes with low capital costs and expensive marginal/energy costs. Desalination is not one of those.
A desal plant is a useful thing and can be run around the clock off traditional energy sources. The "free energy" hours would help lower the costs.
I can even imagine a SETI-like application where people who over-generate power are able to donate it to causes of their preference...
But if you are using traditional energy sources to run it all the time, it is no longer 'solving' the problem of burning off excess energy during peak renewable times.
Someone is having to build a lot of highly wasteful, redundant infrastructure.
Rational cryptocurrency mining firms can use the excess (unstorable) energy by converting it back to money (while the sun shines and the wind blows).
Money > Energy > Money
> Someone is having to build a lot of highly wasteful, redundant infrastructure.
We're nowhere near having the energy infrastructure necessary to support everyone having an electric vehicle yet.
Energy storage is key to maximizing returns from renewables and minimizing irreversible environmental damage.
Kernels, a free hosted Jupyter notebook environment with GPUs
Hey Ben, are these going to support arbitrary CUDA?
At the moment, we're focused on providing great support for the Python and R analytics/machine learning ecosystems. We'll likely expand this in the future, and in the meantime it's possible to hack through many other usecases we don't formally support well.
How do you handle custom environment requirements, whether it’s Python version, library version, or more complex things in the environment that some code might run on?
Basically, suppose I wanted everything that I could define in a Docker container to be available “as the environment” in which the notebook is running. How do I do that?
I ask because I’ve started to see an alarming proliferation of “notebook as a service” platforms that don’t offer that type of full environment spec, if they offer any configuration of the run time environment at all.
I’ve taught probability and data science at university level and worked in machine learning in a variety of businesses too, and I’d say for literally all use cases, from the quickest little pure-pedagogy prototype of a canned Keras model to a heavily customized use case with custom-compiled TensorFlow, different data assets for testing vs ad hoc exploration vs deployment, etc., the absolutely minimum thing needed before anything can be said to offer “reproducibility” is complete specification of the run time environment and artifacts.
The trend to convince people that a little “poke around with scripts in a managed environment” offering is value-additive is dangerous, very similar to MATLAB’s approach to entwine all data exploration with the atrocious development havits that are facilitated by the console environment (and to specifically target university students with free licenses, to use a drug dealer model to get engineers hooked on MATLAB’s workflow model and use that to leverage employers to oblige by buying and standardizing on abjectly bad MATLAB products).
Any time I meet young data scientists I always try to encourage them to avoid junk like that. It’s vital to begin experiments with fully reproducible artifacts like thick archive files or containers, and to structure code into meaningful reproducible units even for your first ad hoc explorations, and to absolutely always avoid linear scripting as an exploratory technique (it is terrible and ineffective for such a task).
Kaggle Kernels seems like a cool idea, so long as the programmer must fully define artifacts that describe the complete entirety of the run time environment, and nobody is sold on the Kool Aid of just linear scripting in some other managed environment.
Each kernel for example could have a link back to a GitHub repo containing a Dockerfile and build scripts for what defined the precise environment the notebook is running in. Now that’s reproducible.
Here are the Kaggle Kernels Dockerfiles:
- Python: https://github.com/Kaggle/docker-python/blob/master/Dockerfi...
- R: https://github.com/Kaggle/docker-rstats/blob/master/Dockerfi...
https://mybinder.org builds containers (and launches free cloud instances) on demand with repo2docker from a (commit hash, branch, or tag) repo URL: https://repo2docker.readthedocs.io/en/latest/config_files.ht...
That’s a great first step! Adding the ability to customize on a per-notebook basis would be impressive.
Solar and wind are coming. And the power sector isn’t ready
I don't know that fatalism and hopelessness are motivating for decision makers (who are seeking greater margins regardless of policy and lobbies).
Is our transformation to 100% clean energy ASAP a certain eventuality? On a long enough timescale, it would be irrational for utilities to not choose both lower cost and more sustainable environmental impact ('price-rational', 'environment-rational').
We should expect storage and generation costs to continue to fall as we realize even just the current pipeline of capitalizable [storage] research.
Solar energy is free.
Solar Just Hit a Record Low Price in the U.S
Relevant bits:
> “On their face, they’re less than a third the price of building a new coal or natural gas power plant,” Ramez Naam, an energy expert and lecturer at Singularity University, told Earther in an email. “In fact, building these plants is cheaper than just operating an existing coal or natural gas plant.”
> There’s a 30 percent federal investment tax credit for solar projects that helps drive down the cost of this and other solar projects. But Naam said even if you take away that credit, “these bids, un-subsidized, are still cheaper than any new coal or gas plants, and possibly cheaper than operating existing plants.”
(emphasis mine)
>> Relevant bits:
>> “On their face, they’re less than a third the price of building a new coal or natural gas power plant,” Ramez Naam, an energy expert and lecturer at Singularity University, told Earther in an email. “In fact, building these plants is cheaper than just operating an existing coal or natural gas plant.”
>> There’s a 30 percent federal investment tax credit for solar projects that helps drive down the cost of this and other solar projects. But Naam said even if you take away that credit, “these bids, un-subsidized, are still cheaper than any new coal or gas plants, and possibly cheaper than operating existing plants.”
I'm assuming that's without factoring in the health cost externalities.
Yes, this is solely for cost of power. The healthcare savings and quality of life improvements are an additional bonus on top of very cheap power.
https://www.sciencenews.org/article/air-pollution-triggering... (Air pollution is triggering diabetes in 3.2 million people each year)
https://www.scientificamerican.com/article/the-other-reason-... (The Other Reason to Shift away from Coal: Air Pollution That Kills Thousands Every Year)
Causal Inference Book
Causal inference (Causal reasoning) https://en.wikipedia.org/wiki/Causal_inference ( https://en.wikipedia.org/wiki/Causal_reasoning )
Tim Berners-Lee is working a platform designed to re-decentralize the web
Does anyone have some links on Solid that aren't media articles? I can't find anything, not even in Tim's homepage.
Home page: https://solid.mit.edu/
Spec: https://github.com/solid/solid-spec
Source: https://github.com/solid/solid
...
From https://news.ycombinator.com/item?id=16615679 ( https://westurner.github.io/hnlog/#comment-16615679 )
> ActivityPub (and OStatus, and ActivityStreams/Salmon, and OpenSocial) are all great specs and great ideas. Hosting and moderation cost real money (which spammers/scammers are wasting).
> Know what's also great? Learning. For learning, we have the xAPI/TinCan spec and also schema.org/Action.
Mastodon has now supplanted GNU StatusNet.
More States Opting to 'Robo-Grade' Student Essays by Computer
edX can automate short essay grading with edx/edx-ora2 "Open Response Assessment Suite" [1] and edx/ease "Enhanced AI scoring engine" [2].
1: https://github.com/edx/edx-ora2 2: https://github.com/edx/ease
... I believe there's also a tool for peer feedback.
Peer feedback/grading on MOOCs is pretty bad in my experience. There’s too much diversity of skills, language ability, etc. And too many people who bring their own biases and mostly ignore any grading instructions.
Peer discussion and feedback are useful in things like college classes. Much less so with MOOCs.
Ask HN: Looking for a simple solution for building an online course
I want to build an online course on graph algorithms for my university. I've tried to find a solution which would let submit, execute and test student's code (implement an online judge), but have had no success. There are a lot of complex LMS and none of them seem to have this feature as a basic functionality.
Are there any good out-of-box solutions? I'm sure I can build a course using Moodle or another popular LMS with some plugin, but I don't want to spend my time customizing things.
I'm interested both in platforms and self-hosted solutions. Thanks!
Maybe look at Jupyter Notebook? It does much of this out of the box, but may not be exactly what you are looking for.
nbgrader is a "A system for assigning and grading Jupyter notebooks." https://github.com/jupyter/nbgrader
jupyter-edx-grader-xblock https://github.com/ibleducation/jupyter-edx-grader-xblock
> Auto-grade a student assignment created as a Jupyter notebook, using the nbgrader Jupyter extension, and write the score in the Open edX gradebook
... networkx is a graph library written in Python which has pretty good docs: https://networkx.github.io/documentation/stable/reference/
There are a few books which feature networkx.
There is now a backprop principle for deep learning on quantum computers
"A Universal Training Algorithm for Quantum Deep Learning" https://www.arxiv-vanity.com/papers/1806.09729/
New research a ‘breakthrough for large-scale discrete optimization’
Looks like it may be this paper:
"An Exponential Speedup in Parallel Running Time for Submodular Maximization without Loss in Approximation" https://www.arxiv-vanity.com/papers/1804.06355/
The ACM STOC 2018 conference links to "The Adaptive Complexity of Maximizing a Submodular Function" http://dl.acm.org/authorize?N651970 https://scholar.harvard.edu/files/ericbalkanski/files/the-ad...
A DOI URI would be great, thanks.
Wind, solar farms produce 10% of US power in the first four months of 2018
Take note: 10% PRODUCED, not 10% CONSUMED.
This is counting all output by wind and solar regardless if it is needed and usable when the power is being produced. This is quite important because wind and solar are not on-demand sources of power.
> This is counting all output by wind and solar regardless if it is needed and usable when the power is being produced. This is quite important because wind and solar are not on-demand sources of power.
I think you have that backwards: in the US, we lack the ability to scale down coal and nuclear plants. Solar and Wind are generally the first to get pulled offline when generated capacity exceeds demand and storage.
TIL this is called "curtailment" and it's an argument that utilities have used to justify not spending on renewables that are saving the environment from global warming (which is going to require more electricity for air conditioning).
Solar energy production peaks around noon. Demand for electricity peaks in the evening. We need storage (batteries with supercapacitors out front) in order to store the difference between peak generation and peak use. Because they're unable to store this extra energy, they temporarily shut down solar and wind and leave the polluting plants online.
Consumers aren't exposed to daily price fluctuations: they get a flat rate that makes it easy to check their bill; so there's no price incentive to e.g. charge an EV at midday when energy is cheapest.
The 'Duck curve' shows this relation between peak supply and demand in electricity markets: https://en.wikipedia.org/wiki/Duck_curve
Developing energy storage capabilities (through infrastructure and open access basic research that can be capitalized by all) is likely the best solution. According to a fairly recent report, we could go 100% renewable with the energy storage tech that exists today.
But there's no money for it. There's money for subsidizing oil production (regardless of harms (!)), but not so much for wind and solar. There's money for responding to natural disasters caused by global warming, but not so much for non-carbon-based energy sources that don't cause global warming. A film called "The Burden: Fossil Fuel, the Military, and National Security" quotes the actual unsubsidized price of a gallon of gasoline.
Wouldn't it be great if there was some kind of computer workload that could be run whenever energy is cheapest ( 'energy spot instances') so that we can accelerate our migration to renewable energy sources that are saving the environment for future generations? If there were people who had strong incentives to create demand for power-efficient chips and inexpensive clean energy.
Where would be if we had continued with Jimmy Carter's solar panels on the roof of the White House (instead of constant war and meddling with competing oil production regions of the world)?
It's good to see wind and solar growing this fast this year. A chart with cost per kWhr or MWhr would be enlightening.
FDA approves first marijuana-derived drug and it may spark DEA rescheduling
Perhaps someone from the US could help me understand the federal vs state legality of cannabis in the USA?
Can a state override any federal law?
Could federal-level law enforcement theoretically charge someone in a cannabis-legal state for drug offenses?
states cannot override any federal law.
the federal government merely tolerates the semi-autonomous nature of states in order to maintain order. but the legality of the federal government's supremacy is well evolved, its power and might is extremely disproportionally higher than any collective of states, and at this point the semi-autonomous nature of states is pure fiction as the federal government can assume jurisdiction over any intra-state matter if it wanted to through its interstate commerce laws.
yes, federal level law enforcement can theoretically charge someone in a cannabis-legal state for drug offenses. this still happens. the discretion of the words of the President, the heads of the DEA and DOJ prevent the MAJORITY of it from happening and also guide the discretion of the courts and public sympathy. so right now, for the last two administrations it has not been a priority to upset the social order in cannabis-legal states. but it can still happen.
> states cannot override any federal law.
Not actually true: https://en.wikipedia.org/wiki/Nullification_(U.S._Constituti...
> not actually true
a concept which requires agreement from the federal courts themselves, who have never upheld a single argument related to this concept.
not actually what?
Selective incorporation: https://en.wikipedia.org/wiki/Incorporation_of_the_Bill_of_R...
10th Amendment: https://en.wikipedia.org/wiki/Tenth_Amendment_to_the_United_...
> The Tenth Amendment, which makes explicit the idea that the federal government is limited to only the powers granted in the Constitution, has been declared to be a truism by the Supreme Court.
Supremacy clause: https://en.wikipedia.org/wiki/Supremacy_Clause
> The Supremacy Clause of the United States Constitution (Article VI, Clause 2) establishes that the Constitution, federal laws made pursuant to it, and treaties made under its authority, constitute the supreme law of the land.[1] It provides that state courts are bound by the supreme law; in case of conflict between federal and state law, the federal law must be applied. Even state constitutions are subordinate to federal law.[2] In essence, it is a conflict-of-laws rule specifying that certain federal acts take priority over any state acts that conflict with federal law
Natural rights ('inalienable rights': Equal rights, Life, Liberty, pursuit of Happiness): https://en.wikipedia.org/wiki/Natural_and_legal_rights
9th Amendment: https://en.wikipedia.org/wiki/Ninth_Amendment_to_the_United_...
> The Ninth Amendment (Amendment IX) to the United States Constitution addresses rights, retained by the people, that are not specifically enumerated in the Constitution. It is part of the Bill of Rights.
If the 9th Amendment recognizes any unenumerated rights of the people (with Supremacy, regardless of selective incorporation), it certainly recognizes those of the Declaration of Independence ((secession from the king ('CSA')), Equality, Life, Liberty, pursuit of Happiness), our non-binding charter which frames the entirety of the Constitutional convention
All of these things have been interpreted by the courts and their conclusion was not like yours
It is nice that you are interested in these things, but they simply cannot be read verbatim and then extrapolated to other things.
This isn't educational for anybody, this is a view that lacks all consensus and all avenues to ever garner consensus in this country.
Again, I ask you to explain how the current law grants equal rights.
https://news.ycombinator.com/item?id=17401906
> We tend to have issues with Equal rights/protections: slavery, voting rights, [school] segregation. Please help us understand how to do this Equally:
>> Furthermore, (1) write a function to determine whether a given Person has a (natural inalienable) right: what information may you require? (2) write a function to determine whether any two Persons have equal rights.
Abolitionists faced similar criticism from on high.
States Can Require Internet Tax Collection, Supreme Court Rules
I have a hunch that this will, in the end, be a massive win for large retailers vs. small ones. The task of figuring out how to calculate tax for all states is more or less the same amount of work regardless of size, which means for someone like Amazon it's more or less trivial, but for a mom-and-pop store it's a major hassle.
Thankfully with Shopify it is extremely easy and straightforward to manage for my wife's small online store. Their platform does a great job properly charging taxes by state, county and city in certain situations. Then using an inexpensive plan from https://www.taxjar.com/ the entire filing and paying process is 100% automated.
In 10 minutes I was able to file and pay all the sales taxes to several state, dozens of California counties and a handful of cities that charge additional taxes on top.
So you assume. As a small business you are unlikely to be audited, but that software could easily be wrong creating a huge minefield and potential liability.
So that would be the software companies' liability. And such business practice can differentiate good ones from the bad ones. I'm seeing a new business market here even.
https://en.wikipedia.org/wiki/Parable_of_the_broken_window
There is zero economic gain from more complex tax rules. Further, the software does not absolve you of liability. At best they may agree to cover it, but that's unlikely and they can also go broke if they get it wrong.
Actually, current sales tax software provided by South Dakota and other states does absolve a merchant for liability if used to calculate sales tax due.
I agree with this approach!
Actually, the federal government should oblige each member state to provide the algorithm, and sign it cryptographically and have it expire every X fixed time interval, and have signed algorithms for the current and next time interval, so that software can automatically fetch and stay up to date.
Then the "business opportunity" of navigating FUD evaporates. Currently any such enterprise charging for such a service can spend a fraction of their budget lobbying against harmonization...
Since it would be an obligation of the states to the federal government, these algorithms (provided by each member state) should be hosted on a fixed federal government site.
Time to start a petition?
This would reduce costs of tax collection for all parties.
What is the most convenient format for this layered geographic data? Are the tax district boundary polygons already otherwise available as open data? What do localities call these? Sales tax tables, sales tax database, machine-readable flat files in an open format with a common schema?
How much tax revenue should it cost to provide such a service on a national level?
States, Counties, Cities, 'Tax Zones'(?) could be required to host tax.state.us.gov or similar with something like Project Open Data JSONLD /data.json that could be aggregated and shared by a server with a URL registry, a task queue service, and a CDN service.
While the Bitcoin tax payments bill passed the Senate and House in Arizona, it was vetoed in May 2018. Seminole County in Florida now allows tax payment with crytocurrencies such as Bitcoin:
https://cointelegraph.com/news/us-seminole-county-florida-to...
> According to a press release, the county will begin accepting Bitcoin (BTC) and Bitcoin Cash (BCH) to pay for services, including property taxes, driver license and ID card fees, as well as tags and titles. The Seminole County Tax Collector will reportedly employ blockchain payments company BitPay, which will allow the county to receive settlement the next business day directly to its bank account in US dollars.
This could also help reduce the costs of tax collection and possibly increase the likelihood of compliance with the forthcoming tax bills!
these are all very good questions, and only a community discussion of people with the right skills and interests can draft a petition, if enough people contribute to the discussion we can make the proposal more reasonable and robust against valid criticisms... but I believe we can make this happen by just starting the discussion. We can bitch on Hacker News, or we can draft a proposal for the different government levels. The more reasonable we draft it, the higher the probability the petition will be a success. I think it wouldn't be hard to argue against this proposal that a legally enforced computation should be open source, i.e. not just the algorithm for the computation but also all the data lists and boundary polygons used in the algorithm...
Ask HN: Do you consider yourself to be a good programmer?
if not, why? how do you validate your achievements?
> For identifying strengths and weaknesses: "Programmer Competency Matrix":
> - http://sijinjoseph.com/programmer-competency-matrix/
> - https://competency-checklist.appspot.com/
> - https://github.com/hltbra/programmer-competency-checklist
Don’t read too much into that. TDD for example is not leveling up it’s an opinionated approach to development.
Automated testing is not a choice in many industries.
If you're not familiar with TDD, you haven't yet achieved that level of mastery.
There's a productivity boost to being able to change quickly without breaking things.
Is all unit/functional/integration testing and continuous integrating TDD? Is it still TDD if you write the tests after you write the function (and before you commit/merge)?
I think this competency matrix is a helpful resource. And I think that learning TDD is an important thing for a good programmer.
There is absolutely no need to follow TDD to be good at testing.
This is all unfounded conjecture: it seems easier to remember which parameter combinations may exist and need to be tested when writing the function; so "let's all write tests later" becomes a black box exercise which is indeed a helpful perspective for review, but isn't the most effective use of resources.
IMHO being convinced that there's only one true and correct methodology (TDD, Scrum, etc.) or paradigm (functional, objective, reactive programming, etc.) is a sign of being a bad programmer.
A good programmer finds common attributes and behaviors and organizes them into namespaced structs/arrays/objects with functions/methods and tests. Abstractly, which terms should we use to describe hierarchical clusters of things with information and behaviors if not those from a known software development or project management methodology?
And a good programmer asks why people might have spent so much time formalizing project development methodologies. "What sorts of product (team) failures are we dealing with here?" is an expensive question to answer as a team.
By applying tenets of Named agile software development methodologies, teams and managers can feel like they're discussing past and current experiences/successes/failures with comparable implementations of approaches that were or are appropriate for different contexts.
To argue the other side, just cherry picking from different methodologies is creating a new methodology, which requires time to justify basically what we already have terms for on the wall over here.
"We just pop tasks off the queue however" is really convenient for devs but can be kept cohesive by defining sensible queues: [kanban] board columns can indicate task/issue/card states and primacy, [sprint] milestone planning meetings can yield complexity 'points' estimates for completable tasks and their subtasks. With team velocity (points/time), a manager can try to appropriately schedule optimal paths of tasks (that meet the SMART criteria (specific, measurable, achievable, relevant, and Time-bound)); instead of fretting with the team over adjusting dates on a Gantt chart (task dependency graph) deadline, the team can
What about your testing approach makes it 'NOT TDD'?
How long should the pre-release static analysis and dynamic analyses take in my fancy DevOps CI TDD with optional CD? Can we release or deploy right now? Why or why not?
'We can't release today because we spent too much time arguing about quotes like "A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines." ("Self Reliance" 1841. Emerson) and we didn't spec out the roof trusses ahead of time because we're continually developing a new meeting format, so we didn't get to that, or testing the new thing, yet.'
A good programmer can answer the three questions in a regular meeting at any time, really:
> 1. What have you completed since the last meeting?
> 2. What do you plan to complete by the next meeting?
> 3. What is getting in your way?
And:
Can we justify refactoring right now for greater efficiency or additional functionality?
The simple solution there is to simply not use specific parameters (outside ovious edge-cases, ie supplying -1 and 2^63 into your memory allocator). Writing a simple reproducible fuzzer is easy for most contained functions.
I find blackbox testing itself also fairly useful. The part where you forget which parameter combinations may occur can be useful since you now A) rely on documentation you made and B) can write your test independent of how you implemented it just like if you had written it beforehand. (Just don't forget to avoid falling into the 'write test to pass function' trap)
IMHO, it's so much easier to write good, comprehensive tests while writing the function (FUT: function under test) because that information is already in working memory.
It's also easier to adversarially write tests with a fresh perspective.
I shouldn't need to fuzz every parameter for every commit. Certainly for releases.
"Building an AppSec Pipeline: Keeping your program, and your life, sane" https://www.owasp.org/index.php/OWASP_AppSec_Pipeline
I mean, I general don't think you should write fuzzers for absolutely everything (most contained functions => doesn't touch a lot of other stuff and few parameters with a known parameter space)
The general solution is to use whatever testing methodology you are comfortable, that is very effective, very efficient and covers a lot of problem space. Of course no testing method does that so you'll have to constantly balance whatever works best (which is why I think pure TDD is overrated)
> Is all unit/functional/integration testing and continuous integrating TDD?
No. They differentiate in the matrix.
> If you're not familiar with TDD, you haven't yet achieved that level of mastery.
That's not true - I've worked on teams with far lower defect rates than the typical TDD team.
TDD can help keep a developer focused - and this can help overall productivity rates - but it doesn't directly help lower defect rates.
> TDD can help keep a developer focused - and this can help overall productivity rates - but it doesn't directly help lower defect rates.
We would need to reference some data with statistical power; though randomization and control are infeasible: no two teams are the same, no two projects are the same, no two objective evaluations of different apps' teams' defect rates are an apples to apples comparison.
Maybe it's the coverage expectation: do not add code that is not run by at least one test.
Handles are the better pointers
A few years ago I wanted to make a shoot-em-up game using C# and XNA I could play with my kids on an Xbox. It worked fine except for slight pauses for GC every once in a while which ruined the experience.
The best solution I found was to get rid of objects and allocation entirely and instead store the state of the game in per-type tables like in this article.
Then I realized that I didn't need per-type tables at all, and if I used what amounted to a union of all types I could store all the game state in a single table, excluding graphical data and textures.
Next I realized that since most objects had properties like position and velocity, I could update those with a single method by iterating through the table.
That led to each "object" being a collection of various properties acted on by independent methods which were things like gravity and collision detection. I could specify that a particular game entity was subject to gravity or not, etc.
The resulting performance was fantastic and I found I could have many thousands of on-screen objects at the time with no hiccups. The design also made it really easy to save and replay game state; just save or serialize the single state table and input for each frame.
The final optimization would have been to write a language that would define game entities in terms of the game components they were subject to and automatically generate the single class that was union of all possible types and would be a "row" in the table. I didn't get around to that because it just wasn't necessary for the simple game I made.
I was inspired to do this from various articles about "component based game design" published at the time that were variations on this technique, and the common thread was that a hierarchical OO structure ended up adding a lot of unneeded complexity for games that hindered flexibility as requirements changed or different behaviors for in-game entities were added.
Edit: This is a good article on the approach. http://cowboyprogramming.com/2007/01/05/evolve-your-heirachy...
> The final optimization would have been to write a language that would define game entities in terms of the game components they were subject to and automatically generate the single class that was union of all possible types and would be a "row" in the table
django-typed-models https://github.com/craigds/django-typed-models
> polymorphic django models using automatic type-field downcasting
> The actual type of each object is stored in the database, and when the object is retrieved it is automatically cast to the correct model class
...
> the common thread was that a hierarchical OO structure ended up adding a lot of unneeded complexity for games that hindered flexibility as requirements changed or different behaviors for in-game entities were added.
So, in order to draw a bounding box for an ensemble of hierarchically/tree/graph-linked objects (possibly modified in supersteps for reproducibility), is an array-based adjacency matrix still fastest?
Are sparse arrays any faster for this data architecture?
> django-typed-models https://github.com/craigds/django-typed-models
Interesting, never came across it. But I've got a Django GitHub library (not yet open source, because it's in a project I've been working with on and off) that does the same for managing GitHub's accounts. An Organization and User inherit the same properties and I downcast them based on their polymorphic type field.
ContentType.model_class(), models.Model.meta.abstract=True, django-reversion, django-guardian
IDK how to do partial indexes with the Django ORM? A simple filter(bool, rows) could probably significantly shrink the indexes for such a wide table.
Arrays are fast if the features/dimensions are known at compile time (if the TBox/schema is static). There's probably an intersection between object reference overhead and array copy costs.
Arrow (with e.g. parquet on disk) can help minimize data serialization/deserialization costs and maximize copy-free data interoperability (with columnar arrays that may have different performance characteristics for whole-scene transformation operations than regular arrays).
Many implementations of SQL ALTER TABLE don't have to create a full copy in order to add a column, but do require a permission that probably shouldn't be GRANTed to the application user and so online schema changes are scheduled downtime operations.
If you're not discovering new features at runtime and your access pattern is generally linear, arrays probably are the fastest data structure.
Hacker News also has a type attribute that you might say is used polymorphically: https://github.com/HackerNews/API/blob/master/README.md#item...
Types in RDF are additive: a thing may have zero or more rdf:type property instances. RDF quads can be stored in one SQL table like:
_id,g,s,p,o,xsd:datatype,xml:lang
... with a few compound indexes that are combinations of (s,p,o) so that triple pattern graph queries like (?s,?p,1) are fast. Partial indexes (SQLite, PostgreSQL,) would be faster than full-table indexes for RDF in SQL, too.
Neural scene representation and rendering
This work is a natural progression from a lot of other prior work in the literature... but that doesn't make the results any less impressive. The examples shown are amazingly, unbelievably good! Really GREAT WORK.
Based on a quick skim of the paper, here is my oversimplified description of how this works:
During training, an agent navigates an artificial 3D scene, observing multiple 2D snapshots of the scene, each snapshot from a different vantage point. The agent passes these snapshots to a deep net composed of two main parts: a representation-learning net and a scene-generation net. The representation-learning net takes as input the agent's observations and produces a scene representation (i.e., a lower-dimensional embedding which encodes information about the underlying scene). The scene-generation network then predicts the scene from three inputs: (1) an arbitrary query viewpoint, (2) the scene representation, and (3) stochastic latent variables. The two networks are trained jointly, end-to-end, to maximize the likelihood of generating the ground-truth image that would be observed from the query viewpoint. See Figure 1 on Page 15 of the Open Access version of the paper. Obviously I'm playing loose with language and leaving out numerous important details, but this is essentially how training works, as I understand it based on a first skim.
EDIT: I replaced "somewhat obvious" with "natural," which better conveys what I actually meant to write the first time around.
I, literally just 15 minutes ago, had a chat with a friend of mine exactly about how what we are doing right now with computer vision is all based on a flawed premise (supervised 2D training set). The human brain works in 3D space (or 3D+time) and then projects all this knowledge in a 2D image.
Here I was, thinking I finally had thought of a nice PhD project and then Deepmind comes along and gets the scoop! Haha.
"Spatial memory" https://en.wikipedia.org/wiki/Spatial_memory
It may be splitting hairs, but I think the mammalian brain, at least, can simulate/remember/imagine additional 'dimensions' like X/Y/Z spin, derivatives of velocity like acceleration/jerk/jounce.
Is space 11 dimensional (M string theory) or 2 dimensional (holographic principle)? What 'dimensions' does the human brain process? Is this capacity innate or learned; should we expect pilots and astronauts to have learned to more intuitively cognitively simulate gravity with their minds?
New US Solar Record – 2.155 Cents per KWh
"Cost of electricity by source" https://en.wikipedia.org/wiki/Cost_of_electricity_by_source
"Electricity pricing" https://en.wikipedia.org/wiki/Electricity_pricing
> United States 8 to 17 ; 37[c] 43[c] [cents USD/kWh]
Ask HN: Is there a taxonomy of machine learning types?
Besides classification and regression, and the unsupervised methods for principle components, clustering and frequent item-sets, what tools are there in the ML toolkit and what kinds of problems are amenable to their use?
Outline of Machine Learning https://en.wikipedia.org/wiki/Outline_of_machine_learning
Machine learning # Applications https://en.wikipedia.org/wiki/Machine_learning#Applications
"machine learning map" image search: https://www.google.com/search?q=machine+learning+map&tbm=isc...
Senator requests better https compliance at US Department of Defense [pdf]
The "Mozilla SSL Configuration Generator" has a checkbox for 'HSTS enabled?' and can generate SSL/TLS configs for Apache, Nginx, Lighttpd, HAProxy, AWS, ELB. https://mozilla.github.io/server-side-tls/ssl-config-generat...
You can select 'nginx', then 'modern', and then 'apache' for a modern Apache configuration.
Are the 'modern' configs FIPS compliant?
What browsers/tools does requiring TLS 1.3 break?
Banks Adopt Military-Style Tactics to Fight Cybercrime
> In a windowless bunker here, a wall of monitors tracked incoming attacks — 267,322 in the last 24 hours, according to one hovering dial, or about three every second — as a dozen analysts stared at screens filled with snippets of computer code.
> Cybercrime is one of the world’s fastest-growing and most lucrative industries. At least $445 billion was lost last year, up around 30 percent from just three years earlier, a global economic study found, and the Treasury Department recently designated cyberattacks as one of the greatest risks to the American financial sector.
Is this type of monitoring possible (necessary, even) with blockchains? Blockchains generally silently disregard bad/invalid transactions. Where could discarded/disregarded transactions and forks be reported to in a decentralized blockchain system? Who would pay for log storage? How redundantly replicated should which data be?
How DDOS resistant are centralized and decentralized blockchains?
Exchanges have risk. In terms of credit fraud: some crypto asset exchanges do allow margin trading, many credit card companies either refuse transactions with known exchanges or charge cash advance interest rates, and all transactions are final.
Exchanges hold private keys for customers' accounts, move a lot to offline cold storage, and maybe don't do a great job of explaining that YOU SHOULD NOT LEAVE MONEY ON AN EXCHANGE. One should transfer funds to a different account; such as a hardware or paper wallet or a custody service.
Do/can crypto asset exchanges participate in these exercises? To what extent do/can blockchains help solve for aspects of our unfortunately growing cybercrime losses?
Premined blockchains could reportedly handle card/chip/PIN transaction volumes today.
No, Section 230 Does Not Require Platforms to Be “Neutral”
> It’s foolish to suggest that web platforms should lose their Section 230 protections for failing to align their moderation policies to an imaginary standard of political neutrality. Trying to legislate such a “neutrality” requirement for online platforms—besides being unworkable—would be unconstitutional under the First Amendment.
... https://en.wikipedia.org/wiki/Section_230_of_the_Communicati...
Ask HN: Do battery costs justify “buy all sell all” over “net metering”?
Are batteries the primary justification for "buy all sell all" over "net metering"?
Are next-gen supercapacitors the solution?
> Ask HN: Do battery costs justify "buy all sell all" over "net metering"?
> Are batteries the primary justification for "buy all sell all" over "net metering"?
> Are next-gen supercapacitors the solution?
With "Net Metering", electric utilities buy consumers' excess generated energy at retail or wholesale rates. https://en.wikipedia.org/wiki/Net_metering
With "Buy All, Sell All", electric utilities require consumers to sell all of the energy they generate from e.g. solar panels (usually at wholesale prices, AFAIU) and buy all of the energy they consume at retail rates. They can't place the meter after any local batteries.
Do I have this right?
Net metering:
(used-generated) x (retail || wholesale)
Buy all, sell all:
(used x retail) - (generated x wholesale)
For the energy generating consumer, net metering is a better deal: they have power when the grid is down, and they keep or earn more for the energy generation capability they choose to invest in.
Break-even on solar panels happens sooner with net metering.
Utilities argue that: maintaining grid storage and transfer costs money, which justifies paying energy generating consumers less than they pay for more constant sources of energy like dams, wind farms, and commercial solar plants.
Building a two-way power transfer grid costs money. Batteries require replacement after a limited number of cycles. Spiky or bursting power generation is not good for batteries because they don't get a full cycle. [Hemp] supercapacitors can smooth out that load and handle many more partial charge and discharge cycles.
Is energy storage the primary justifying cost driver for "buy all, sell all"?
What investments are needed in order to more strongly incentivize clean energy generation? Do we need low cost supercapacitors to handle the spiky load?
Are these utilities granted a monopoly? Are they price fixing?
Energy demand from blockchain mining has not managed to keep demand constant so that utilities have profit to invest in clean energy generation and a two-way smart grid that accommodates spiky consumer energy generation. Demand for electricity is falling as we become less wasteful and more energy efficient. As the cost of renewable energy continues to fall (and become less expensive than nonrenewables), there should be more margin for energy utilities which cost-rationally and environmentally-rationally choose to buy renewable energy and sell it to consumers.
Please correct me with the appropriate terminology.
How can we more strongly incentivize consumer solar panel investments?
Here's a discussion about the lower costs of hemp supercapacitors as compared with graphene super capacitors: https://news.ycombinator.com/item?id=16800693
""" Hemp supercapacitors might be a good solution to the energy grid storage problem. Hemp absorbs carbon, doesn't leave unplowable roots in the fields, returns up to 70% of nutrients to the soil, and grows quickly just about anywhere. Hemp bast fiber is normally waste. Hemp anodes for supercapacitors are made from the bast fiber that is normally waste.
Graphene is very useful; but industrial production of graphene is dangerous because lungs and blood-brain barrier.
Hemp is an alternative to graphene for modern supercapacitors (which now have much greater [energy density] in wH/kg)
"Hemp Carbon Makes Supercapacitors Superfast” https://www.asme.org/engineering-topics/articles/energy/hemp...
> “Our device’s electrochemical performance is on par with or better than graphene-based devices,” Mitlin says. “The key advantage is that our electrodes are made from biowaste using a simple process, and therefore, are much cheaper than graphene.”
> Graphene is, however, expensive to manufacture, costing as much as $2,000 per gram. [...] developed a process for converting fibrous hemp waste into a unique graphene-like nanomaterial that outperforms graphene. What’s more, it can be manufactured for less than $500 per ton.
> Hemp fiber waste was pressure-cooked (hydrothermal synthesis) at 180 °C for 24 hours. The resulting carbonized material was treated with potassium hydroxide and then heated to temperatures as high as 800 °C, resulting in the formation of uniquely structured nanosheets. Testing of this material revealed that it discharged 49 kW of power per kg of material—nearly triple what standard commercial electrodes supply, 17 kW/kg.
https://scholar.google.com/scholar?hl=en&q=hemp+supercapacit....
https://en.wikipedia.org/wiki/Supercapacitor
I feel like a broken record mentioning this again and again. ""'
Portugal electricity generation temporarily reaches 100% renewable
Currently we are passing through an atypical wind and rain period. Our dams are full and pouring out water as we are producing more energy than we consume.
Despite all of this energy prices don't drop as they are bound to one of the few legal monopolies in Europe: the grid operation cartel (REN in the article, there is only one allowed for each EU country).
Also this feels as fake news since we have a few very old (and historically insecure, like sines power plant and carregado power plant) fossil fuel power plants that are still operating with no signs of slowing down while making profits through several scams in chemical engineering their way into the 0 emissions lot.
are batteries (or some sort of storage) in the infrastructure plan for the future?
Hemp supercapacitors might be a good solution to the energy grid storage problem. Hemp absorbs carbon, doesn't leave unplowable roots in the fields, returns up to 70% of nutrients to the soil, and grows quickly just about anywhere.
Hemp bast fiber is normally waste. Hemp anodes for supercapacitors are made from the bast fiber that is normally waste.
Graphene is very useful; but industrial production of graphene is dangerous because lungs and blood-brain barrier.
Hemp is an alternative to graphene for modern supercapacitors (which now have much greater power density in wH/kg)
"Hemp Carbon Makes Supercapacitors Superfast” https://www.asme.org/engineering-topics/articles/energy/hemp...
> “Our device’s electrochemical performance is on par with or better than graphene-based devices,” Mitlin says. “The key advantage is that our electrodes are made from biowaste using a simple process, and therefore, are much cheaper than graphene.”
> Graphene is, however, expensive to manufacture, costing as much as $2,000 per gram. [...] developed a process for converting fibrous hemp waste into a unique graphene-like nanomaterial that outperforms graphene. What’s more, it can be manufactured for less than $500 per ton.
> Hemp fiber waste was pressure-cooked (hydrothermal synthesis) at 180 °C for 24 hours. The resulting carbonized material was treated with potassium hydroxide and then heated to temperatures as high as 800 °C, resulting in the formation of uniquely structured nanosheets. Testing of this material revealed that it discharged 49 kW of power per kg of material—nearly triple what standard commercial electrodes supply, 17 kW/kg.
https://scholar.google.com/scholar?hl=en&q=hemp+supercapacit...
https://en.wikipedia.org/wiki/Supercapacitor
I feel like a broken record mentioning this again and again.
If you're going to be mentioning this again in the future please correct your usage of power/energy density. Power density is measured in W/kg, energy density is measured in Wh/kg. Supercapacitors tend to excel in the former but be poor in the latter. You mentioned power density but used units for energy density. This happens so often in media that I feel the need to correct it even in a comment.
> please correct your usage of power/energy density. Power density is measured in W/kg, energy density is measured in Wh/kg. Supercapacitors tend to excel in the former but be poor in the latter.
I'd update the units; good call. You may have that confused? Traditional supercapacitors have had lower power density and faster charging/discharging. Graphene and hemp somewhat change the game, AFAIU.
It makes sense to put supercapacitors in front of the battery banks because they last so many cycles and because they charge and discharge so quickly (a very helpful capability for handling spiky wind and solar loads).
I think you may still be a little confused. Power density is the rate at which energy can be added to or drawn from the the cell per unit mass. So faster charging and discharging means high power density. Energy density is the total amount of energy that can be stored per unit mass. Supercapacitors are typically higher in power density and lower in energy density than batteries[1].
You're right that it makes sense to put supercapacitors in front of the battery banks for the reasons you said.
[1] http://berc.berkeley.edu/storage-wars-batteries-vs-supercapa...
I must have logically assumed that rate of charge and discharge include time (hours) in the unit: Wh/kg.
My understanding is that there's usually a curve over time t that represents the charging rate from empty through full.
[edit]
"C rate"
Battery_(electricity)#C_rate https://en.wikipedia.org/wiki/Battery_(electricity)#C_rate
Battery_charger#C-rates https://en.wikipedia.org/wiki/Battery_charger#C-rates
> Charge and discharge rates are often denoted as C or C-rate, which is a measure of the rate at which a battery is charged or discharged relative to its capacity. As such the C-rate is defined as the charge or discharge current divided by the battery's capacity to store an electrical charge. While rarely stated explicitly, the unit of the C-rate is [h^−1], equivalent to stating the battery's capacity to store an electrical charge in unit hour times current in the same unit as the charge or discharge current.
It does sound amazing and economical, like almost too good to be true, but I very much hope it is true. What are the downsides? Is there a degradation problem or something similar? Other than the stoner connection in people's minds, what kind of resistance is there to this? Why isn't it widely known?
You know, I'm not sure. This article is from a few years ago now and there's not much uptake.
It may be that most people dismiss supercapacitors based on the stats for legacy (pre-graphene/pre-hemp) supercapacitors: large but quick and long-lasting.
It may be that hemp is taxed at up to 90% because it's a controlled substance in the US (but not in Europe, Canada, or China; where we must import shelled hemp seeds from). A historical accident?
GPU Prices Drop ~25% in March as Supply Normalizes
How do these new GPUs compare to those from 10 years ago in terms of FLOPs per Watt? https://en.wikipedia.org/wiki/Performance_per_watt
The new ASICs for Ethereum mining can't be solely responsible for this percent of the market.
(Note that NVIDIA's stock price is up over 1700% over the past 10 years. And that Bitcoin mining on CPUs and GPUs hasn't been profitable for quite awhile. In 2007, I don't think we knew that hashing could be done on GPUs; though there were SSL accelerator cards that were mighty expensive)
Apple says it’s now powered by renewable energy worldwide
As this article [1] explains, Apple does not (and cannot) actually run on 100% renewable energy globally, as any of its stores/premises/facilities that are connected to local municipal power grids will use whatever power generation method is used on that grid, and that is still likely to be fossil fuel in most locations.
But they purchase Renewable Energy Certificates to offset their use of non-renewable energy, so they can make the claim that their net consumption of non-renewable electricity is negative.
[1] https://www.theverge.com/2018/4/9/17216656/apple-renewable-e...
This is weird. So if I run a wind farm that generates 1 MW(...h? why is this an energy unit and not power? but whatever...) and I use all of that electricity myself, I can also sell 1 REC to someone else so that they claim they run on green electricity. Which means that now either (a) I have to legally claim I run on dirty electricity (which is a lie on its face and make no sense???) or (b) we both claim we run on green power, double-dipping and screwing up the accounting of greenness.
Am I misunderstanding something? How does this work?
No that's how it works. That's also the reason why Norway, despite only using hydro, has only 40-50% renewable energy in some statistics. They sell green energy certificates to consumers abroad (e.g. in Germany). Officially, Norwegians then use coal power whereas in reality it's all hydro power. There isn't even enough transmission capacity to the south to get that kind of exchange physically.
Wow! And I just realized there seems to be another loophole: that means (say) a company like Apple could start a separate power company in Norway based on hydro power, have that company completely waste 100% of the energy it produce there, and yet "buy" the equivalent REC in another jurisdiction where they run on coal and suddenly get to 100% "green" power... potentially even making more money in tax credits, if there are any, all while consuming more and more dirty power without actually helping anybody shift to renewable energy. Right?
I think it comes down to giving them credit for funding the construction of massive clean energy projects, even if they don't exclusively use the electricity from that project themselves.
Look at Microsoft. They just funded a deal to build out an absolutely massive solar farm in Virginia.
https://blogs.microsoft.com/on-the-issues/2018/03/21/new-sol...
They do have facilities in state that will only need a fraction of that power to be 100% green, and the excess will be pumped into the local Virginia power grid and used by consumers there.
Basically, Microsoft is saying that they funded the project to generate excess green energy in one place to offset the dirty energy they consume in areas where there is no local green power option available.
100% renewable energy by purchasing and funding renewable energy is an outstanding acheivement.
Is there another statistic for measuring how many KWhr or MWhr are sourced directly from renewable energy sources (or, more logically, 'directly' from batteries + hemp supercapacitors between use and generation)?
Hackers Are So Fed Up with Twitter Bots They’re Hunting Them Down Themselves
This is an interesting approach. Maybe Twitter shouldn't solve the fake accounts problem directly, maybe they should come up with an evaluation criteria and then create a market for identifying fake accounts.
If their evaluation criteria is good, they could get away with 0 cost to build the best possible system (motivated by competition on a market).
There's an open call for papers/proposals for handling the deluge. "Funding will be provided as an unrestricted gift to the proposer's organization(s)" ... "Twitter Health Metrics Proposal Submission" https://blog.twitter.com/official/en_us/topics/company/2018/...
A real hacker move would be to just leave Twitter and go to Mastodon https://joinmastodon.org
Are you suggesting that Mastodon has a better system for identifying harassment, spam, and spam accounts? Or that, given that they're mostly friendly early adopters, they haven't yet encountered the problem?
It seems to me like you don't understand the crucial difference between Twitter and Mastodon.
There's no such thing as Mastodon, a singular social network. Mastodon is a series of instances that talk to each other. A sysadmin running the instance can do whatever he pleases in his instance, including closing the registration, banning entire instances from communicating with his instance, and enforcing whichever rules he wants to enforce.
Mastodon doesn't deal with such issues at all. It's sysadmins running Mastodon instances that are supposed to deal with such issues.
It's more like reddit, where mods of subreddits have nearly complete authority over their own space on the social network, than it is like Twitter, in which a single entity is in charge.
Mastodon is a federated system like StatusNet/GNU Social.
So, in your opinion, Mastodon nodes - by virtue of being federated - would be better equipped to handle the spam and harassment volume that Twitter is subject to?
I find that hard to believe.
ActivityPub (and OStatus, and ActivityStreams/Salmon, and OpenSocial) are all great specs and great ideas. Hosting and moderation cost real money (which spammers/scammers are wasting).
Know what's also great? Learning. For learning, we have the xAPI/TinCan spec and also schema.org/Action.
“We’re committing Twitter to increase the health and civility of conversation”
First Amendment protections apply to suits brought by the government. Civil suits are required to prove damages ("quantum of loss").
There are many open platforms. (I've contributed to those as well). Some are built on open standards. None of said open platforms have procedures or resources for handling the onslaught of disrespectful trash that the people we've raised eventually use these platforms for communicating at other people who have feelings and understand the Golden Rule.
https://en.wikipedia.org/wiki/Golden_Rule
The initial early adopters (who have other better things to do) are fine: helpful, caring, critical, respectful; healthy. And then everyone else comes surging in with hate, disrespect, and vitriol; unhealthy. They don't even realize that being hateful and disrespectful is making them more depressed. They think that complaining and talking smack to people is changing the world. And then they turn off the phone or log out of the computer, and carry on with their lives.
No-one taught them to be the positive, helpful energy they want to attract from the world. No-one properly conditioned them to either respectfully disagree according to the data or sit down and listen. No-one explained to them that a well-founded argument doesn't fit in 140 or 280 characters, but a link and a headline do. No-one explained to them that what they write on the internet lasts forever and will be found by their future interviewers, investors, jurors, and voters. No-one taught them that being respectful and helpful in service of other people - of the group's success, of peaceful coexistence - is the way to get ahead AND be happy. "No-one told me that."
Shareholders of public corporations want to see growth in meaningless numbers, foreign authoritarian governments see free expression as a threat to their ever-so-fragile self-perceptions, political groups seek to frame and smear and malign and discredit (because they are so in need of group acceptance; because money still isn't making them happy), and there are children with too much free time reading all of these.
No-one is holding these people accountable: we need transparency and accountability. We need to focus on more important goals and feel good about helping; about volunteering our time to help others be happier.
Instead, now that these haters and scam artists have all self-identified, we must spend our time conditioning their communications until they learn to respectfully disagree on facts and data or go somewhere else. "That's how you feel? Great. How does that make your victim feel?" is the confrontation that some people are seeking from companies that set out to serve free speech and provide a forum for citizens to share the actual news.
Who's going to pay for that? Can they sue for their costs and losses? Advertisers do not want a spot next to hateful and disrespectful.
"How dare you speak of censorship in such veiled terms!?" Really? They're talking about taking down phrases like "kill" and "should die"; not phrases like "I disagree because:"
So, now, because there are so many hateful economically disadvantaged people in the world with nothing better to do and no idea how to run a business or keep a job with benefits, these companies need to staff 24 hour a day censors to take down the hate and terror and gang recruiting within one hour. What a distorted mirror of our divisively fractured wealth inequality, indeed.
"Ban gangs ASAP, please: they'll just go away"
How much does it cost to pay prison labor to redundantly respond to this trash? Are those the skills they need to choose a different career with benefits and savings that meet or exceed inflation when they get out?
What is the procedure for referring threats of violence to justice in your jurisdiction? Are there wealthy individuals in your community who would love to contribute resources to this effort? Maybe they have some region-specific pointers for helping the have-nots out here trolling like it's going to get them somewhere they want to be in life?
Let me share a little story with you:
A person walks into a bar/restaurant, flicks off the bartender/waiter, orders 5 glasses of free water, starts plastering ads to the walls and other peoples' tables, starts making threats to groups of people cordially conversing, and walks out.
Gitflow – Animated in React
Thanks! A command log would be really helpful too.
The HubFlow docs contain GitFlow docs and some really helpful diagrams: https://datasift.github.io/gitflow/IntroducingGitFlow.html
I change the release prefix to 'v' so that the git tags for the release look like 'v0.0.1' and 'v0.1.0':
git config --replace-all gitflow.prefix.versiontag v
git config --replace-all hubflow.prefix.versiontag v
Sure.. will add the command log
Ask HN: How feasible is it to become proficient in several disciplines?
For example to become a professional in:
- back-end api development
- DevOps
- Data Engineer (big data, data science, ML, etc)
It is feasible, though as with any type of specialization, you're then a "jack of all trades, master of none". Maybe a title like "Full Stack Data Engineer" would be descriptive.
You could write an OAuth API for accepting and performing analysis of datasets (model fitting / parameter estimation; classification or prediction), write a test suite, write Kubernetes YAML for a load-balanced geodistributed dev/test/prod architecture, and continuously deploy said application (from branch merges, optionally with a manual confirmation step; e.g. with GitLab CI) and still not be an actual Data Engineer.
After rising for 100 years, electricity demand is flat
Cryptocurrency mining is about to consume more electricity than home usage in Iceland.[1] I assume it's similar in other places that have cheaper electricity.
Seems that power companies should encourage consumers to mine Bitcoin. Problem solved.
[1] https://qz.com/1204840/iceland-will-use-more-electricity-min...
> Seems that power companies should encourage consumers to mine Bitcoin. Problem solved.
Blockchains will likely continue to generate considerable demand for electricity for the foreseeable future.
Blockchain firms can locate where energy is cheapest. Currently that's in countries where energy prices go negative due to excess capacity and insufficient energy storage resources (batteries, [hemp/graphene] supercapacitors, water towers).
With continued demand, energy companies can continue to invest in new clean energy generation alternatives.
Unfortunately, in the current administration's proposed budget, funding for ARPA-E is cancelled and allocated to clean coal; which Canada, France, and the UK are committed to phasing out entirely by ~2030.
A framework for evaluating data scientist competency
Something HTML with local state like "Programmer Competency Matrix" would be great.
http://sijinjoseph.com/programmer-competency-matrix/
Levi Strauss to use lasers instead of people to finish jeans
Chaos Engineering: the history, principles, and practice
awesome-chaos-engineering lists a bunch of chaos engineering resources and tools such as Gremlin: https://github.com/dastergon/awesome-chaos-engineering
Scientists use an atomic clock to measure the height of a mountain
Quantum_clock#More_accurate_experimental_clocks: https://en.wikipedia.org/wiki/Quantum_clock#More_accurate_ex...
> In 2015 JILA evaluated the absolute frequency uncertainty of their latest strontium-87 optical lattice clock at 2.1 × 10−18, which corresponds to a measurable gravitational time dilation for an elevation change of 2 cm (0.79 in) on planet Earth that according to JILA/NIST Fellow Jun Ye is "getting really close to being useful for relativistic geodesy".
AFAIU, this type of geodesy isn't possible with 'normal' time structs. Are nanoseconds enough?
"[Python-Dev] PEP 564: Add new time functions with nanosecond resolution" https://mail.python.org/pipermail/python-dev/2017-October/14...
The thread you linked makes a good point: there isn't really any reason to care about the actual time, only the relative time. These sorts of clocks just use 256- or 512-bit counters; it's not like they're having overflow issues.
Resources to learn project management best practices?
My side project is beginning to attract interest from a few people who would like to hop on board. At this point I am just doing what feels familiar and sensible, but the project manager perspective is new to me. Are there any sort of articles/books/podcasts/etc that could clue me into how to become better at it?
Project Management: https://wrdrd.github.io/docs/consulting/software-development... ... #requirements-traceability, #work-breakdown-structure (Mission, Project, Goal/Objective #n; Issue #n, - [ ] Task)
"Ask HN: How do you, as a developer, set measurable and actionable goals?" https://westurner.github.io/hnlog/#story-15119635
- Burndown Chart, User Stories
... GitHub and GitLab have milestones and reorderable issue boards. I still like https://waffle.io for complexity points; though you can also just create labels for e.g. complexity (Complexity-5) and priority (Priority-5).
Ask HN: Thoughts on a website-embeddable, credential validating service?
Reading Troy Hunt's password release V2 blog post [0], I came across the NIST recommendation to prevent users from creating accounts with passwords discovered in data breaches. This got me thinking: would a website admin (ex. small business owner with a custom website) benefit from a service that validates user passwords? The idea is to create a registration iframe with forms for email, password, etc., which would check hashed credentials against a database of data from breaches. Additionally, client-side validation would enforce rules recommended by the NIST's Digital Identity Guidelines [1], which would relieve admins from implementing their own rules. I'm sure there are additional security features that can be added.
1. Have you seen a need for this type of service, and could you see this being adopted at all?
2. Do you know of a service like this? I've looked, no hits so far.
3. Does the architecture seem sound?
[0]: https://www.troyhunt.com/ive-just-launched-pwned-passwords-version-2/
[1]: https://www.nist.gov/itl/tig/projects/special-publication-800-63
blockchain-certificates/cert-verifier-js: https://github.com/blockchain-certificates/cert-verifier-js
> A library to enable parsing and verifying a Blockcert. This can be used as a node package or in a browser. The browserified script is available as verifier.js.
https://github.com/blockchain-certificates/cert-issuer
> The cert-issuer project issues blockchain certificates by creating a transaction from the issuing institution to the recipient on the Bitcoin blockchain that includes the hash of the certificate itself.
... We could/should also store X.509 cert hashes in a blockchain.
Exactly what part of such a service would benefit from anything related to a blockchain?
Are you asking me why blockcerts stores certs in a blockchain?
Or whether using certs (really long passwords) is a better option than submitting unhashed passwords on a given datetime to a third-party in order to make sure they're not in the pwned passwords tables?
I was just reading about a company trying to make self-sovereign identity including actual certs (like degrees and such) an accessible and widely applicable/acceptable technology using Ethereum blockchain. I thought it showed some real practicality and promise. I believe it begins with U- forgot the name.Perhaps UPort? Anyhow, I'd be interested in hearing from anyone here about why that might be a bad or good idea. I don't personally have the skill in that tech to know.
Known Traveler Digital Identity system is a "new model for airport screening and security that uses biometrics, cryptography and distributed ledger technologies."
Blockcerts are for academic credentials, AFAIU.
[EDIT]
Existing blockchains have a limited TPS (transactions per second) for writes; but not for reads. Sharding and layer-2 (sidechains) do not have the same assurances. I'm sure we all remember how cryptokitties congested the txpool during the Bitcoin futures launch.
Thank you. I looked into "clear", one of the airport known traveler ID systems. (I'm assuming there are others) It's pretty cool/concerning. Takes ~ 8 min to load a traveler into its system. Thanks for reminding me of TPS---the info on read vs write.
Ask HN: What's the best algorithms and data structures online course?
These aren't courses, but from answers to "Ask HN: Recommended course/website/book to learn data structure and algorithms" :
Data Structure: https://en.wikipedia.org/wiki/Data_structure
Algorithm:https://en.wikipedia.org/wiki/Algorithm
Big O notation:https://en.wikipedia.org/wiki/Big_O_notation
Big-O Cheatsheet: http://bigocheatsheet.com
Coding Interview University > Data Structures: https://github.com/jwasham/coding-interview-university/blob/...
OSSU: Open Source Society University > Core CS > Core Theory > "Algorithms: Design and Analysis, Part I" [&2] https://github.com/ossu/computer-science/blob/master/README....
"Algorithms, 4th Edition" (2011; Sedgewick, Wayne): https://algs4.cs.princeton.edu/
Complexity Zoo > Petting Zoo (P, NP,): https://complexityzoo.uwaterloo.ca/Petting_Zoo
While perusing awesome-awesomeness [1], I found awesome-algorithms [2] , algovis [3], and awesome-big-o [4].
[1] https://github.com/bayandin/awesome-awesomeness
[2] https://github.com/tayllan/awesome-algorithms
Using Go as a scripting language in Linux
I, too, didn't realize that shebang parsing is implemented in the `binfmt_script` kernel module.
Does this persist across reboots?
echo ':golang:E::go::/usr/local/bin/gorun:OC' | sudo tee /proc/sys/fs/binfmt_misc/registerNo, but different init systems may autoload formats based on some configuration files. Systemd, for instance: https://www.freedesktop.org/software/systemd/man/systemd-bin...
Guidelines for enquiries regarding the regulatory framework for ICOs [pdf]
This is a helpful table indicating whether a Payment, Utility, Asset, or Hybrid coin/token: is a security, qualifies under Swiss AML payment law.
The "Minimum information requirements for ICO enquiries" appendix seems like a good set of questions for evaluating ICOs. Are there other good questions to ask when considering whether to invest in a Payment, Utility, Asset, or Hybrid ICO?
Are US regulations different from these clear and helpful regulatory guidelines for ICOs in Switzerland?
> Are there other good questions to ask when considering whether to invest in a Payment, Utility, Asset, or Hybrid ICO?
This paper doesn seem to cover that, only on how regulators should treat investments.
On investing in ICOs, the questions are the same as any other IPO. And, in most cases, just speculation.
The Benjamin Franklin method for learning more from programming books
> Read your programming book as normal. When you get to a code sample, read it over
> Then close the book.
> Then try to type it up.
According to a passage in "The Autobiography of Benjamin Franklin" (1791) regarding re-typing from "The Spectator"
https://en.wikipedia.org/wiki/The_Autobiography_of_Benjamin_...
Avoiding blackouts with 100% renewable energy
I notice that cases A and C require batteries for storage.
Should there be a separate entry for new gen supercapacitors? Supercapacitors built with both graphene and hemp have different Max Charge Rate (GW), Max Discharge Rate (GW), and Storage (TWh) capacities than even future-extrapolated batteries and current supercapacitors.
https://en.wikipedia.org/wiki/Supercapacitor
The cost and capabilities stats in this article look very promising:
"Hemp Carbon Makes Supercapacitors Superfast” https://www.asme.org/engineering-topics/articles/energy/hemp...
> “Our device’s electrochemical performance is on par with or better than graphene-based devices,” Mitlin says. “The key advantage is that our electrodes are made from biowaste using a simple process, and therefore, are much cheaper than graphene.”
> Graphene is, however, expensive to manufacture, costing as much as $2,000 per gram. [...] developed a process for converting fibrous hemp waste into a unique graphene-like nanomaterial that outperforms graphene. What’s more, it can be manufactured for less than $500 per ton.
> Hemp fiber waste was pressure-cooked (hydrothermal synthesis) at 180 °C for 24 hours. The resulting carbonized material was treated with potassium hydroxide and then heated to temperatures as high as 800 °C, resulting in the formation of uniquely structured nanosheets. Testing of this material revealed that it discharged 49 kW of power per kg of material—nearly triple what standard commercial electrodes supply, 17 kW/kg.
https://scholar.google.com/scholar?hl=en&q=hemp+supercapacit...
To be clear, supercapacitors are an alternative to li-ion batteries.
"Matching demand with supply at low cost in 139 countries among 20 world regions with 100% intermittent wind, water, and sunlight (WWS) for all purposes" (Renewable Energy, 2018) https://web.stanford.edu/group/efmh/jacobson/Articles/I/Comb...
Ask HN: What are some common abbreviations you use as a developer?
These are called 'codelabels'. They're great for prefix-tagging commit messages, pull requests, and todo lists:
BLD: build
BUG: bug
CLN: cleanup
DOC: documentation
ENH: enhancement
ETC: config
PRF: performance
REF: refactor
RLS: release
SEC: security
TST: test
UBY: usability
DAT: data
SCH: schema
REQ: requirement
REQ: request
ANN: announcement
STORY: user story
EPIC: grouping of user stories
There's a table of these codelabels here: https://wrdrd.github.io/docs/consulting/software-development...
Someday TODO FIXME XXX I'll get around to:
- [ ] DOC: create a separate site/organization for codelabels
- [ ] ENH: a tool for creating/renaming GitHub labels with unique foreground and background colors
YAGNI: Ya' ain't gonna need it
LOL, lulz
DRY: Don't Repeat Yourself
KISS: Keep It Super Simple
MVC: Model-View-Controller
MVT: Model-View-Template
MVVM: Model-View-View-Model
UI: User Interface
UX: User Experience
GUI: Graphical User Interface
CLI: Command Line Interface
CAP: Consistency, Availability, Partition tolerance
DHT: Distributed Hash Table
ETL: Extract, Transform, and Load
ESB: Enterprise Service Bus
MQ: Message Queue
VM: Virtual Machine
LXC: Linux Containers
[D]VCS, RCS: [Distributed] Version/Revision Control System
XP: Extreme Programming
CI: Continuous Integration
CD: Continuous Deployment
TDD: Test-Driven Development
BDD: Behavior-Driven Development
DFS, BFS: Depth/Breadth First Search
CRM: Customer Relationship Management
CMS: Content Management System
LMS: Learning Management System
ERP: Enterprise Resource Planning system
HTTP: Hypertext Transfer Protocol
HTTP STS: HTTP Strict Transport Security
REST: Representational State Transfer
API: Application Programming Interface
HTML: Hypertext Markup Language
DOM: Document Object Model
LD: Linked Data
LOD: Linked Open Data
URI: Uniform Resource Indicator
URN: Uniform Resource Name
URL: Uniform Resource Locator
UUID: Universally Unique Identifier
RDF: Resource Description Format
RDFS: RDF Schema
OWL: Web Ontology Language
JSON-LD: JSON Linked Data
JSON: JavaScript Object Notation
CSVW: CSV on the Web
CSV: Comma Separated Values
CIA: Confidentiality, Integrity, Availability
ACL: Access Control List
RBAC: Role-Based Access Control
MAC: Mandatory Access Control
CWE: Common Weakness Enumeration
CVE: Common Vulnerabilities and Exposures
XSS: Cross-Site Scripting
CSRF: Cross-Site Request Forgery
SQLi: SQL Injection
ORM: Object-Relational Model
AUC: Area Under Curve
ROC: Receiver Operating Characteristic
DL: Description Logic
RL: Reinforcement Learning
CNN: Convolutional Neural Network
DNN: Deep Neural Network
IS: Information Systems
ROI: Return on Investment
RPU: Revenue per User
MAU: Monthly Active Users
DAU: Daily Active Users
STEM: Science, Technology, Engineering, Mathematics/Medicine
STEAM: STEM + Arts
W3C: World-Wide-Web Consortium
GNU: GNU's not Unix
WRDRD: WRD R&D
... The Sphinx ``.. index::`` directive makes it easy to include index entries for acronym forms, too https://wrdrd.github.io/docs/genindex
There Might Be No Way to Live Comfortably Without Also Ruining the Planet
"A good life for all within planetary boundaries" (2018) https://www.nature.com/articles/s41893-018-0021-4
> Abstract: Humanity faces the challenge of how to achieve a high quality of life for over 7 billion people without destabilizing critical planetary processes. Using indicators designed to measure a ‘safe and just’ development space, we quantify the resource use associated with meeting basic human needs, and compare this to downscaled planetary boundaries for over 150 nations. We find that no country meets basic needs for its citizens at a globally sustainable level of resource use. Physical needs such as nutrition, sanitation, access to electricity and the elimination of extreme poverty could likely be met for all people without transgressing planetary boundaries. However, the universal achievement of more qualitative goals (for example, high life satisfaction) would require a level of resource use that is 2–6 times the sustainable level, based on current relationships. Strategies to improve physical and social provisioning systems, with a focus on sufficiency and equity, have the potential to move nations towards sustainability, but the challenge remains substantial.
> "Radical changes are needed if all people are to live well within the limits of the planet," [...]
> "These include moving beyond the pursuit of economic growth in wealthy nations, shifting rapidly from fossil fuels to renewable energy, and significantly reducing inequality.
> "Our physical infrastructure and the way we distribute resources are both part of what we call provisioning systems. If all people are to lead a good life within the planet's limits then these provisioning systems need to be fundamentally restructured to allow for basic needs to be met at a much lower level of resource use."
Perhaps ironically, our developments in service of sustainability (resource efficiency) needs for a civilization on Mars are directly relevant to solving these problems on Earth.
Recycle everything.
Survive without soil, steel, hydrocarbons, animals, oxygen.
Convert CO2, sunlight, H20, and geothermal energy to forms necessary for life.
https://en.wikipedia.org/wiki/Colonization_of_Mars
Algae, carbon capture, carbon sequestration, lab grown plants, water purification, solar power, [...]
Mars requires a geomagnetic field in order to sustain an atmosphere in order to [...].
"The Limits to Growth" (1972, 2004) [1] very clearly forecasts these same unsustainable patterns of resource consumption: 'needs' which exceed and transgress our planetary biophysical boundaries.
The 17 UN Sustainable Development Goals (#GlobalGoals) [2] outline our worthwhile international objectives (Goals, Targets, and Indicators). The Paris Agreement [3] sets targets and asks for commitments from nation states (and businesses) to help achieve these goals most efficiently and most sustainably.
In the US, the Clean Power Plan [4] was intended to redirect our national resources toward renewable energy with far less external costs. Direct and indirect subsidies for nonrenewables are irrational. Are subsidies helpful or necessary to reach production volumes of renewable energy products and services?
There are certainly financial incentives for anyone who chooses to invest in solving for the Global Goals; and everyone can!
[1] https://en.wikipedia.org/wiki/The_Limits_to_Growth
[2] http://www.un.org/sustainabledevelopment/sustainable-develop...
Multiple GWAS finds 187 intelligence genes and role for neurogenesis/myelination
> We found evidence that neurogenesis and myelination—as well as genes expressed in the synapse, and those involved in the regulation of the nervous system—may explain some of the biological differences in intelligence.
re: nurture, hippocampal plasticity and hippocampal neurogenesis also appear to be affected by dancing and omega-3,6 (which are transformed into endocannabinoids by the body): https://news.ycombinator.com/item?id=15109698
Could we solve blockchain scaling with terabyte-sized blocks?
These numbers in a computational model (or even Jupyter notebooks) would be useful.
We may indeed need fractional satoshis ('naks').
With terabyte blocks, lightning network would be unnecessary: at least for TPS.
There will need to be changes to account for quantum computing capabilities somewhere in the future timeline of Bitcoin (and everything else in banking and value-producing industry). Probably maybe a different hash function instead of just a routine difficulty increase (and definitely something other than ECDSA, which isn't a primary cost). $1.3m/400k a year to operate a terabyte mining rig with 50Gbps bandwidth would affect decentralization; though maybe not any more than it already is affected now.
https://en.bitcoin.it/wiki/Weaknesses#Attacker_has_a_lot_of_... (51%)
Confidence intervals for these numbers would be useful.
Casper PoS and beyond may also affect future Bitcoin volume estimates.
Ask HN: Do you have ADD/ADHD? How do you manage it?
Also, how has it affected your CS career? I feel that transitioning to management would help, as it does not require lengthy periods of concentration, but rather distributed attention for shorter periods.
Music. Headphones. Chillstep, progressive, chillout etc. from di.fm. Long mixes from SoundCloud with and without vocals. "Instrumental"
Breathe in through the nose and out through the mouth.
Less sugar and processed foods. Though everyone has a different resting glucose level.
Apparently it's called alpha-pinene.
Fidget things. Rubberband, paperclip.
The Pomodoro Technique: work 25 minutes, chill for 5 (and look at something at least 20 feet away (20-20-20 rule))
Lists. GTD. WBS.
Exercise. Short walks.
Ask HN: How to understand the large codebase of an open-source project?
Hello All!
what are techniques you all used to learn and understand a large codebase? what are the tools you use?
What is the best way to learn to code from absolute scratch?
We have been hosting a Ugandan refugee in our home in Oakland for the past 9 months and he wants to learn how to code.
Where is the best place for him to start from absolute scratch? What resources can we point him to? Who can help?
Here's an answer to a similar question: "Ask HN: How to introduce someone to programming concepts during 12-hour drive?" https://news.ycombinator.com/item?id=15454421
https://learnxinyminutes.com/docs/python3/ (Python3)
https://learnxinyminutes.com/docs/javascript/ (Javascript)
https://learnxinyminutes.com/docs/git/ (Git)
https://learnxinyminutes.com/docs/markdown/ (Markdown)
Read the docs. Read the source. Write docstrings. Write automated tests: that's the other half of the code.
Keep a journal of your knowledge as e.g. Markdown or ReStructuredText; regularly pull the good ones from bookmarks and history into an outline.
I keep a tools reference doc with links to Wikipedia, Homepage, Source, Docs: https://wrdrd.github.io/docs/tools/
And a single-page log of my comments: https://westurner.github.io/hnlog/
> To get a job, "Coding Interview University": https://github.com/jwasham/coding-interview-university
Wow, thank you so much. This is incredibly useful.
Tesla racing series: Electric cars get the green light – Roadshow
Tesla Racing Circuit ideas for increasing power discharge rate, reducing heat, and reducing build weight:
Hemp supercapacitors (similar power density as graphene supercapacitors and li-ion, lower cost than graphene)
Active cooling. Modified passive cooling.
Biocomposite frame and panels (stronger and lighter than steel and aluminum (George Washington Carver))
> Biocomposite frame and panels (stronger and lighter than steel and aluminum (George Washington Carver))
"Soybean Car" (1941) https://en.wikipedia.org/wiki/Soybean_car
What happens if you have too many jupyter notebooks?
These days there is a tendency in data analysis to use Jupyter Notebooks. But what happens if you have too many jupyter notebooks? For example, there are more than a hundred.
Actually, you start creating some modules. However, it is less convenient to work with them compared to what was before. It happens that you should code in web interface, somewhere in similar to the notepad++ form or you should change your IDLE.
Personally, I work in Pycharm and so far I couldn't assess remote interpreter or VCS. It is because pickle files or word2vec weighs too much (3gb+) and so I don't want to download/upload them. Also Jupyter is't cool in pycharm.
Do you have better practices in your companies? How to correctly adjust IDLE? Do you know about any possible substitution for the IPython notebook in the world of data analysis?
> what happens if you have too many jupyter notebooks? For example, there are more than a hundred.
Like anything else, Jupyter Notebook is limited by the CPU and RAM of the system hosting the Tornado server and Jupyter kernels.
At 100 notebooks (or even just one), it may be a good time to factor common routines into a packaged module with tests and documentation.
It's actually possible (though inefficient) to import code from Jupyter notebooks with ipython/ipynb (pypi:ipynb): https://github.com/ipython/ipynb ( https://jupyter-notebook.readthedocs.io/en/stable/examples/N... )
> Actually, you start creating some modules. However, it is less convenient to work with them compared to what was before. It happens that you should code in web interface, somewhere in similar to the notepad++ form or you should change your IDLE.
The Spyder IDE has support for .ipynb notebooks converted to .py (which have the IPython prompt markers in them). Spyder can connect an interpreter prompt to a running IPython/Jupyter kennel. There's also a Spyder plugin for Jupyter Notebook: https://github.com/spyder-ide/spyder-notebook
> Personally, I work in Pycharm and so far I couldn't assess remote interpreter or VCS. It is because pickle files or word2vec weighs too much (3gb+) and so I don't want to download/upload them.
Remote data access times can be made faster by increasing the space efficiency of the storage format, increasing the bandwidth of the connection, moving the data to the code, or moving the code to the data.
> Do you have better practices in your companies?
There are a number of [Reproducible] Data Science cookiecutter templates which have a directory for notebooks, module packaging, and Sphinx docs: https://cookiecutter.readthedocs.io/en/latest/readme.html#da...
Refactoring increases testability and code reuse.
> How to correctly adjust IDLE?
I don't think I understand the question?
"Configuring IPython" https://ipython.readthedocs.io/en/stable/config/index.html
Jupyter > "Installation, Configuration, and Usage" https://jupyter.readthedocs.io/en/latest/projects/content-pr...
> Do you know about any possible substitution for the IPython notebook in the world of data analysis?
From https://en.wikipedia.org/wiki/Notebook_interface :
> > "Examples of the notebook interface include the Mathematica notebook, Maple worksheet, MATLAB notebook, IPython/Jupyter, R Markdown, Apache Zeppelin, Apache Spark Notebook, and the Databricks cloud."
There are lots of Jupyter kernels for different tools and languages (over 100; including for other 'notebook interfaces'): https://github.com/jupyter/jupyter/wiki/Jupyter-kernels
And there are lots of Jupyter integrations and extensions: https://github.com/quobit/awesome-python-in-education/blob/m...
Cancer ‘vaccine’ eliminates tumors in mice
The article is about this study:
"Eradication of spontaneous malignancy by local immunotherapy" http://stm.sciencemag.org/content/10/426/eaan4488
> In situ vaccination with low doses of TLR ligands and anti-OX40 antibodies can cure widespread cancers in preclinical models.
Boosting teeth’s healing ability by mobilizing stem cells in dental pulp
Tideglusib
https://en.wikipedia.org/wiki/Tideglusib
> "Promotion of natural tooth repair by small molecule GSK3 antagonists" https://www.nature.com/articles/srep39654
> [...] Here we describe a novel, biological approach to dentine restoration that stimulates the natural formation of reparative dentine via the mobilisation of resident stem cells in the tooth pulp.
This Biodegradable Paper Donut Could Let Us Reforest the Planet
"These drones can plant 100,000 trees a day" https://news.ycombinator.com/item?id=16260892
Drones that can plant 100k trees a day
> It’s simple maths. We are chopping down about 15 billion trees a year and planting about 9 billion. So there’s a net loss of 6 billion trees a year.
This is a regional thing [0] though. We need to plant the trees in Latin America, Caribbean, and Sub-Saharan Africa. The rest of the world is gaining forests.
[0] http://www.telegraph.co.uk/news/2016/03/23/deforestation-whe...
Now, the question is, which industry sector cuts the largest percentage of trees. Now, answering that question, is there a way to do the same thing without cutting trees?
Answering that question and then executing a business plan is probably worth billions.
Planting trees to combat 6 billion trees lost every year is a pure expense, no profit to be made at all. At least not if you won't cut them down a few decades later.
A very good way to absorb atmospheric carbon is to plant new trees and cut down and use old ones for anything else but burning it.
"This Biodegradable Paper Donut Could Let Us Reforest The Planet" https://news.ycombinator.com/item?id=16261101
I think this kind of thing is funny for us westerners. I recently found this tech has been used in arid countries for possibly hundreds or thousands of years.
https://duckduckgo.com/?q=ollas+gardening&t=lm&iax=images&ia...
I do realize the ollas are for more permanent gardens, but it's the same concept. I do hope the paper version gets used, there are plenty of places that could benefit from it.
What are some YouTube channels to progress into advanced levels of programming?
There are some cool YouTube channel suggestions on https://news.ycombinator.com/item?id=16224165 But I wanted to know which of those are great to progress into advanced level of programming? Which of the channels teach advanced techniques?
These aren't channels, but a number of the links are marked with "(video)":
https://github.com/jwasham/coding-interview-university
https://github.com/jwasham/coding-interview-university/blob/...
Thanks
Multiple issue and pull request templates
+1
Default: /ISSUE_TEMPLATE.md
/ISSUE_TEMPLATE/<name>.md</name>
Default: /PULL_REQUEST_TEMPLATE.md
/PULL_REQUEST_TEMPLATE/<name>.md</name>
You can leave the last S for off (for savings) if you want /ISSUE_TEMPLATE(S)/ or /PULL_REQUEST_TEMPLATE(S)/
Five myths about Bitcoin’s energy use
Regarding the proof-of-stake part: Ethereum devs are also working on sharding which will make scaling on the Ethereum Blockchain way easier. I really think that Ethereum will be the No.1 cryptocurrency in the near future. Bitcoin devs showed plenty of times, that they are not capable of keeping pace with demand (couldn't even get the block size thing right). Bitcoin is virtually dead. No one can really use it for real world transactions. Plus Bitcoin in reality is a really central coin, completely in the hands of the miners. Even if the Bitcoin devs decided to go with PoS, the miners wouldn't agree.
Proof of Work (Bitcoin*, ...), Proof of Stake (Ethereum Casper), Proof of Space, Proof of Research (GridCoin, CureCoin,)
Plasma (Ethereum) and Lightning Network (BitCoin (SHA256), Litecoin (scrypt),) will likely offload a significant amount of transaction volume and thereby reduce the kWh/transaction metrics.
> But electricity costs matter even more to a Bitcoin miner than typical heavy industry. Electricity costs can be 30-70% of their total costs of operation.
> [...] If Bitcoin mining really does begin to consume vast quantities of the global electricity supply it will, it follows, spur massive growth in efficient electricity production—i.e. the green energy revolution. Moore’s Law was partially a story about incredible advances in materials science, but it was also a story about incredible demand for computing that drove those advances and made semiconductor research and development profitable. If you want to see a Moore’s-Law-like revolution in energy, then you should be rooting for, and not against, Bitcoin. The fact is that the Bitcoin network, right now, is providing a $200,000 bounty every 10 minutes (the mining reward) to the person who can find the cheapest energy on the planet.
This is ridiculous. The economy is already incentivized to find cheaper electricity by forces far more powerful than Bitcoin. By this logic, you would defend a craze for trying to boil the sea.
If the market had internalized the external health, environmental, and defense costs of nonrenewable energy, we would already have cheap, plentiful renewable energy. But we don't: the market is failing to optimize for factors other than margin. (New Keynesian economics admits market failure, but not non-rationality.)
So, (speculative_valuation - cost) is the margin. Whereas with a stock in a leveraged high-frequency market with shorting, (shareholder_equity - market_cap) is explainable in terms of the market information that is shared.
So, it's actually (~$200K-(n_kwhrs*cost_kwhr)) for whoever wins the block mining lottery (which is about every 10 minutes and can be anyone who's mining).
But the point about Bitcoin maintaining demand for and while we move to competitive lower cost renewable energy and greater efficiency is good.
What we should hope to see is the blockchain industry directly investing in clean energy capacity development in order to rationally minimize their primary costs and maximize environmental sustainability.
It is kind of missing the point though. If everyone was competing to make electrical single-person planes to help people commute to work it would also increase demand for renewable/electrical energy - but can we agree, that just using electrical cars instead of planes is going to take a lot less electricity overall?
Yes, and then energy prices would decrease due to less demand. Blockchain energy usage maintains demand for energy; which keeps prices high enough that production of renewables can profitably compete with nonrenewables while we reach production volumes of solar, wind, and hemp supercapacitors for grid storage.
> Throughout the first half of 2008, oil regularly reached record high prices.[2][3][4][5] Prices on June 27, 2008, touched $141.71/barrel, for August delivery in the New York Mercantile Exchange [...] The highest recorded price per barrel maximum of $147.02 was reached on July 11, 2008.
At that price, there's more demand for renewables (such as electric vehicles and solar panels)
> Since late 2013 the oil price has fallen below the $100 mark, plummeting below the $50 mark one year later.
https://en.wikipedia.org/wiki/World_oil_market_chronology_fr...
... Energy costs and inflation are highly covariate. (Trouble is, CPI All rarely ever goes back down)
Bitcoin's energy use will tend to raise to match the mining profits. As long as both BTC's price and transaction fees keep raising, the energy miners spend will continue to raise. BTC's price is going up due to speculation, and transaction fees due to blocks being full.
Even at current levels, BTC's built in block rewards dominate tx fees (12.50BTC built in + 1.38 tx fees). The 12.5 built in reward is essentially a subsidy for mining.
Ask HN: Which programming language has the best documentation?
Ask HN: Recommended course/website/book to learn data structure and algorithms
I am a full-time Android developer who does most of his programming work in Java. I am a non CS graduate so didn't study Data structure and algorithms course in university so I am not familiar with this subject which is hindering my prospect of getting better programming jobs. There are so many resources out there on this subject that I am unable to decide which one is the best for my case. Could someone please point me out in the right direction. Thanks.
Data Structure: https://en.wikipedia.org/wiki/Data_structure
Algorithm: https://en.wikipedia.org/wiki/Algorithm
Big O notation: https://en.wikipedia.org/wiki/Big_O_notation
Big-O Cheatsheet: http://bigocheatsheet.com
Coding Interview University > Data Structures: https://github.com/jwasham/coding-interview-university/blob/...
OSSU: Open Source Society University > Core CS > Core Theory > "Algorithms: Design and Analysis, Part I" [&2] https://github.com/ossu/computer-science/blob/master/README....
"Algorithms, 4th Edition" (2011; Sedgewick, Wayne): https://algs4.cs.princeton.edu/
Complexity Zoo > Petting Zoo (P, NP,): https://complexityzoo.uwaterloo.ca/Petting_Zoo
While perusing awesome-awesomeness [1], I found awesome-algorithms [2] , algovis [3], and awesome-big-o [4].
[1] https://github.com/bayandin/awesome-awesomeness
[2] https://github.com/tayllan/awesome-algorithms
Why is quicksort better than other sorting algorithms in practice?
The top-voted response is helpful: throwing away constants as in Big-O notation is misleading, average cases aren't the case; Sedgewick Algorthms book.
ORDO: a modern alternative to X.509
There are a number of W3C specs for this type of thing.
Linked Data Signatures (ld-signatures) relies upon a graph canonicalization algorithm that works with any RDF format (RDF/XML, JSON-LD, Turtle,)
> The signature mechanism can be used across a variety of RDF data syntaxes such as JSON-LD, N-Quads, and TURTLE, without the need to regenerate the signature
https://w3c-dvcg.github.io/ld-signatures/
A defined way to transform ORDO to RDF would be useful for WoT graph applications.
WebID can express X509 certs with the cert ontology. {cert:X509Certificate, cert:PGPCertificate,} rdfs:subClassOf cert:Certificate
https://www.w3.org/ns/auth/cert
https://www.w3.org/2005/Incubator/webid/spec/
ld-signatures is newer than WebID.
(Also, we should put certificates in a blockchain; just like Blockcerts (JSON-LD))
Wine 3.0 Released
Kimbal Musk is leading a $25M mission to fix food in US schools
Spinzero – A Minimal Jupyter Notebook Theme
+1. The Computer Modern serif fonts look legit. Like LaTeX legit.
Now, if we could make the fonts unscalable and put things in two columns (in order to require extra scrolling and 36 character wide almost-compiling copy-and-pasted code samples without syntax highlighting) we'd be almost there!
I searched for Computer Modern fonts and they're all available here: http://canopus.iacp.dvo.ru/~panov/cm-unicode/
I am surprised why these beauties are not widely adopted on websites and such. I agree, they just look very disciplined and professional.
I'd hope someday these relics are hosted on Google Fonts.
What does the publishing industry bring to the Web?
Q: What does the publishing industry bring to the Web?
A: PDF hosting, comments, a community of experts
FWIU, Publishing@W3C proposes WPUB [1] instead of PDF or MHTML for 'publishing' http://schema.org/ScholarlyArticle .
How do WPUB canonical identifiers (which reference/redirect(?) to the latest version of the resource) work with W3C Web Annotations attached to e.g. sentences within a resource identified with a URI? When the document changes, what happens to the attached comments? This is also a problem with PDFs: with a filename like document-20180111-v01.pdf and a stable(!) URL like http://example.org/document-20180111-v01.pdf, we can add Web Annotations to that URI; but with a new URI, those annotations are lost.
Git is a blockchain
Bitcoin is very much inspired by git; though in terms of immutability it's more similar to mercurial and subversion (git push -f)
Git accepts whatever timestamp a node chooses to add to a commit. This can cause interesting sorts in terms of chronological and topological sort orders.
Without an agreed-upon central git server there is not a canonical graph.
You can use GPG signatures with Git, but you need to provide your own keyserver and then there's still no way to enforce permissions (e.g. who can ALTER, UPDATE, or DELETE which files).
Git is a directed acyclic graph (DAG). Not a chain. Blockchains are chains to prevent double-spending (e.g. on a different fork).
Bitcoin was accepted by The Linux Foundation (Linus Torvalds wrote Git): https://lists.linuxfoundation.org/pipermail/bitcoin-dev/
It's always been a mix of inspiration in my mind:
- Torrent P2P file sharing.
- Git like data structure and protocol
- Immutability from functional programming
- Public key cryptography
Show HN: Convert Matlab/NumPy matrices to LaTeX tables
LaTeX must be escaped in order to prevent LaTeX injection.
AFAIU, numpy.savetxt does not escape LaTeX characters?
Jupyter Notebook rich object display protocol checks for obj._repr_latex_() when converting a Jupyter notebook from .ipynb to LaTeX.
The Pandas _repr_latex_() function calls to_latex(escape=True † ). https://github.com/pandas-dev/pandas/blob/master/pandas/core...
†* The default value of escape ️ (and a few other presentational parameters) is determined from the display.latex.escape option: https://pandas.pydata.org/pandas-docs/stable/options.html?hi... *
df = pd.read_csv('filename.csv', ); df.to_latex(escape=True)
Or, with a Jupyter notebook:
df = pd.read_csv('filename.csv', ); df
# $ jupyter convert --to latex filename.ipynb
Wouldn't it be great if there was a LaTeX incantation that allowed for specifying that the referenced dataset URI (maybe optionally displayed also as a table) is a premise of the analysis; with RDFa and/ or JSONLD in addition to LaTeX PDF? That way, an automated analysis tool could identify and at least retrieve the data for rigorous unbiased analyses.
http://schema.org/ScholarlyArticle
#StructuredPremises
A Year of Spaced Repetition Software in the Classroom
What a great article about using Anki during class in a Language Arts curriculum.
NIST Post-Quantum Cryptography Round 1 Submissions
Are there any blogs that talk about what’s required in a crypto algorithm to withstand quantum computing? Is a straightforward increasing the key size enough? Or are there new paradigms explored? Are these algorithms symmetric or asymmetric?
This paper lists a few of the practical concerns for quantum-resistant algos (and proposes an algo that wasn't submitted to NIST Post-Quantum Cryptography Round 1):
"Quantum attacks on Bitcoin, and how to protect against them" https://arxiv.org/abs/1710.10377 (~2027?)
A few Quantum Computing and Quantum Algorithm resources: https://news.ycombinator.com/item?id=16052193
Responsive HTML (arxiv-vanity/engrafo, PLoS,) or Markdown in a Jupyter notebook (stored in a Git repo with a tag and maybe a DOI from figshare or Zenodo) really would be far more useful than comparing LaTeX equations rendered into PDFs.
What are some good resources to learn about Quantum Computing?
Quantum computing: https://en.wikipedia.org/wiki/Quantum_computing
Quantum algorithm: https://en.wikipedia.org/wiki/Quantum_algorithm
Quantum Algorithm Zoo: http://math.nist.gov/quantum/zoo/
Jupyter notebooks:
* QISKit/qiskit-tutorial > "Exploring Quantum Information Concepts" https://nbviewer.jupyter.org/github/QISKit/qiskit-tutorial/b...
* jrjohansson/qutip-lectures > "Lecture 0 - Introduction to QuTiP - The Quantum Toolbox in Python" https://nbviewer.jupyter.org/github/jrjohansson/qutip-lectur...
http://qutip.org/tutorials.html
* sympy/quantum_notebooks https://nbviewer.jupyter.org/github/sympy/quantum_notebooks/...
https://github.com/topics/quantum
krishnakumarsekar/awesome-quantum-machine-learning: https://github.com/krishnakumarsekar/awesome-quantum-machine...
arxiv quant-ph: https://arxiv.org/list/quant-ph/recent
Gridcoin: Rewarding Scientific Distributed Computing
In all honesty this is what I've been waiting for in terms of a useful cryptocurrency. Now if we could only decentralize the control of what projects the processing goes towards with smart contracts then we could have a coin with more actual utility. Imagine the hash rate of the BTC network going towards some useful calculations.
> Imagine the hash rate of the BTC network going towards some useful calculations.
""" CureCoin Reaches #1 Ranking on Folding@home
As of the afternoon of August 29, 2017 (Eastern Time), the Curecoin Team 224497 earned the world's #1 rank on Stanford's Folding@home - a protein folding simulation Distributed Computing Network (DCN). In a little over 3 years, the team (including our merge-folding partners at Foldingcoin) collectively produced 160 billion points worth of molecular computations to support research in the areas of cancer, Alzheimer's, Huntington's, Parkinson's, Infectious Disease as well as helping scientists uncover new molecular dynamics through groundbreaking computational techniques. """
> Imagine the hash rate of the BTC network going towards some useful calculations.
Unfortunatly BTC mining now runs almost entirely on ASICs that can't be used to compute anything but SHA-256.
Imagine and equivalent amount of computer power in an alternative future. I mean, you're right, and this limitation is a flaw in bitcoin, but you've also missed the point.
It's also hard to call it a flaw in Bitcoin, considering it was an intentional design decision.
Even intentional design decisions can be flawed.
There's a pretty hard limit bounding the optimizability of SHA256. That's why hashcash uses a cryptographic hash function.
There may be - or, very likely are - shortcuts for proof of research better than Grover's; which, when found, will also be very useful for science and medicine. However, that advantage is theoretically destabilizing for a distributed consensus network; which is also a strange conflict in incentives.
Sort of like buying "buy gold" commercials when the market was heading into the worst recession since the Great Depression.
SSL accelerators may benefit from the SHA256 ASIC optimizations incentivized by the bitcoin design.
"""The accelerator provides the RSA public-key algorithm, several widely used symmetric-key algorithms, cryptographic hash functions, and a cryptographically secure pseudo-random number generator"""
GPU prices are also lower now; probably due to demand pulling volume. The TPS (transactions per second) rate is doing much better these days.
How would you solve the local daretime problem in order with Git and signatures?
Power Prices Go Negative in Germany
The article doesn’t actually answer the questions it purports to answer. Better source: https://www.cleanenergywire.org/factsheets/why-power-prices-....
Energy markets are artificial markets designed to create various price signals that result in certain incentives on both generation and demand, subject to numerous constraints. One constraint is that demand and supply must balance. The grid can’t store much energy. Oversupply can cause grid frequency to go above 50/60 Hz, threatening grid stability: https://www.e-education.psu.edu/ebf483/node/705. Power prices go negative when there is too much generation capacity online at a given instant, relative to demand. That creates incentives for generators that can shut down (like natural gas) to do so.
Negative power prices are not a good thing for consumers. A negative price in the wholesale electric markets does not mean the electricity is "less than free." Obviously, even wind power or solar always costs positive money to generate in real terms. Instead, it signals a mismatch between generation capacity, storage capacity, and demand. In a grid with adequate storage capacity, negative prices would be extremely rare.
I think you're missing the critical piece of your explanation of why negative prices aren't good for consumers:
The reason that negative power prices look like they would be good for consumers is that it seems like they should lower their monthly bills. But in practice their bill should stay the same, even if there were significant periods during which energy prices were negative. Why? The negative prices don't indicate that the cost of power production is negative for the power company, so the power company is losing money. The power company has to recoup those losses somehow, and they do it by charging more when power prices are positive.
In order to take advantage of negative power prices, a power consumer would have to dynamically increase their power consumption in response to the negative prices. If they have any significant power storage capacity, maybe they could store power and sell it back to the grid when prices go positive, or maybe turn on their mining rig while prices are negative, if they go negative frequently.
I'll just add in some empirical results. German power has risen to be some of the most expensive in Europe since they started the Energiewende. They have very low wholesale prices (driving the coal producers out of business) combined with very high retail prices (might be top 3 in Europe?).
The high retail price appears to be driven by the mechanisms driving the Energiewende ie transition to wind and solar.
https://www.cleanenergywire.org/factsheets/what-german-house...
https://www.ovoenergy.com/guides/energy-guides/average-elect...
The NYT article reads as though Germany is benefiting from the switch to renewables, and that the only problems are that sometimes there's so much power around that they have to pay people to use it.
It mentions high cost of electricity, and that it's due to fees and "renewable investment costs" but then immediately hand-waves that away because "household energy bills have been rising over all anyway."
When combined with your information above (those hand-wavy fees actually account for fully 50% of the costs, with 24% being the renewables surcharge) it would seem that the NYT is being misleading. It seems they want to convey the idea that the renewables are an immediate good thing for everyone (which I do not take issue with politically) while downplaying the significant costs to consumers. Am I missing something?
If I'm not, then this does nothing but contribute to the current view of American media as being intentionally misleading when it suits their interests.
No, I don't think you're missing something here.
The price for electricity is indeed very high in Germany, rising to new heights with the new year, again.
The problem is, unsurprisingly, regulation and policy, i.e. the fees you already mentioned, and the subsidies for industrial usage, etc.
So, yes, it actually seems that the NYT is misleading here.
Let's leave this as an exercise for the reader, to judge if anyone should really be surprised here.
I always think of this old "Trust, but verify" quibble, and that it is actually based on an old Russian proverb. The irony.. :)
> The price for electricity is indeed very high in Germany
It's not only very high, but the second highest in the world, and we are about to take over the first spot [1]
> The problem is, unsurprisingly, regulation and policy, i.e. the fees you already mentioned, and the subsidies for industrial usage, etc.
The problem are the subsidies for all the green energy. It's not only the direct costs but also the costs for grid interventions (turning on/off capacity), paying for renewables even if they are not producing because there is too much energy available, the huge requirements for new energy lanes from north to south, getting rid of nuclear power etc.
Subsidies for industrial usage is often quoted as increasing the costs by interested parties, but it's also a direct need of the Energiewende, because the economic damage would be gigantic. Unless you want to get the energy intensive factories out of your country, you have to factor in those costs.
> So, yes, it actually seems that the NYT is misleading here.
Yes, as does most media - especially in Germany. Those negative prices are no win for any german. Why else is it, that we will soon pay the highest prices for electricity in the world?
[1] http://www.sueddeutsche.de/wirtschaft/energiekosten-in-oecd-...
"Several countries in Europe have experienced negative power prices, including Belgium, Britain, France, the Netherlands and Switzerland."
> Yes, as does most media - especially in Germany. Those negative prices are no win for any german. Why else is it, that we will soon pay the highest prices for electricity in the world?
AFAIU, it's because you're aggressively shaping the energy market in order to reduce health and environmental costs now.
The technical issue here is that batteries are not good enough yet; and [hemp] supercapacitors are not yet at the volume needed to lower the costs. So, maintaining a high price for energy keeps the market competitive for renewables which have positive negative externalities.
Bitcoin is an energy arbitrage
In addition to relocating to where energy is the least expensive, Bitcoin creates incentive for miners to lower the local cost of energy: invest in renewable energy.
Renewable Energy / Clean Energy is now less expensive than alternatives; with continued demand, the margins are at least maintained.
> In addition to relocating to where energy is the least expensive, Bitcoin creates incentive for miners to lower the local cost of energy: invest in renewable energy.
We have lots of direct and effective subsides for nonrenewable energy in the United States. And some for renewables, as well. For example [1] average effective tax rate over all money making companies: 26%
"Coal & Related Energy": 0.69%
"Oil/Gas (integrated)": 8.01%
"Power": 29.22%
"Green and Renewable Energy": 26.42%
[1] "Tax Rates by Sector (US)" (January 2017) http://pages.stern.nyu.edu/~adamodar/New_Home_Page/datafile/...
X-posting here from the article's comments:
The price reflects the confidence investors have in the security's ability to meet or exceed inflation and in the information security of the network.
Volatility adds value for algo traders: say the prices are [1, 101, 51, 101, 51, 201]:
(101-1)+(101-51)+(201-51)=300
(201-1)=200
For the average Joe looking at the vested options they're hodling, though, volatility is unfriendly.
When e.g. algo-traders are willing to buy in when the price starts to fall, they're making liquidity; which some exchanges charge less for.
Enigma Catalyst (Zipline) is one way to backtest and live-trade cryptocurrencies algorithmically.
There are now more than 200k pending Bitcoin transactions
At 20 transactions per second it's a delay of 3 hours. (200000/20/60/60)
"The bitcoin network's theoretical maximum capacity sits between 3.3 to 7 transactions per second."
The OT link does say "Transactions Per Second 22.54".
The solutions for this 3 hour backlog of unconfirmed transactions include: implementing SegWit, increasing the blocksize, and Lightning Network.
I think the link counts incoming transactions. I.e. you can always schedule more, but it doesn't mean they're going to be acted on in a reasonable timeframe.
What ORMs have taught me: just learn SQL (2014)
ORMs:
- Are maintainable by a team. "Oh, because that seemed faster at the time."
- Are unit tested: eventually we end up creating at least structs or objects anyway, and then that needs to be the same everywhere, and then the abstraction is wrong because "everything should just be functional like SQL" until we need to decide what you called "the_initializer2".
- Can make it very easy to create maintainable test fixtures which raise exceptions when the schema has changed but the test data hasn't.
- Prevent SQL injection errors by consistently parametrizing queries and appropriately quoting for the target SQL dialect. (One of the Top 25 most frequent vulnerabilities). This is especially important because most apps GRANT both UPDATE and DELETE; if not CREATE TABLE and DROP TABLE to the sole app account.
- Make it much easier to port to a new database; or run tests with SQLite. With raw SQL, you need the table schema in your head and either comprehensive test coverage or to review every single query (and the whole function preceding db.execute(str, *params))
- May be the performance bottleneck for certain queries; which you can identify with code profiling and selectively rewrite by hand if adding an index and hinting a join or lazifying a relation aren't feasible with the non-SQLAlchemy ORM that you must use.
- Should provide a way to generate the query at dev or compile-time.
- Should make it easy to DESCRIBE the query plans that code profiling indicates are worth hand-optimizing (learning SQL is sometimes not the same as learning how a particular database plans a query over tables without indexes)
- Make managing db migrations pretty easy.
- SQLAlchemy really is great. SQLAlchemy has eager loading to solve the N+1 query problem. Django is often more than adequate; and has had prefetch_related() to solve the N+1 query problem since 1.4. Both have an easy way to execute raw queries (that all need to be reviewed for migrations). Both are much better at paging without allocating a ton of RAM for objects and object attributes that are irrelevant now.
- Make denormalizing things from a transactional database with referential integrity into JSON really easy; which webapps and APIs very often need to do.
Is there a good JS ORM? Maybe in TypeScript?
I've used http://bookshelfjs.org/ but I'd stay away from it, only felt cumbersome. No productivity gain.
It's built on top a query builder Knex (http://knexjs.org/) which is decent.
Objection.js and the Knex query builder are excellent. Think ORM light with full access to SQL.
Show HN: An educational blockchain implementation in Python
> It is NOT secure neither a real blockchain and you should NOT use this for anything else than educational purposes.
It would be nice if non-secure parts of implementation or design were clearly marked.
What's the point of education article, if bad examples aren't clearly marked as bad? If MD5 usage is the only issue, author could easily replace it with SHA and get rid of the warning at the start. If there are other issues, how can a reader know which parts to trust?
Even if fixing bad/insecure parts are "left as an exercise for the reader", learning value of the article would be much greater if those parts would be at least pointed at.
OP here.
erikb is spot on in the sibling comment. This hasn't been expert-reviewed, hasn't been audited so I'm pretty confident there is a bug somewhere that I don't know about.
It's educational in the sense that I tried as best a I could to implement the various algorithmic parts (mining, validating blocks & transactions, etc...).
I originally used MD5 because I thought I would do more exploration regarding difficulty and MD5 is faster to compute than SHA. In the end, I didn't do that exploration, so I could easily replace MD5 with SHA. I'll update the notebook to use SHA, but I'm still not gonna remove the warning :)
I'll also try to point out more explicitly which parts I think are not secure.
> I'll also try to point out more explicitly which parts I think are not secure.
Things I've noticed:
* Use of floating point arithmetic.
* Non-reproducible serialization in verify_transaction can produce slightly different, but equivalent JSON, which leads to rejecting transactions if produced JSON is platform-dependent (e.g. CRLFs, spaces vs tabs).
* Miners can perform DoS by creating a pair of blocks referencing each other (recursive call in verify_block is made before any sanity checks or hash checks, so they can modify block's ancestor without worrying about changing its hash).
* mine method can loop forever due to integer overflow.
* Miners can put in block a transaction with output sum greater than input sum - only place where it is checked is in compute_fee and no path from verify_block leads there.
Those are all very good points I didn't think about, thanks for these.
I'll fix the two bugs with verify_block and the possibility for a miner to inject invalid a output > input transaction.
I'll add a note for the 3 others.
For deterministic serialization (~canonicalization), you can use sort_keys=True or serialize OrderedDicts. For deseialization, you'd need object_pairs_hook=collections.OrderedDict.
Most current blockchains sign a binary representation with fixed length fields. In terms of JSON, JSON-LD is for graphs and it can be canonicalized. Blockcerts and Chainpoint are JSON-LD specs:
> Blockcerts uses the Verifiable Claims MerkleProof2017 signature format, which is based on Chainpoint 2.0.
https://github.com/blockchain-certificates/cert-verifier-js/...
FYI, dicts are now ordered by default as of Python 3.6.
That's an implementation detail, and shouldn't be relied upon. If you want an ordered dictionary, you should use collections.OrderedDict.
It's now the spec for 3.6+.
> #python news: @gvanrossum just pronounced that dicts are now guaranteed to retain insertion order. This is the end of a long journey.
https://twitter.com/raymondh/status/941709626545864704
More here: https://www.reddit.com/r/Python/comments/7jyluw/dict_knownor...
OrderedDicts are backwards-compatible and are guaranteed to maintain order after deletion.
Thanks! Simplest explanation I've seen.
Here's an nbviewer link (which, like base58, works on/over a phone): https://nbviewer.jupyter.org/github/julienr/ipynb_playground...
Note that Bitcoin does two rounds of SHA256 rather than one round of MD5. There's also a "P2P DHT" (peer-to-peer distributed hash table) for storing and retrieving blocks from the blockchain; instead of traditional database multi-master replication and secured offline backups.
> ERROR:root:Invalid transaction signature, trying to spend someone else's money ?
This could be more specific. Where would these types of error messages log to?
My mistake, it's BitTorrent that has a DHT. Instead of finding the most network local peer with the block identified by a (prev_hash, hash) hash table key, the Bitcoin blockchain broadcasts all messages to all nodes; which must each maintain a complete backup of the entire blockchain.
"Protocol documentation" https://en.bitcoin.it/wiki/Protocol_documentation
MSU Scholars Find $21T in Unauthorized Government Spending
Unauthorized federal spending (in these two departments) 1998-2015: $21T
Federal debt (2017): $20T
$ 20,000,000,000,000 USD
Would a blockchain for government expenditures help avoid this type of error?
We already now have https://usaspending.gov ( https://beta.usaspending.gov ) and expenditure line item metadata.
Would having traceable money in a distributed ledger help us keep track of money collected from taxpayers?
Obviously, the volatility of most cryptocurrencies would be disadvantageous for purposes of transferring and accounting for government spending. Isn't there a way to peg a cryptocurrency to the USD; even with Quantitative Easing? How is Quantitative Easing different from just deciding to print trillions more 'coins' in order to counter debt or inflation or deflation; why is the government in debt at all?
re: Quantitative Easing
https://en.wikipedia.org/wiki/Quantitative_easing
Say I have $100 in my Social Security Fund (in very non-aggressive investments which need to meet or exceed inflation) and the total supply of money (including paper notes and numbers in debit and credit columns of various public and private databases) the total supply of money is $1T with $1T in debt; if 1T is printed to pay for that debt, is my $100 in retirement savings then worth $50? Or is it more complex than that?
[deleted]
Universities spend millions on accessing results of publicly funded research
Are there good open source solutions for journal publishing? (HTML abstract, PDFs, comments, ...)?
Yes- quite a few beyond what's already been listed, in fact:
http://www.theoj.org/ https://plos.github.io/ambraproject/index.html https://v4.pubpub.org/
Ambra is being discontinued! http://blogs.plos.org/plos/2017/12/ceo-letter-to-the-communi...
Edit: And theoj doesn't really appear to be maintained anymore either...
> Ambra is being discontinued!
The article mentions the discontinuation of Aperta but nothing about Ambra?
https://plos.github.io/ambraproject/Developer-Overview.html
I'm sorry, I just realised that as well - my mistake.
An Interactive Introduction to Quantum Computing
Part 2 mentions two quantum algorithms that could be used to break Bitcoin (and SSH and SSL/TLS; and most modern cryptographic security systems): Shor's algorithm for factorization and Grover's search algorithm.
Part 2: http://davidbkemp.github.io/QuantumComputingArticle/part2.ht...
Shor's algorithm: https://en.wikipedia.org/wiki/Shor%27s_algorithm
Grover's algorithm: https://en.wikipedia.org/wiki/Grover%27s_algorithm
I don't know what heading I'd suggest for something about how concentration of quantum capabilities will create dangerous asymmetry. (That is why we need post-quantum ("quantum resistant") hash, signature, and encryption algorithms in the near future.)
Quantum attacks on Bitcoin, and how to protect against them (ECDSA, SHA256)
"Quantum attacks on Bitcoin, and how to protect against them (ECDSA, SHA256)" https://www.arxiv-vanity.com/papers/1710.10377/
> […] On the other hand, the elliptic curve signature scheme used by Bitcoin is much more at risk, and could be completely broken by a quantum computer as early as 2027, by the most optimistic estimates.
From https://csrc.nist.gov/Projects/Post-Quantum-Cryptography :
> NIST has initiated a process to solicit, evaluate, and standardize one or more quantum-resistant public-key cryptographic algorithms. Nominations for post-quantum candidate algorithms may now be submitted, up until the final deadline of November 30, 2017.
Project Euler
After hearing about it for years, I decided to start working through Project Euler about two weeks ago. It really is much more about math than programming, although it's a lot of fun to take on the problems with a language that has tail call optimization because so many of the problems involve recurrence relations.
I like that the problems are constructed in a way that usually punishes you for trying to use brute force. Sometimes there's a problem that doesn't have a more elegant solution, though, as if to remind us that brute force often works remarkably well.
I agree with Project Euler being mostly about math. I prefer Codewars for practicing programming or learning a new language.
Euler was a mathematician after all. Now thinking about it I wonder what project Djikstra would look like, or say maybe project Stallman.
There's a Project Rosalind (named after Rosalind Franklin) which is sort of like Project Euler for bioinformatics: http://rosalind.info/about/
I like https://rosalind.info bioinformatics problems because:
- There are problem explanations and an accompanying textbook.
- You can structure the solutions with unit tests that test for known good values.
- There's a graph of problems.
Who’s Afraid of Bitcoin? The Futures Traders Going Short
Wall Street being able to buy Bitcoin might increase demand, but they might also do things that would at least at times amplify sell pressure like panic selling, shorting it, leveraged shorting, margin calls
Statement on Cryptocurrencies and Initial Coin Offerings
A little emphasis, focusing on the aftermath from the SEC's July report on the DAO:
> "Following the issuance of the 21(a) Report, certain market professionals have attempted to highlight utility characteristics of their proposed initial coin offerings in an effort to claim that their proposed tokens or coins are not securities. Many of these assertions appear to elevate form over substance. Merely calling a token a “utility” token or structuring it to provide some utility does not prevent the token from being a security."
I think people are viewing this as an attack on crypto, when its actually just common sense. People put too much faith the 'Contract' half of 'Ethereum/Smart Contract'
Basically. Today ICOs are selling tokens as shares of equity in their company, or similar. Which you can then sell on.
The problem is these companies essentially reserve the right to disregard that contract and could then sell their company, domestically or overseas, for cash, without recompensating any token holders.
Securities regulation and law stops that. But the tokens do need to be lawful securities in order for the court to recognize them.
Otherwise how can the court help you, and who else is going to help you?
> I think people are viewing this as an attack on crypto, when its actually just common sense.
> […] The problem is these companies essentially reserve the right to disregard that contract and could then sell their company, domestically or overseas, for cash, without recompensating any token holders.
> Securities regulation and law stops that. But the tokens do need to be lawful securities in order for the court to recognize them.
This. IRS regards coins and tokens as capital gains taxable things regardless of whether they qualify as securities. SEC exists to protect investors from scams and unfair dealing. In order to protect investors, SEC regulates issuance of securities.
Ask HN: How do you stay focused while programming/working?
I often find myself "needing" to take a mini-break after just a few minutes of concerted effort while coding. In particular, this often occurs after I've made a tiny breakthrough, prompting me to reward myself by checking Twitter or HN. This bad habit quickly derails any momentum. What are some tips to increase focus stamina and avoid distraction?
It's not exactly new and exciting, but I found that listening to calm, instrumental music helps me focus. Mostly Ambient. If you do not like electronic music, Stars Of The Lid or Bohren & Der Club Of Gore are very much worth checking out.
Also, https://mynoise.net/ has worked wonders for me.
In both cases, it seems that unstructured audio input, like, occupies the parts of my mind that would otherwise distract me.
> It's not exactly new and exciting, but I found that listening to calm, instrumental music helps me focus. Mostly Ambient.
Same. Lounge, Ambient, Chillout, Chillstep (https://di.fm has a bunch of great streams. SoundCloud and MixCloud have complete replayable sets, too.)
I've heard that videogame soundtracks are designed to not be distracting; to help focus.
A Hacker Writes a Children's Book
The rhymes and illustrations look great! Is there a board book edition?
Other great STEM and computers books for kids:
"A is for Array"
"Lift-the-Flap Computers and Coding"
"Computational Fairy Tales"
"Hello Ruby: Adventures in Coding"
"Python for Kids: A Playful Introduction To Programming"
"Lauren Ipsum: A Story About Computer Science and Other Improbable Things"
"Rosie Revere, Engineer"
"Ada Byron Lovelace and the Thinking Machine"
"HTML for Babies: Volume 1 of Web Design for Babies"
"What Do You Do With a Problem?"
"What Do You Do With an Idea?"
"ABCs of Mathematics", "The Pythagorean Theorem for Babies", "Non-Euclidian Geometry for Babies", "Introductory Calculus for Infants", "ABCs of Physics", "Statistical Physics for Babies", "Netwonian Physics for Babies", "Optical Physics for Babies", "General Relativity for Babies", "Quantum Physics for Babies", "Quantum Information for Babies", "Quantum Entanglement for Babies"
"ELI5": "Explain like I'm five"
Someone should really make a list of these.
Ask HN: Do ISPs have a legal obligation to not sell minors' web history anymore?
I guess COPPA is still in place and in theory it applies to ISPs, although they may be allowed to assume that all traffic from a household comes from the bill payer who is presumably over 13.
http://kellywarnerlaw.com/childrens-online-privacy-protectio...
https://www.theverge.com/2017/3/31/15138526/isp-privacy-bill...
Tech luminaries call net neutrality vote an 'imminent threat'
> “The current technically-incorrect order discards decades of careful work by FCC chairs from both parties, who understood the threats that Internet access providers could pose to open markets on the Internet.”
Paid prioritization is that threat.
Again, streaming video content for all ages is not more important than online courses.
Ask HN: Can hashes be replaced with optimization problems in blockchain?
CureCoin.
From https://curecoin.net/knowledge-base/about-curecoin/what-is-c... :
> Curecoin allows owners of both ASIC and GPU/CPU hardware to earn. Curecoin puts ASICs to work at what they are good at–securing a blockchain, while it puts GPUs and CPUs to work with work items that can only be done on them–protein folding. While still having a secure blockchain, it supports, and thus is supported by, scientific research.
...
From "CureCoin Reaches #1 Ranking on Folding@home" https://www.newswire.com/news/bio-research-loves-curecoin-ga... :
> As of the afternoon of August 29, 2017 (Eastern Time), the Curecoin Team 224497 earned the world's #1 rank on Stanford's Folding@home - a protein folding simulation Distributed Computing Network (DCN). In a little over 3 years, the team (including our merge-folding partners at Foldingcoin) collectively produced 160 billion points worth of molecular computations to support research in the areas of cancer, Alzheimer's, Huntington's, Parkinson's, Infectious Disease as well as helping scientists uncover new molecular dynamics through groundbreaking computational techniques.
From https://news.ycombinator.com/item?id=15843795 :
> Gridcoin (Berkeley 2013) is built on Proof-of-Stake and Proof-of-Research. Gridcoin is used as payment for computing resources contributed to BOINC.
> I doubt that volatility would be welcome on the Gridcoin blockchain: Wikipedia lists "6.5% Inflation. 1.5% Interest + 5% Research Payments APR" under the Supply Growth infobox attribute.
Ask HN: What could we do with all the mining power of Bitcoin? Fold Protein?
Instead of buzzing SHA-512 in circles like busy bees ad infinitum, is there any way we can use these calculations productively?
Instead of algo-trading the stock markets?!
There are a number of distributed computing projects (e.g. SETI@home): https://en.wikipedia.org/wiki/List_of_distributed_computing_...
The Ethereum White Paper lists a number of applications for blockchains: https://github.com/ethereum/wiki/wiki/White-Paper
(BitCoin is built on SHA-256, Ethereum is built on Keccak-256 (~SHA-3))
Proof-of-Stake is a lower energy alternative to Proof-of-Work with tradeoffs: https://github.com/ethereum/wiki/wiki/Proof-of-Stake-FAQ
Unfortunately, IDK of another way to find secure consensus (blockchains are consensus protocols) in a DDOS-resistant way with unsolved problems?
> Unfortunately, IDK of another way to find secure consensus (blockchains are consensus protocols) in a DDOS-resistant way with unsolved problems?
Gridcoin (Berkeley 2013) is built on Proof-of-Stake and Proof-of-Research. Gridcoin is used as payment for computing resources contributed to BOINC.
I doubt that volatility would be welcome on the Gridcoin blockchain: Wikipedia lists "Supply growth 6.5% Inflation. 1.5% Interest + 5% Research Payments APR" under the Supply Growth infobox attribute.
No CEO needed: These blockchain platforms will let ‘the crowd’ run startups
How much energy does Bitcoin mining really use?
The Actual FCC Net Neutrality Repeal Document. TLDR: Read Pages 82-87 [pdf]
Here are some links to the relevant antitrust laws:
Sherman Antitrust Act (1890) https://en.wikipedia.org/wiki/Sherman_Antitrust_Act
Aspen Skiing Co. v. Aspen Highlands Skiing Corp. (1985) https://en.wikipedia.org/wiki/Aspen_Skiing_Co._v._Aspen_High....
Transparency in network management and paid prioritization practices and agreements will be relevant.
"We find that antitrust law, in combination with the transparency rule we adopt, is particularly well-suited to addressing any potential or actual anticompetitive harms that may arise from paid prioritization arrangements." (p.147)
If antitrust law is sufficient, as you've found, there would be no need for Title II Common Carrier regulation in any industry.
We can call phone numbers provided by any company at the same rate because phone companies are regulated as Title II Common Carriers. ISPs are also common carriers.
"Public airlines, railroads, bus lines, taxicab companies, phone companies, internet service providers,[3] cruise ships, motor carriers (i.e., canal operating companies, trucking companies), and other freight companies generally operate as common carriers."
The 5 most ridiculous things the FCC says in its new net neutrality propaganda
> The Federal Communications Commission put out a final proposal last week to end net neutrality. The proposal opens the door for internet service providers to create fast and slow lanes, to block websites, and to prioritize their own content. This isn’t speculation. It’s all there in the text.
Great. Payola. Thanks Verizon!
Does the FTC have the agreement information needed to hear the anti-trust cases that are sure to result from what are now complaints to the FCC (an organization with network management expertise) being redirected to the FTC?
Title II is the appropriate policy set for ISPs; regardless of how lucrative horizontal integration with content producers seems.
FCC's Pai, addressing net neutrality rules, calls Twitter biased
No. Censoring hate speech by banning people who are verbally assaulting others (in violation of Terms of Service that they agreed to) is a very different concern than requiring common carriers to equally prioritize bits.
If we extend "you must allow people to verbally assault others (because free speech applies to the government)" to TV and radio, what do we end up with?
Note that the FCC fines non-cable TV (broadcast radio and TV) for cursing on air. See "Obscene, Indecent and Profane Broadcasts" https://www.fcc.gov/consumers/guides/obscene-indecent-and-pr...
How can you ask social media companies to do something about fake news (the vast majority of which served to elect the current administration (which nominated this FCC chairman)) while also lambasting them for upholding their commitment to providing a hate-free experience for net citizens and paying advertisers?
"Open Internet": No blocking. No throttling. No paid prioritization.
It would be easier for us to understand the "Open Internet" rules if the proposed "Restoring Internet Freedom" page wasn't crudely pasted over (redirected to from) the page describing the current Open Internet rules. www.fcc.gov/general/open-internet (current policy) now redirects to www.fcc.gov/restoring-internet-freedom (proposed policy).
ISPs blocking, throttling, or paid-prioritizing Twitter, Netflix, Fox, or CNN for everyone is a different concern than responding to individuals who are threatening others with hate speech.
The current policy ("Open Internet") means that you can use the bandwidth cap that you pay for for whatever legal content you please.
The proposed policy ("Restoring Internet Freedom") means that internet businesses will need to pay every ISP in order to not be slower than the big guys who can afford to pay-to-play (~"payola"). https://en.wikipedia.org/wiki/Payola
A curated list of Chaos Engineering resources
Never having heard off 'Chaos Engineering', this seems like a bad case of 'Cargo Cult Engineering'.
That starts with the term 'chaos', which has a well-defined meaning in Chaos Theory, where it is quite obviously borrowed from: small changes in input lead to large changes in output. Neither distributed systems in general, and especially not the sort of system this engineering strives to build, fit that definition. In fact, they are the exact opposite: every part of a typical web stack is already build to mitigate changing demands such as traffic peaks or attacks.
The mumbo jumbo around "defining a steady state" and "disproving the null hypothesis" seems like a veneer of sciency on a rather well-known concept: testing.
A supreme court justice once said: "Good writing is a $10 thought in a 5 cent sentence". This is the opposite.
"Resilience Engineering" would be a good alternative term for these failure scenario simulations and analyses.
Glossary of Systems Theory > A > Adaptive capacity:
> Adaptive capacity: An important part of the resilience of systems in the face of a perturbation, helping to minimise loss of function in individual human, and collective social and biological systems
Technology behind Bitcoin could aid science, report says
Bloom is working on non-academic credit building and scoring.
Hyperledger brings together many great projects and tools which have numerous applications in science and industry.
Is a blockchain necessary? Could we instead just sign JSONLD records with ld-signatures and store them in an eventually or strongly consistent database we all contribute resources to synchronizing and securing?
That's just the centralization or decentralization question.
We can do it all centralized already but we would also all need to trust whoever is hosting this data and trust every single person who has the ability to enter the data.
Less nodes you need to trust the better, in a centralized solution where everyone can contribute you need to be able to trust everyone.
In a decentralized system where everyone can contribute, you don't need to trust anyone but give up benefits of centralization such as speed, performance and usability.
> We can do it all centralized already but we would also all need to trust whoever is hosting this data and trust every single person who has the ability to enter the data.
There are plenty of ways to minimize trust required with traditional cryptography though, this is not all or nothing, we have been doing this since PGP. You can get the overwhelming majority of the benefits with none of the drawbacks.
But how else are you going to hype an ICO with claims about the size of a market and get people who don't understand how blockchains work to give your their BTC/ETH?
Git hash function transition plan
> Some hashes under consideration are SHA-256, SHA-512/256, SHA-256x16, K12, and BLAKE2bp-256.
Not sure what K12 is (Keccak?), but BLAKE2 is a very attractive option.
Vintage Cray Supercomputer Rolls Up to Auction
Google is officially 100% sun and wind powered – 3.0 gigawatts worth
+1000.
TIL this is called "Corporate Renewable Energy Procurement". https://www.google.com/search?q=Corporate+Renewable+Energy+P...
PPA: Power Purchase Agreement https://en.wikipedia.org/wiki/Power_purchase_agreement
Interactive workflows for C++ with Jupyter
QuantStack/xeus-cling https://github.com/QuantStack/xeus-cling
QuantStack/xwidgets https://github.com/QuantStack/xwidgets
QuantStack/xplot (bqplot) https://github.com/QuantStack/xplot
Vanguard Founder Jack Bogle Says ‘Avoid Bitcoin Like the Plague’
Over the past 7 years, Bitcoin has outperformed every security and portfolio that Jack Bogle has recommended.
This is pretty disrespectful to Jack Bogle.
Vanguard is almost singlehandedly responsible for returning trillions of dollars of costs, in the form of fees and underperformance by active managers, back to investors. Millions of investors have benefited.
Bitcoin has been a bubble since $1 and $100 to these people.
What evidence is there that it isn't a bubble? People buy Bitcoin only because they think they can sell it higher. Eventually, you will run out of greater fools.
Nasdaq Plans to Introduce Bitcoin Futures
My guess is that this is probably pretty meaningless.
There are a few things going against them.
- The CBOE and CME are both much larger futures exchanges and are going to be offering futures first
- since you can't net out futures contracts from different exchanges this means they tend to become winner take all
> One way Nasdaq seeks to differentiate itself seems to be in the amount of data it uses for pricing the digital currency contracts. VanEck Associates Corp., which recently withdrew plans for a bitcoin exchange-traded fund, will supply the data used to price the contracts, pulling figures from more than 50 sources, according to the person.
This might be interesting as one of the things that everyone is worried about is price manipulation.
If you haven't thought about how futures work with respect to margin and marking at the end of the trading day you need to know that you can be required to deposit more money into your margin account if the futures trade moves against you on any given day.
This means the marking price is very important and lost of institutional money is worried that the exchanges are easy to manipulate.
see: http://openmarkets.cmegroup.com/3785/understanding-margin-ch...
> Nasdaq’s product will reinvest proceeds from the spin-off back into the original bitcoin in a way meant to make the process more seamless for traders, the person said.
This is awesome,, right now the CBOE and CME both have punted on the question of forks saying, they'll have a best efforts to figure it out.
> One way Nasdaq seeks to differentiate itself seems to be in the amount of data it uses for pricing the digital currency contracts. VanEck Associates Corp., which recently withdrew plans for a bitcoin exchange-traded fund, will supply the data used to price the contracts, pulling figures from more than 50 sources, according to the person. That appears to exceed CME’s plan to use four sources, and Cboe’s one. Nasdaq’s contracts will be cleared by Options Clearing Corp., the person said.
BitMEX bitcoin futures are already online. IDK how many price sources they pull?
Aren't there a few other companies already selling Bitcoin futures?
In general, when the CME enters a market for futures, they take all of the air out of the room. I don't think it's realistic to believe NASDAQ can compete with them.
https://twitter.com/officialmcafee/status/935900326007328768
Well John McAfee thinks bitcoin will hit 1 million by 2020.
Pump and dumpers gonna pump and dump. https://medium.com/@DEFCON_2015/why-mgt-was-delisted-from-ny...
Or, large investment banking houses will step in and create naked shorting opportunities to inflate sell pressure creating 'death spirals' to drive prices down and scoop them up and extreme discounts. This happens in the traditional public markets everyday.
> Or, large investment banking houses will step in and create naked shorting opportunities to inflate sell pressure creating 'death spirals' to drive prices down and scoop them up and extreme discounts. This happens in the traditional public markets everyday.
Is there a term for this?
Yes, this can happen in a few different ways and is the reason why Ycombinator created SAFEs. When you have a public company you will get offers for what are called "credit lines", "debt financing" or "convertible notes". They are traditionally used to create death spirals https://www.investopedia.com/ask/answers/06/deathspiralbond.... as the size of your float increases by you, executive director (CEO/CFO), as a public company "issuing" more stock to cover the loan. The more you issue, the less you're worth until somebody comes along scoops you up and re-engineers the cap table which is a restructuring. However, manipulation can occur within institutions as well: https://news.ycombinator.com/threads?id=KasianFranks&next=14...
Ask HN: Where do you think Bitcoin will be by 2020?
I have a friend who believes it will be $100,000 per BitCoin and his reasoning is 'supply and demand'.
There will be around 18M bitcoins in 2020. [1][2]
[1] https://en.bitcoin.it/wiki/Controlled_supply
This paper [3] suggests we'll be needing to upgrade to quantum-secure hash functions instead of ECDSA before 2027.
[3] "Quantum attacks on Bitcoin, and how to protect against them" https://arxiv.org/abs/1710.10377
Hopefully, Ethereum will have figured out a Proof of Stake [4] solution for distributed consensus which is as resistant to DDOS as Proof of Work; but with less energy consumption (thereby, unfortunately or fortunately, un-incentivizing clean energy as a primary business goal).
[4] https://github.com/ethereum/wiki/wiki/Proof-of-Stake-FAQ
Ask HN: Why would anyone share trading algorithms and compare by performance?
I was speaking with a person years my senior awhile back, and sharing information about the Quantopian platform (which allows users to backtest and share trading algorithms); and he asked me "why would anyone share their trading algorithms [if they're making any money]?"
I tried "to help each other improve their performance". Is there a better way to explain to someone who spends their time reading forums with no objective performance comparisons over historical data why people would help each other improve their algorithmic trading algorithms?
Catalyst, like Quantopian, is also built on top of Zipline; but for cryptocurrencies. https://enigmampc.github.io/catalyst/example-algos.html
Zipline (backtesting and live trading of algorithms with initialize(context) and handle_data(context, data) functions; with the SPY S&P 500 ETF as a benchmark) https://github.com/quantopian/zipline
Pyfolio (for objectively comparing the performance of trading strategies over time) https://github.com/quantopian/pyfolio
...
"Community Algorithms Migrated to Quantopian 2" https://www.quantopian.com/posts/community-algorithms-migrat...
- "Reply to minimum variance w/ contrast" seems to far outperform the S&P 500.
Ask HN: CS papers for software architecture and design?
Can you please point me to some papers that you consider very influential for your work or that you believe they played significant role on how we structure our software nowdays?
"The Architecture of Open Source Applications" Volumes I & II http://aosabook.org/en/
"Manifesto for Agile Software Development" https://en.wikipedia.org/wiki/Agile_software_development#The...
"Catalog of Patterns of Enterprise Application Architecture" https://martinfowler.com/eaaCatalog/
Fowler > Publications ("Refactoring ",) https://en.wikipedia.org/wiki/Martin_Fowler#Publications
"Design Patterns: Elements of Reusable Object-Oriented Software" (GoF book) https://en.wikipedia.org/wiki/Design_Patterns
.
UNIX Philosophy https://en.wikipedia.org/wiki/Unix_philosophy
Plan 9 https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs
## Distributed Systems
CORBA > Problems and Criticism (monolithic standards, oversimplification,): https://en.wikipedia.org/wiki/Common_Object_Request_Broker_A...
Bulk Synchronous Parallel: https://en.wikipedia.org/wiki/Bulk_synchronous_parallel
Paxos: https://en.wikipedia.org/wiki/Paxos_(computer_science)
Raft: https://en.wikipedia.org/wiki/Raft_(computer_science) #Safety
CAP theorem: https://en.wikipedia.org/wiki/CAP_theorem
Keeping a Lab Notebook [pdf]
I'd love to hear some thoughts about keeping a "lab notebook" for ML experiments. I use Jupyter Notebooks when playing around with different ML models, and I find that it really helps to document my thought process with notes and comments. It also seems that the ML workflow is very 'experiment' driven. I'm always thinking "Hm, I think if I tweak this hyperparameter this way, or adjust this layer this way, then I'll get a better result because X". Thus, I have a bit of a hypothesis and proposed experiment. I run that model, and see if it improved or not.
Then, I run into an issue where I can either: 1. overwrite the original model with my new hyperparameters/design and re-run and analyze or 2. keep adding to the same notebook "page" with a new hypothesis/test/analysis loop, thus making the notebook pretty large. With number 1, I often want to backtrack and re-reference how a previous experiment went, but I lose that history. With number 2, it seems to get big pretty quickly, and coming back to the same notebook requires more setup, and "searching" the history gets more cumbersome.
Does anyone try using a separate notebook page for each experiment, maybe with a timestamp or "version"? Or is there a better way to do this in a single notebook? I am thinking that something like "chapters" could help me here, and it seems like this extension might help me: https://github.com/minrk/ipython_extensions#table-of-content...
These are ASCII-sortable:
0001_Introduction.ipynb
0010_Chapter-1.ipynb
ISO8601 w/ UTC is also ASCII sortable.
# Jupyter notebooks as lab notebooks
## Disadvantages
### Mutability
With a lab notebook, you can cross things out but they're still there.
- [ ] ENH: Copy cell and mark as don't execute (or wrap with ```language\n``` and change the cell type to markdown)
- [ ] ENH: add a 'Save and {git,} Commit' shortcut
CoCalc (was: SageMathCloud) has (somewhat?) complete notebook replay with a time slider; and multi-user collaborative editing. ("Time-travel is a detailed history of all your edits and everything is backed up in consistent snapshots.")
### Timestamps
You must add timestamps by hand; i.e. as #comments or markdown cells.
- [ ] ENH: add a markdown cell with a timestamp (from a configurable template) (with a keyboard shortcut)
### Project files
You must manage the non-.ipynb sources separately. (You can create a new file or folder. You can just drag and drop to upload. You can open a shell tab to `git status diff commit` and `git push`, if the Jupyter/JupyterHub/CoCalc instance has network access to e.g. GitLab or GitHub)
## Advantages
### Reproducibility Executable I/O cells
The version_information and/or watermark extensions will inline the software versions that were installed when the notebook was last run
Dockerfile for OS config
Conda environment.yml (and/or pip requirements.txt and/or pipenv Pipfile) for further software dependencies
BinderHub can rebuild a docker image on receipt of a webhook from a got repo, push the built image to a docker image repository, and then host prepared Jupyter instances (with Kubernetes) which contain (and reproducibly archive) all of the preinstalled prerequisites.
Diff: `git diff`, `nbdime`
### Publishing
You can generate static HTML, HTML slides with RevealJS, interactive HTML slides with RISE, executable source with comments (e.g. a .py file), LaTeX, and PDF with 'Save as' or `jupyter-convert --to`. You can also create slides with nbpresent.
MyBinder.org and Azure Notebooks have badges for e.g. a README.md or README.rst which launch a project executably in a docker instance hosted in a cloud. CoCalc and Anaconda Cloud also provide hosted Jupyter Notebook projects.
You can template a gradable notebook with nbgrader.
GitHub renders .ipynb notebooks as HTML. Nbviewer renders .ipynb notebooks as HTML.
There are more than 90 Jupyter Kernels for languages other than Python.
https://github.com/quobit/awesome-python-in-education#jupyte...
How to teach technical concepts with cartoons
There's not a Wikipedia page for "visual metaphor", but there are pages for "visual rhetoric" https://en.wikipedia.org/wiki/Visual_rhetoric and "visual thinking" https://en.wikipedia.org/wiki/Visual_thinking
Negative space can be both meaningful and useful later on.
I learned about visual thinking and visual metaphor in application to business communications from "The Back of the Napkin: Solving Problems and Selling Ideas with Pictures" http://www.danroam.com/the-back-of-the-napkin/
Fact Checks
Indeed, fact checking systems are only as good as the link between identity credentialing services and a person.
http://schema.org/ClaimReview (as mentioned in this article) is a good start.
A few other approaches to be aware of:
"Reality Check is a crowd-sourced on-chain smart contract oracle system" [built on the Ethereum smart contracts and blockchain]. https://realitykeys.github.io/realitycheck/docs/html/
And standards-based approaches are not far behind:
W3C Credentials Community Group https://w3c-ccg.github.io/
W3C Verifiable Claims Working Group https://www.w3.org/2017/vc/WG/
W3C Verifiable News https://github.com/w3c-ccg/verifiable-news
In terms of verifying (or validating) subjective opinions, correlational observations, and inferences of causal relations; #LinkedMetaAnalyses of documents (notebooks) containing structured links to their data as premises would be ideal. Unfortunately, PDF is not very helpful in accomplishing that objective (in addition to being a terrible format for review with screen reader and mobile devices): I think HTML with RDFa (and/or CSVW JSONLD) is our best hope of making at least partially automated verification of meta analyses a reality.
DHS orders agencies to adopt DMARC email security
From https://www.cyberscoop.com/dhs-dmarc-mandate/ :
> By Jan. 2018, all federal agencies will be required to implement DMARC across all government email domains.
> Additionally, by Feb. 2018, those same agencies will have to employ Hypertext Transfer Protocol Secure (HTTPS) for all .gov websites, which ensures enhanced website certifications.
The electricity for 1BTC trade could power a house for a month
The article seems to imply that a 1BTC transaction requires 200kWh of energy.
First, what is the source for that number?
Second, what is the business interest of the quoted individual? Are they promoting competing services?
Third, how much energy does the supposed alternative really take, by comparison?
How much energy do these aspects of said business operations require:
- Travel to and from the office for n employees
- Dry cleaning for n employees' work clothes
- Lights for an office of how many square feet
- Fraud investigations in hours worked, postal costs, wait times, CPU time and bandwidth to try and fix data silos' ledgers' transaction ids and time skew; with a full table JOIN on data nobody can only have for a little while from over here and over there
- Desktop machines' idle hours
- Server machines' idle hours
With low cost clean energy, these businesses are profitable; with a very different cost structure than traditional banking and trading.
PAC Fundraising with Ethereum Contracts?
I'll cc this here with formatting changes (extra \n and ---) for Hacker News:
---
### Background
- PAC: Political Action Committee https://en.wikipedia.org/wiki/Political_action_committee
- https://github.com/holographicio/awesome-token-sale
### Questions
- Is Civic PAC fundraising similar to e.g. a Crowdsale or a CappedCrowdsale or something else entirely, in terms of ERC20 OpenZeppelin solidity contracts?
- Would it be worth maintaining an additional contract for [PAC] "fundraising" with terminology that campaigns can understand; or a terminology map?
- Compared to just accepting donations at a wallet address, or just accepting credit/debt card donations, what are the risks of a token sale for a PAC?
--- Is there any way to check for donors' citizenship? (When/Where is it necessary to check donors' citizenship (with credit/debit cards or cryptocoins/cryptotokens?))
- Compared to just accepting donations at a wallet address, or just accepting credit/debt card donations, what are the costs of a token sale for a PAC?
--- How much gas would such a contract require?
- Compared to just accepting donations at a wallet address, or just accepting credit/debt card donations, what are the benefits of a token sale for a PAC?
---- Lower transaction fees than credit/debit cards?
---- Time limit (practicality, marketing)
---- Cap ("we only need this much")
---- Refunds in the event of […]
### Objectives
- Comply with all local campaign finance laws
--- Collect citizenship information for a Person
--- Collect citizenship information for an Organization 'person'
- Ensure that donations hold value
- Raise funds
- Raise funds up to a cap
- (Optionally?) collect names and contact information ( https://schema.org/Person https://schem
Thanks for the wealth of resources in this post. Here are a few more:
"Python for Finance: Analyze Big Financial Data" (2014, 2018) https://g.co/kgs/qkY8J6 ... https://pyalgo.tpq.io also includes the "Finance with Python" course and this book as a PDF and Jupyter notebooks.
Quantopian put out a call for the best Value Investing algos (implemented in quantopian/zipline) awhile back. This post links to those and other value investing resources: https://westurner.github.io/hnlog/#comment-19181453 (Ctrl-F "econo")
"Lectures in Quantitative Economics as Python and Julia Notebooks" https://news.ycombinator.com/item?id=19083479 links to these excellent lectures and a number of tools for working with actual data from FRED, ECB, Eurostat, ILO, IMF, OECD, UNSD, UNESCO, World Bank, Quandl.
One thing that many finance majors, courses, and resources often fail to identify is the role that startup and small businesses play in economic growth: jobs, GDP, return on direct capital investment. Most do not succeed, but it is possible to do better than index funds and have far more impact in terms of sustainable investment than as an owner of a nearly-sure-bet index fund that owns some shares and takes a hands-off approach to business management, research, product development, and operations.
Is it possible to possess a comprehensive understanding of finance and economics but still not have personal finance down? Personal finance: r/personalfinance/wiki, "Consumer science (a.k.a. home economics) as a college major" https://news.ycombinator.com/item?id=17894632