Contents^
Items^
Salesforce Sustainability Cloud Becomes Generally Available
Are there similar services for Sustainability Reporting and accountability? https://en.wikipedia.org/wiki/Sustainability_reporting
Httpx: A next-generation HTTP client for Python
> Fully type annotated.
This is a huge win compare to requests. AFAIKT requests is too flexible (read: easier to misuse) and difficult to add type annotation now. The author of requests gave a horrible type annotation example here [0].
IMO at this time when you evaluate a new Python library before adopting, "having type annotation" should be as important as "having decent unit test coverage".
The huge win against requests is that Https is fully async.. you can download 20 files in parallel without not too much effort. Throughout in python asyncio is amazing, something similar to node or perhaps better... That's the main point.
FWIW, requests3 has "Type-annotations for all public-facing APIs", asyncio, HTTP/2, connection pooling, timeouts, etc https://github.com/kennethreitz/requests3
Sorry, have you checked the source? Are these features there or only announced? Has requests added a timeout by default finally?
It looks like requests is now owned by PSF. https://github.com/psf/requests
But IDK why requests3 wasn't transferred as well, and why issues appear to be disabled on the repo now.
The docs reference a timeout arg (that appears to default to the socket default timeout) for connect and/or read https://3.python-requests.org/user/advanced/#timeouts
And the tests reference a timeout argument. If that doesn't work, I wonder how much work it would be to send a PR (instead of just talking s to Ken and not contributing any code)
>But IDK why requests3 wasn't transferred as well, and
That's thing... Who knows..
TIL requests3 beta works with httpx as a backend: https://github.com/not-kennethreitz/team/issues/21#issuecomm...
If requests3 is installed, `import requests' imports requests3
BlackRock CEO: Climate Crisis Will Reshape Finance
+1. From the letter: https://www.blackrock.com/us/individual/larry-fink-ceo-lette...
> The money we manage is not our own. It belongs to people in dozens of countries trying to finance long-term goals like retirement. And we have a deep responsibility to these institutions and individuals – who are shareholders in your company and thousands of others – to promote long-term value.
> Climate change has become a defining factor in companies’ long-term prospects. Last September, when millions of people took to the streets to demand action on climate change, many of them emphasized the significant and lasting impact that it will have on economic growth and prosperity – a risk that markets to date have been slower to reflect. But awareness is rapidly changing, and I believe we are on the edge of a fundamental reshaping of finance.
> The evidence on climate risk is compelling investors to reassess core assumptions about modern finance. Research from a wide range of organizations – including the UN’s Intergovernmental Panel on Climate Change, the BlackRock Investment Institute, and many others, including new studies from McKinsey on the socioeconomic implications of physical climate risk – is deepening our understanding of how climate risk will impact both our physical world and the global system that finances economic growth.
Environmental, social and corporate governance > Responsible investment: https://en.wikipedia.org/wiki/Environmental,_social_and_corp...
Corporate social responsibility: https://en.wikipedia.org/wiki/Corporate_social_responsibilit...
UN-supported PRI: Principles for Responsible Investment (2,350 signatories (2019-04)) https://en.wikipedia.org/wiki/Principles_for_Responsible_Inv...
A lot of complex “scalable” systems can be done with a simple, single C++ server
Many developers severely underestimate how much workload can be served by a single modern server and high-quality C++ systems code. I've scaled distributed workloads 10x by moving them to a single server and a different software architecture more suited for scale-up, dramatically reducing system complexity as a bonus. The number of compute workloads I see that actually need scale-out is vanishingly small even in industries known for their data intensity. You can often serve millions of requests per second from a single server even when most of your data model resides on disk.
We've become so accustomed to extremely inefficient software systems that we've lost all perspective on what is possible.
Can you expand on this? I have some pretty massive compute loads that need to be scaled onto a cluster with 100+ workers for most computations. This is after I use a library called dask that graphically does its own mapreduce optimisation inside its modules. This is all for a relatively small 250GB raw data file that I keep in a csv (and need to convert to SQL at some point).
Are you saying this can be optimised to fit inside a single 10 core server in terms of compute loads?
Don't know why you're being downvoted but I'll assume your question is genuine.
You use a cluster when your data and compute requirements are large and parallel enough that the tax paid on network latency trumps the 10-20X speedup you get on SSD and 1000X speedup you get from just keeping data in RAM.
250 Gigs is tiny enough that you could probably much get better performance running on high memory instance in AWS or GCP. You'll generally have to write your own multiprocessing code though which is fairly simple - your existing library may also be able to support it.
I once actually ran this kind of workload on just my laptop using a compiled language that performed better than pyspark on a cluster.
I'd love to keep it in RAM if I could. The problem is, the library I'm familiar with (pandas) typically seems to take more memory than the original csv file once it loads it onto memory. I know this is due to bad data types but in certain cases, I cannot get around those.
However, even if I could load it all into memory at once, and assuming it takes 200 gb, I'm still using a master's student access to a cluster. So I get preempted like it's nobody's business. Hence why I prefer a smaller memory footprint even if I take up cpus at variable rates through a single execution.
I did try to write my own multiprocessing code for this, but the operations are sometimes too complicated (like groupby) for me to rewrite everything from the ground up. If I'm not reliant on serial data communication between processes (like you'd need to sort a column), I can get it done pretty easily. In fact, I wrote my data cleaning code with this and cleaned up the entire file in half an hour because single chunks didn't rely on others.
However, if you have some idea of how to run these computational loads in parallel in python or any other language on single compute instances (like the size of a laptop's memory of 16 gb), I'd really love to see it. Thanks.
Numpy supports memory mapping `ndarrays` which can back a DataFrame in pandas. This lets you access a dataset far larger than will fit in RAM as if it lived in RAM. Provided it's on fast SSD storage you'll have speedy access to the data and can process huge chunks at once.
Can you provide a link to this please? My current knowledge is that all numpy data lives in memory, and pandas itself has a feature to fragment any data into iterables so I can read upto my memory limit. I cannot use this feature due to the serial nature of some of the operations that I alluded to (I'd have to almost rewrite the entire library for some of these complicated operations like groupby and sorting).
I do have fast SSD storage because it's on the scratch drive of a cluster and from what I've seen it can do ~300-400 MB/s easily. I haven't had a chance to test more than that since I'm mostly memory constrained in much of my testing.
My current attempt is to push this data into a pure database handling system like SQL so that I can query it. But like I said, I work with a less-than-stellar set of tools and I have to literally set up a postgres server from ground up to write to it. Which shouldn't be a big deal except when it's on a non-root user and I have to keep remapping dependencies (took 5-6 hours to set it up on the instance I have access to).
My other option was to write the entire 250 GB to a sqllite database using the sqlalchemy library in Python, but that seems to fail whether I do it with parallel writes, or serial writes. In both cases, it fails after I create ~64-70 tables.
Dask groupby example: https://examples.dask.org/dataframes/02-groupby.html
> Generally speaking, Dask.dataframe groupby-aggregations are roughly same performance as Pandas groupby-aggregations, just more scalable.
The dask.distributed scheduler can also run on one high-RAM instance (with threads or processes) https://docs.dask.org/en/latest/setup.html
Pandas docs > Ecosystem > Out-of-core: https://pandas.pydata.org/pandas-docs/stable/ecosystem.html#...
Reading from Parquet into Apache Arrow is much faster than CSV because the data can just be directly loaded into RAM. https://ursalabs.org/blog/2019-10-columnar-perf/
If you have GPU instances, cuDF has a Pandas-like API on top of Apache Arrow. https://github.com/rapidsai/cudf
> Built based on the Apache Arrow columnar memory format, cuDF is a GPU DataFrame library for loading, joining, aggregating, filtering, and otherwise manipulating data.
> cuDF provides a pandas-like API that will be familiar to data engineers & data scientists, so they can use it to easily accelerate their workflows without going into the details of CUDA programming.
Dask-ML makes scalable scikit-learn, XGBoost, TensorFlow really easy. https://dask-ml.readthedocs.io/en/latest/
... re: the OT: While it's possible to write C++ code that's really fast, it's generally inflexible, expensive to develop, and dangerous for devs with experience in their respective domains of experience to write. Much saner to put a Python API on top and optimize that during compilation.
There are a few C++ frameworks in the top quartile of the TechEmpower framework benchmarks. https://www.techempower.com/benchmarks/
Hardware/hosting is relatively cheap. Developers and memory vulnerabilities aren't.
Unfortunately, I'm not sure what's wrong with dask, but it doesn't work properly on my cluster. I tested it on an exceedingly simple operation - find all unique values in a very big column (5 billion rows, but I know for a fact that there are only 500-502 unique values in there). With a 100 workers, it still failed. Now this is an embarrassingly parallel operation that can be implemented trivially. So I'm not sure if there's a problem with my cluster or if dask just does not work with slurm clusters very well.
https://docs.dask.org/en/latest/setup/hpc.html says dask-jobqueue handles "PBS, SLURM, LSF, SGE and other resource managers"
"Dask on HPC, what works and what doesn't" https://github.com/dask/dask-blog/issues/5
Maybe you should spend some time developing a job visualization system for end users from scratch, for end users with lots of C, JS, and HTML experience https://jobqueue.dask.org/en/latest/interactive.html
Yes, I think the library still isn't ready for non HPC experts to use without tinkering. I don't have that level of expertise. I'm a data person who can handle working tools.
Warren Buffett is spending billions to make Iowa 'the Saudi Arabia of wind'
It's both cost-rational and environment-rational to invest heavily in clean energy (with or without the comparatively paltry tax incentives).
The long-term costs of climate change and inaction are unfortunately still mostly external costs to energy producers. We should expect that to change as we start developing competencies in evaluating the costs and frequency of weather disasters exacerbated by anthropogenic climate change. We all get to pay for floods, fires, tornados, hurricanes, landslides, blizzards, and the gosh darn heat.
Insurance firms clearly see these costs. Our military sees the costs of responding to natural disasters. Local economies see the costs of months and years spent on disaster relief; on just getting back up to speed so that they can generate profit from selling goods and services (and pay taxes to support disaster relief efforts essential to operational readiness).
The cost per kilowatt hour of wind (and solar) energy is now lower than operating existing dirty energy plants that dump soot on our crops, air, and water.
With wind, they talk about the "alligator curve". With solar, it's the "duck curve". Grid-scale energy storage is necessary for reaching 100% renewable energy as soon as possible.
Iowa's renewable energy tax incentives are logically aligned with international long-term goals:
UN Sustainable Development Goal 7: Affordable and Clean Energy https://www.globalgoals.org/7-affordable-and-clean-energy
Goal 13: Climate Action https://www.globalgoals.org/13-climate-action
SDG Target 12.6: "Encourage companies to adopt sustainable practices and sustainability reporting" (CSR; e.g. GRI Sustainability Reporting Standards that we can score portfolios with)
https://www.undp.org/content/undp/en/home/sustainable-develo... :
> Rationalize inefficient fossil-fuel subsidies that encourage wasteful consumption by removing market distortions, in accordance with national circumstances, including by restructuring taxation and phasing out those harmful subsidies, where they exist, to reflect their environmental impacts, taking fully into account the specific needs and conditions of developing countries and minimizing the possible adverse impacts on their development in a manner that protects the poor and the affected communities
...
> Thanks. How can I say "try and only run this [computational workload] in zones with 100% PPA offsets or 100% directly sourced #CleanEnergy"? #Goal7 #Goal11 #Goal12 #Goal13 #GlobalGoals #SDGs
It makes good business sense to invest in clean energy to take advantage of tax incentives, minimize future costs to other business units (e.g. insurance, taxes), and earn the support of investors choosing portfolios with long term environmental (and thus economic) sustainability as a primary objective.
Scientists Likely Found Way to Grow New Teeth for Patients
"Scientists Have Discovered a Drug That Fixes Cavities and Regrows Teeth" https://futurism.com/neoscope/scientists-have-discovered-thi...
Tideglusib https://en.wikipedia.org/wiki/Tideglusib
> Tooth repair mechanisms that promotes dentine reinforcement of a sponge structure until the sponge biodegrades, leaving a solid dentine structure. In 2016, the results of animal studies were reported in which 0.14 mm holes in mouse teeth were permanently filled.
Very interesting mechanism.
GSK3 inhibitors are interesting, but we don't understand much the subtypes that are likely to exist: "GSK-3 appears to both promote and inhibit apoptosis, and this regulation varies depending on the specific molecular and cellular context."
Still, I believe better GSK3i will find a role in autologous bone-marrow grafting therapies to fight senescence: with TERC/TERT overexpression (the opposite of DeGrey WILT ideas) to send new stem cells to the tissues - just like how you can find the donors chromosomes in most tissues of grafted patient.
Announcing the New PubMed
I'm actually part of the team developing the new PubMed. Very curious and interested to know what the hacker news community thinks and feels about their experience. https://pubmed.gov/labs
This looks great: I like the search timeline, the ability to easily search for free full-text meta-analyses (a selection bias we should all be aware of), the MeSH term listing in a reasonably-sized font, and that there's schema.org/ImageObject metadata within the page, but there's no [Medical]ScholarlyArticle metadata?
I've worked with Google Scholar (:o) [1], Semantic Scholar (Allen Institute for AI) [2], Meta (Chan Zuckerberg Institute) [3], Zotero, Mendeley and a number of other tools for indexing and extracting metadata and graph relations from https://schema.org/ScholarlyArticle and MedicalScholarlyArticles . Without RDFa (or Microdata, or JSON-LD) in PDF, there's a lot of parsing that has to go down in order to get a graph from the citations in the article. Each service adds value to this graph of resources. Pushing forward on publishing linked research that's reproducible (#LinkedResearch, #LinkedReproducibility) is a worthwhile investment in meta-research that we have barely yet addressed:
> http://Schema.org/NewsArticle .citation: https://schema.org/citation ... Wouldn't it be great if NewsArticles linked to the ScholarlyArticle and/or Notebook CreativeWorks that they're .about (with reified relations)?
> A practical use case: Alice wants to publish a ScholarlyArticle [1] (in HTML with structured data, as a PDF) predicated upon Datasets [2] (as CSV, CSVW JSONLD, XLSX (DataDownload)) with static HTML (and no special HTTP headers). 1 https://schema.org/ScholarlyArticle 2 https://schema.org/Dataset*
> B wants to build a meta analysis: to collect a # of ScholarlyArticles and Dataset DataDownloads; review study controls and data; merge, join, & concatenate Datasets if appropriate, and inductively or deductively infer a conclusion and suggestions for further studies of variance*
The Linked Open Data Cloud shows the edges, the relations, the structured data links between very many (life sciences) datasets: https://lod-cloud.net/ . https://5stardata.info/en/ lists TimBL's suggested 5-start deployment schema for Open Data; which culuminates in publishing linked open data in non-proprietary formats that uses URIs to describe and link to things.
Could any of these [1][2][3][4][5] services cross-link the described resources, given a common URI identifier such as https://schema.org/identifier and/or https://schema.org/url ? ORCID is a service for generating stable identifiers for researchers and publishers who have names in common but different emails. W3C DID solves for this need in a different way.
When I check an article result page with the OpenLink OSDS extension (or any of a number of other tools for extracting structured data from HTML pages (and documents!) https://github.com/CodeForAntarctica/codeforantarctica.githu... ), there could be quite a bit more data there for search engines, browser extensions, and meta-research tools.
Is this something like ElasticSearch on the backend? It is possible to store JSON-LD documents in the search index. I threw together elasticsearchjsonld to "Generate JSON-LD @contexts from ElasticSearch JSON Mappings" for the OpenFDA FAERS data a few year ago. That's not GraphQL or SPARQL, but it's something and it's Linked Data.
re: "Canada's Decision To Make Public More Clinical Trial Data Puts Pressure On FDA" https://news.ycombinator.com/item?id=21232183
> We really could get more out of this data through international collaboration and through linked data (e.g. URIs for columns). See: "Open, and Linked, FDA data" https://github.com/FDA/openfda/issues/5#issuecomment-5392966... and "ENH: Adverse Event Count / 'Use' Count Heatmap" https://github.com/FDA/openfda/issues/49 . With sales/usage counts, we'd have a denominator with which we could calculate relative hazard.
W3C Web Annotations handle threaded comments and highlights; reviewing the reviewers is left as an exercise for the reader. Does Zotero still make it easy to save the bibliographic metadata for one or more ScholarlyArticles from PubMed to a collection in the cloud (and add metadata/annotations)?
Sorry to toot my own horn here. Great job on this. This opens up many new opportunities for research.
[1] https://scholar.google.com
[2] https://www.semanticscholar.org/
Pubmed publishes its dataset for download. Its rather large but update files come frequently. Its amazing. I beleive NIH adds the MESH terms.
ftp://ftp.ncbi.nlm.nih.gov/pubmed/
We had someone do a project with it. downloaded the dataset and used it and create a tool to do some searches that we found useful to find colaborators: (last author, working on a specific gene, paper counts, most recent).
Searching by Mesh Terms across species, and search with orthologs.
The dataset sometimes has a hard time disambiguating names (I think the european dataset assigns Ids to names)
To make sure your feedback is heard, please use "Feedback button" found in bottom right corner of https://pubmed.gov/labs
Ask HN: Is it worth it to learn C in 2020?
The GNU/Linux kernel, FreeBSD kernel, Windows kernel, MacOS kernel, Python, Ruby, Perl, PHP, NodeJS, and NumPy are all written in C. If you want to review and contribute code, you'd need to learn C.
There are a number of coding guidelines e.g. for safety-critical systems where bounded running time and resource consumption are essential. These coding guidelines and standards are basically only available for C, C++, and Ada. https://github.com/stanislaw/awesome-safety-critical/blob/ma...
Even though modern languages have garbage-collection that runs whenever it feels like it, It's helpful to learn about memory management in C (or C++). You'll appreciate object destructor methods that free memory and sockets and file handles that much more. Reference cycles in object graphs are easier to handle with modern C++ than with C. Are there RAII (Resource Acquisition is Initialization) "smart pointers" that track reference counts in C?
Without OO namespacing, in C, function names are often prefixed with namespaces. How many ways could a struct be initialized? When can I free that memory?
When strace prints a syscall, what is that?
Is it necessary to learn C? Somebody needs to maintain and improve the C-based foundation for most of our OSs and very many of our fancy scripting languages. C can be very unforgiving: it's really easy to do it wrong, and there's a lot to keep in mind at once: the cognitive burden is higher with C (and then still with ASM and WebASM) than with an interpreted (or compiled) duck-typed 3GL scripting language with first-class functions.
What's a good progression that includes syntax, finding and reading the libc docs, Make/CMake/Autotools, secure recommended compiler flags for GCC (CPPFLAGS, CFLAGS, LDFLAGS) and LLVM Clang?
C: https://learnxinyminutes.com/docs/c/
C++: https://learnxinyminutes.com/docs/c++/
Links to the docs for Libc and other tools: https://westurner.github.io/tools/#libc
xeus-cling is a Jupyter kernel for C++ (and most of C) that works with nbgrader. https://github.com/QuantStack/xeus-cling
What's a better unit-testing library for C/C++ than gtest? https://github.com/google/googletest/
Amazing comment, thank you for your detailed input and the awesome resources. I want to get into C because I like network and system programming but lack the underlying knowledge and experience. These links will help me start on that new path. Very much appreciated sir :)
For network programming, you might consider asynchronous programming with coroutines. C++20 has them and they're already supported in LLVM. For C, there are a number of implementations of coroutines: https://en.wikipedia.org/wiki/Coroutine#Implementations_for_...
> Once a second call stack has been obtained with one of the methods listed above, the setjmp and longjmp functions in the standard C library can then be used to implement the switches between coroutines. These functions save and restore, respectively, the stack pointer, program counter, callee-saved registers, and any other internal state as required by the ABI, such that returning to a coroutine after having yielded restores all the state that would be restored upon returning from a function call. Minimalist implementations, which do not piggyback off the setjmp and longjmp functions, may achieve the same result via a small block of inline assembly which swaps merely the stack pointer and program counter, and clobbers all other registers. This can be significantly faster, as setjmp and longjmp must conservatively store all registers which may be in use according to the ABI, whereas the clobber method allows the compiler to store (by spilling to the stack) only what it knows is actually in use.
CPython's asyncio implementation (originally codenamed 'tulip') is written in C and IMHO much easier to use than callbacks like Twisted and JS before Promises and the inclusion of tulip-like async/await keywords in ECMAscript. Uvloop - based on libuv, like Node - is apparently the fastest asyncio event loop. CPython Asyncio C module source: https://github.com/python/cpython/blob/master/Modules/_async... Asyncio docs: https://docs.python.org/3/library/asyncio.html
(When things like file or network I/O are I/O bound, the program can yield to allow other asynchronous coroutines ('async') to run on that core. With network programming, we're typically waiting for things to send or reply.)
Return-oriented-programming > Return-into-library technique is an interesting read regarding system programming :) https://en.wikipedia.org/wiki/Return-oriented_programming#Re...
Free and Open-Source Mathematics Textbooks
This is a good list of books. Unfortunately many of the links are broken? Probably just my luck, but the first few "with Sage" books I excitedly selected unfortunately 404'd. I'll send an email.
> Moreover, the American Institute of Mathematics maintains a list of approved open-source textbooks. https://aimath.org/textbooks/approved-textbooks/
I also like the (free) Green Tea Press books: Think Stats, Think Bayes, Think DSP, Think Complexity, Modeling and Simulation in Python, Think Python 2e: How To Think Like a Computer Scientist https://greenteapress.com/wp/
And IDK how many times I've recommended the book for the OCW "Mathematics for Computer Science" course: https://ocw.mit.edu/courses/electrical-engineering-and-compu...
There may be a newer edition than the 2017 version of the book: https://courses.csail.mit.edu/6.042/spring17/mcs.pdf
Does anyone have a good strategy for archiving and downloading the textbooks from aimath.org? They are all excellently formatted in HTML, but I am not certain what the best way to get the complete book would be.
When I have books in form of webpages I normally write a small crawler in python, extract the text div with beautifulsoup, add <hn> tags for chapter names and throw them all together in html form. Add a cover image and combine everything with pandoc.
Nothing fancy but works reliable in an automated fashion
I've found MIT's "Mathematics for Computer Science" actually hard to read, and in my opinion it cannot really be called a proper book, more like a collection of notes, as it skips over many details, jumping to conclusions, definitions, results, without giving you an intuitive explanation. In that regards it's actually hard to follow.
It's fast-going. You may want to start with something more introductory, like:
Levin http://discrete.openmathbooks.org/dmoi3.html
Hammack https://www.people.vcu.edu/~rhammack/BookOfProof/
the early parts of Newstead https://infinitedescent.xyz/
Make CPython segfault in 5 lines of code
Applications Are Now Open for YC Startup School – Starts in January
I'm presently living in a fairly rural part of the US due to an illness in the family. In the town of 14,000 I currently reside in it's pretty difficult to network in a meaningful way and talk about my company with folks that can give guidance and feedback. Really excited for Startup School and so grateful that YC has put this together.
> In the town of 14,000 I currently reside in it's pretty difficult to network in a meaningful way and talk about my company with folks that can give guidance and feedback
GitLab and Zapier are examples of all remote former YC companies.
"GitLab Handbook" https://about.gitlab.com/handbook/
"The Ultimate Guide to Remote Work: Lessons from a team of over 200 remote workers" https://zapier.com/learn/remote-work/
Were those companies remote when they did YC? I can't recall exactly where offhand but I seem to remember Paul Graham (or was it Sam Altman?) saying that they were strongly against remote teams in the early stages. It was a few years ago though, I wonder if their opinions have changed.
Startup School is now designed as a remote program.
It'd be interesting to hear from them about building all remote team culture with transparency and accountability. Are text-chat "digital stand up meetings" with quality transcripts of each team member's responses to the three questions enough? ( Yesterday / Today and Tomorrow / Obstacles // What did I do since the last time we met? What will I do before the next time we meet? What obstacles are blocking my progress? )
Or are there longer term planning sessions focusing on a plan for delivering value on a far longer term than first getting the MVP down and maximizing marginal profit by minimizing costs?
‘Adulting’ is hard. UC Berkeley has a class for that
+1 for Life Skills for Adulting and also Home Economics including Family and Meal Planning.
A bunch of resources from "Consumer science (a.k.a. home economics) as a college major" https://news.ycombinator.com/item?id=17894632 : CS 007: Personal Finance for Engineers, r/personalfinance/wiki, Healthy Eating Plate, Khan Academy > Science > Health and Medicine
And also, Instant Pot. The Instant Pot pressure cooker is your key to nutrient preservation and ultimate happiness.
Founder came back after 8 years to rewrite flash photoshop in canvas/WebGL
Five cities account for vast majority of growth in U.S. tech jobs: study
> Boston, San Francisco, San Jose, Seattle, and San Diego—accounted for more than 90% of the nation’s innovation-sector growth during the years 2005 to 2017
Why 2005 and why'd it drop off post-2017?
Also surprised DC and NYC aren't on the list. Lots of new companies and growth in those areas, but I guess they may not qualify as "innovation-sector growth".
> To that end, the present paper proposes that Congress assemble and award to a select set of metropolitan areas a major package of federal innovation inputs and supports that would accelerate their innovation-sector scale-up. Along these lines, we envision Congress establishing a rigorous competitive process by which the most promising eight to 10 potential growth centers would receive substantial financial and regulatory support for 10 years to become self-sustaining new innovation centers. Such an initiative would not only bring significant economic opportunity to more parts of the nation, but also boost U.S. competitiveness on the global stage.
"Potential growth centers" sounds promising.
Don’t Blame Tech Bros for the Housing Crisis
If there is demand for housing, we would expect people to be finding land and building housing unless there are policies that prevent this (and/or long commutes that people don't want to suffer) or higher-value opportunities.
If the city wanted residential areas (over commercial tax revenue giants), the city should have zoned residential.
The people elect city leaders. The people all want affordable housing.
With $4.5b from corporations and nowhere to build but out or up, high rise residential is the most likely outcome. (Which is typical for dense urban areas that have prioritized and attracted corporate tax revenue over affordable housing)
... Effing scooter bros with their scooters and their gold rush money and their tiny houses.
[Edit: more than] One company says "I will pay you $10,000 to leave the Bay Area / Silicon Valley" Because there's a lot of tech talent (because universities and opportunities) but ridiculously high expenses.
What an effectual headline from NY.
Docker is just static linking for millenials
No, LXC does quite a bit more than static linking. An inability to recognize that likely has nothing to do with generation.
Can you launch a process in a chroot, with cgroups? Okay, now upgrade everything it's linked with (without breaking the host/build system)
Configure a host-only network for a few processes – running in separate cgroups – without DHCP.
Criticize Docker? Rootless builds and containers are essentially impossible. Buildah and podman make rootless builds possible without a socket. Like sysvinit, though, IDK how well centralized logging (and logshipping, and logged crashes and restarts) works without that socket.
Given comments like this, it's likely that you've never built a chroot for a different distro. Or launchd a process with cgroups.
Show HN: Bamboolib – A GUI for Pandas (Python Data Science)
This looks excellent. The ability to generate the Python code for the pandas dataframe transformations looks to be more useful than OpenRefine, TBH.
How much work would it be to use Dask (and Dask-ML) as a backend?
I see the OneHotEncoder button. Have you considered integration with Yellowbrick? They've probably already implemented a few of your near-future and someday roadmap items involving hyperparameter selection and model selection and visualization? https://www.scikit-yb.org/en/latest/
This video shows more of the advanced bamboolib features: https://youtu.be/I0a58h1OCcg
The live histogram rebinning looks useful. Recently I read about a 'shadowgram' / ~KDE approach with very many possible bin widths translucently overlaid in one chart. https://stats.stackexchange.com/questions/68999/how-to-smear...
Yellowbrick also has a bin width optimization visualization in yellowbrick.target.binning.BalancedBinningReference: https://www.scikit-yb.org/en/latest/api/target/binning.html
Great work.
Thank you for your feedback and support :) Are you currently using OpenRefine?
We are currently thinking about providing other dataframe libraries like dask or pyspark and similar. However, we are a little bit unsure on how to make sure that there is user demand before we implement it. It is not a complete rewrite but it would require some additional abstractions at some points in the library. And we need to check if some features might not be available any more. Would dask support be a reason to buy for you?
Great hint with yellowbrick and yes, we are considering some of those features as well if there is a useful place in the library.
In general, we are also thinking about ways how you can extend the library for yourself so that you can add your own analyses/charts of choice and then they will come up again the right point in time. In case that this is useful.
In the past, I've looked at OpenRefine and Jupyter integration. Once I've learned to do data transformation with pandas and sklearn with code, I'll report back to you.
Pandas-profiling has a number of cool descriptive statistics features as well. https://github.com/pandas-profiling/pandas-profiling
There's a new IterativeImputer in Scikit-learn 0.22 that it'd be cool to see visualizations of. https://twitter.com/TedPetrou/status/1197150813707108352 https://scikit-learn.org/stable/modules/impute.html
A plugin model would be cool; though configuring the container every time wouldn't be fun. Some ideas about how we could create a desktop version of binderhub in order to launch REES-compatible environments on our own resources: https://github.com/westurner/nbhandler/issues/1
Dask has only a subset of Pandas available.
Could you send me a link to the docs where they say which ones are not included in Pandas? Would love to take a closer look at his.
Set difference and/or intersection of dir(pd.DataFrame) and dir(dask.DataFrame) with inspect.getargspec and inspect.doc would be a useful document for either or both projects.
pyfilemods generates a ReStructuredText document with introspected API comparisons. "Identify and compare Python file functions/methods and attributes from os, os.path, shutil, pathlib, and path.py" https://github.com/westurner/pyfilemods
Battery-Electric Heavy-Duty Equipment: It's Sort of Like a Cybertruck
> They’ve created a single platform that can be easily modified to do any number of jobs. For instance, their flagship product, the Dannar 4.00, can accept over 250 attachments from CAT, John Deere, or Bobcat. […] Having interoperability with so many different types of equipment, one platform can easily perform many tasks over the course of a year. This is a huge win for cash strapped municipalities. Why would a company or municipality opt to have a backhoe parked all winter long when it could be doing another job?
Does it have regenerative brakes?
Tools for turning descriptions into diagrams: text-to-picture resources
There's a sphinx extension (sphinxcontrib-sdedit) [1] for sdedit ("Quick Sequence Diagram Editor") which does require Java [2].
CSR: Corporate Social Responsibility
> Proponents argue that corporations increase long-term profits by operating with a CSR perspective, while critics argue that CSR distracts from businesses' economic role.
... The 3 Pillars of Corporate Sustainability: Environmental, Social, Economic https://www.investopedia.com/articles/investing/100515/three...
Three dimensions of sustainability: (Environment (Society (Economy))) https://en.wikipedia.org/wiki/Sustainability#Three_dimension...
What are some of the corporate sustainability reporting standards?
How can I score a candidate portfolio with sustainability metrics in order to impact invest with maximum impact?
> What are some of the corporate sustainability reporting standards?
From https://en.wikipedia.org/wiki/Sustainability_reporting#Initi... :
>> Organizations can improve their sustainability performance by measuring (EthicalQuote (CEQ)), monitoring and reporting on it, helping them have a positive impact on society, the economy, and a sustainable future. The key drivers for the quality of sustainability reports are the guidelines of the Global Reporting Initiative (GRI),[3] (ACCA) award schemes or rankings. The GRI Sustainability Reporting Guidelines enable all organizations worldwide to assess their sustainability performance and disclose the results in a similar way to financial reporting.[4] The largest database of corporate sustainability reports can be found on the website of the United Nations Global Compact initiative.
The GRI (Global Reporting Initiative) Standards are now aligned with the UN Sustainable Development Goals (#GlobalGoals). https://en.wikipedia.org/wiki/Global_Reporting_Initiative
>> In 2017, 63 percent of the largest 100 companies (N100), and 75 percent of the Global Fortune 250 (G250) reported applying the GRI reporting framework.[3]
> How can I score a candidate portfolio with sustainability metrics in order to impact invest with maximum impact?
Does anybody have solutions for this? AFAIU, existing cleantech funds are more hand-picked than screened according to sustainability fundamentals.
GTD Tickler file – a proposal for text file format
Taskwarrior is also built upon the todo.txt format. [1]
Taskw supports various task dates – { due: scheduled: wait: until: recur: } [2]
Taskw supports various named dates like soq/eocq, som/eom (start/end of [current] quarter, start/end of month), tomorrow, later [3]
Taskw recurring tasks (recur:) use the duration syntax: weekly/wk/w, monthly/mo, quarterly/qtr, yearly/yr, … [4]
Pandas has a "date offset" "frequency string" microsyntax that supports business days, quarters, and years; e.g. BQuarterEnd, BQuarterBegin [5]
IDK how usable by other tools these date string parsers are.
W/ just a text editor, having `todo.txt`, `daily.todo.txt`, and `weekly.todo.txt` (and `cleanhome.todo.txt` and `hygiene.todo.txt` with "## heading" tasks that get lost @where +sorting) works okay.
I have physical 43 folders, too: A 12 month and a 31 day expanding file. [6]
[2] https://taskwarrior.org/docs/using_dates.html
[3] https://taskwarrior.org/docs/named_dates.html
[4] https://taskwarrior.org/docs/durations.html
[5] https://pandas.pydata.org/pandas-docs/stable/user_guide/time...
Ask HN: Any suggestion on how to test CLI applications?
Hello HN!
I've been looking at alternatives on how to test command line applications, specifically, for example, exit codes, output messages and whatnot. I've seen "bats" https://github.com/sstephenson/bats and Bazel for testing but I'm curious as what other tools people use in a day to day basis. UI testing is nice with tools like Cypress.io and maybe there's something out there that isn't as popular but it's useful.
Thoughts?
pytest-docker-pexpect: https://github.com/nvbn/pytest-docker-pexpect
Pexpect: https://pexpect.readthedocs.io/en/stable/
pytest with subprocess.popen (or Sarge) may be sufficient for checking return codes and checking stdout and stderr output streams. Pytest has tmp_path and tmpdir fixtures that provide less test isolation than Docker containers: http://doc.pytest.org/en/latest/tmpdir.html
sarge.Capture.expect() takes a regex and returns None if there's no match: https://sarge.readthedocs.io/en/latest/tutorial.html#looking...
The Golden Butterfly and the All Weather Portfolio
The Golden Butterfly (is a modified All Weather Portfolio)
> Stocks: 20% Domestic Large Cap Fund (Vanguard’s VTI or Goldman Sach’s JUST), 20% Domestic Small Cap Value (Vanguard’s VBR)
> Bonds: 20% Long Term (Vanguard’s BLV), 20% Short Term (Vanguard’s BSV)
> Real Assets: 20% Gold (SPDR’s GLD)
The All Weather Portfolio:
> Stocks: 30% Domestic Total Stock Market (VG total stock)
> Bonds: 40% Long Term, 15% Intermediate-Term
> Real Assets: 7.5% Commodities, 7.5% Gold
Canada's Decision To Make Public More Clinical Trial Data Puts Pressure On FDA
We really could get more out of this data through international collaboration and through linked data (e.g. URIs for columns). See: "Open, and Linked, FDA data" https://github.com/FDA/openfda/issues/5#issuecomment-5392966... and "ENH: Adverse Event Count / 'Use' Count Heatmap" https://github.com/FDA/openfda/issues/49
With sales/usage counts, we'd have a denominator with which we could calculate relative hazard.
Python Alternative to Docker
Shiv does not solve for what containers and Docker/Podman/Buildah/Containerd solve for: re-launching processes at boot and failure, launching processes in chroots or cgroups (with least privileges), limiting access to network ports, limiting access to the host filesystem, building chroots / images, [...]
You can run build tools like shiv with a RUN instruction in a Dockerfile and get some caching.
You can build a zipapp with shiv (in a build container) and run the zipapp in a container.
Should the zipapp contain the test suite(s) and test_requires so that the tests can be run in an environment most similar to production?
It's much easier to develop with code on the filesystem (instead of in a zipapp).
It's definitely faster to read the whole zipapp into RAM than to stat and read each imported module from the filesystem once at startup.
There may be a better post title than the current "Python Alternative to Docker"? Shiv is a packaging utility for building Python zipapps. Shiv is not an alternative to process isolation with containers (or VMs)
$6B United Nations Agency Launches Bitcoin, Ethereum Crypto Fund
"UNICEF launches Cryptocurrency Fund: UN Children’s agency becomes first UN Organization to hold and make transactions in cryptocurrency" https://www.unicef.org/press-releases/unicef-launches-crypto...
From https://www.unicefusa.org/ :
> UNICEF USA helps save and protect the world's most vulnerable children. UNICEF USA is rated one of the best charities to donate to: 89% of every dollar spent goes directly to help children.
Timsort, the Python sorting algorithm
Here are the Python 3 docs for sorting [1], in-place list.sort() [2], and sorted() [3] (which makes a sorted copy of the references). And the Timsort Wikipedia page [4].
[1] https://docs.python.org/3/howto/sorting.html#sort-stability-...
[2] https://docs.python.org/3/library/stdtypes.html#list.sort
Supreme Court allows blind people to sue retailers if websites aren't accessible
"a11y": Accessibility
https://a11yproject.com/ has patterns, a checklist for checking web accessibility, resources, and events.
awesome-a11y has a list of a number of great resources for developing accessible applications: https://github.com/brunopulis/awesome-a11y
In terms of W3C specifications [1], you've got: WAI-ARIA (Web Accessibility Initiative: Accessibile Rich Internet Applications) [2], and WCAG: Web Content Accessibility Guidelines [3]. The new W3C Payment Request API [4] makes it easy for browsers to offer a standard (and probably(?) already accessible) interface for the payment data entry screen, at least.
There are a number of automated accessibility testing platforms. "[W3C WAI] Web Accessibility Evaluation Tools List" [5] lists quite a few. Can someone recommend a good accessibility testing tools? Is Google Lighthouse (now included with Chrome Devtools and as a standalone script) a good tool for accessibility reviews?
[1] https://github.com/brunopulis/awesome-a11y/blob/master/topic...
[2] https://www.w3.org/TR/using-aria/
[3] https://www.w3.org/WAI/standards-guidelines/wcag/
Streamlit: Turn a Python script into an interactive data analysis tool
Cool!
requests_cache caches HTML requests into one SQLite database. [1] pandas-datareader can cache external data requests with requests-cache. [2]
dask.cache can do opportunistic caching (of 2GB of data). [3]
How does streamlit compare to jupyter voila dashboards (with widgets and callbacks)? They just launched a new separate github org for the project. [4] There's a gallery of voila dashboard examples. [5]
> Voila serves live Jupyter notebooks including Jupyter interactive widgets.
> Unlike the usual HTML-converted notebooks, each user connecting to the Voila tornado application gets a dedicated Jupyter kernel which can execute the callbacks to changes in Jupyter interactive widgets.
> - By default, voila disallows execute requests from the front-end, preventing execution of arbitrary code.
[1] https://github.com/reclosedev/requests-cache
[2] https://pandas-datareader.readthedocs.io/en/latest/cache.htm...
[3] https://docs.dask.org/en/latest/caching.html
[4] https://github.com/voila-dashboards/voila
[5] https://blog.jupyter.org/a-gallery-of-voil%C3%A0-examples-a2...
Acess control and resource exhaustion are challenges with building any {Flask, framework_x,} app [from Jupyter notebooks]. First it's "HTTP Digest authentication should be enough for now"; then it's "let's use SSO and LDAP" (and review every release); then it's "why is it so sloww?". JupyterHub has authentication backends, spawners, and per-user-container/vm resource limits.
> Each user on your JupyterHub gets a slice of memory and CPU to use. There are two ways to specify how much users get to use: resource guarantees and resource limits. [6]
[6] https://zero-to-jupyterhub.readthedocs.io/en/latest/user-res...
Some notes re: voila and JupyterHub:
> The reason for having a single instance running voila only is to allow non JupyterHub users to have access to the dashboards. So without going through the Hub auth flow.
> What are the requirements in your case? Voila can be installed in the single user Docker image, so that each user can also use it on their own server (as a server extension for example). [7]
Scott’s Supreme Quantum Supremacy FAQ
Who even asked these questions?
I question this. All of this.
Literally people on HN have been asking many of these questions for years whenever QC is discussed.
Ask HN: How do you handle/maintain local Python environments?
I'm having some trouble figuring out how to handle my local Python. I'm not asking about 2 vs 3 - that ship has sailed - I'm confused on which binary to be using. From the way I see it, there's at least 4 different Pythons I could be using:
1 - Python shipped with OS X/Ubuntu
2 - brew/apt install python
3 - Anaconda
4 - Getting Python from https://www.python.org/downloads/
And that's before getting into how you get numpy et al installed. What's the general consensus on which to use? It seems like the OS X default is compiled with Clang while brew's version is with GCC. I've been working through this book [1] and found this thread [2]. I really want to make sure I'm using fast/optimized linear algebra libraries, is there an easy way to make sure? I use Python for learning data science/bioinformatics, learning MicroPython for embedded, and general automation stuff - is it possible to have one environment that performs well for all of these?
[1] https://www.amazon.com/Python-Data-Analysis-Wrangling-IPython/dp/1449319793
[2] https://www.reddit.com/r/Python/comments/46r8u0/numpylinalgsolve_is_6x_faster_on_my_mac_than_on/
I just use Anaconda. It's basically the final word in monolithic Python distributions.
For data science/numerical computation, all batteries included. It also has fast optimized linear algebra (MKL) plus extras like dask (paralellization), numba, out of the box. No fuss no muss. No need to fiddle with anything.
Everything else is a "pip install" or "conda install" away. Virtual envs? Has it. Run different Python versions on the same machine via conda environments? Has it. Web dev with Django etc.? All there. Need to containerize? miniconda.
The only downside? It's quite big and takes a while to install. But it's a one time cost.
I also prefer conda for the same reasons.
Precompiled MKL is really nice. Conda and conda-forge now build for aarch64. There are very few wheels for aarch64 on PyPI. Conda can install things like Qt (IPython-qt, spyder,) and NodeJS (JupyterLab extensions).
If I want to switch python versions for a given condaenv (instead of just creating a new condaenv for a different CPython/PyPy version), I can just run e.g. `conda install -y python=3.7` and it'll reinstall everything in the depgraph that depended on the previous python version.
I always just install miniconda instead of the whole anaconda distribution. I always create condaenvs (and avoid installing anything in the root condaenv) so that I can `conda-env export -f environment.yml` and clean that up.
BinderHub ( https://mybinder.org/ ) creates docker containers from {git repos, Zenodo, FigShare,} and launches them in free cloud instances also running JupyterLab by building containers with repo2docker (with REES (Reproducible Execution Environment Specification)). This means that all I have to do is add an environment.yml to my git repo in order to get Binder support so that people can just click on the badge in the README to launch JupyterLab with all of the dependencies installed.
REES supports a number of dependency specifications: requirements.txt, Pipfile.lock, environment.yml, aptSources, postBuild. With an environment.yml, I can install the necessary CPython/PyPy version and everything else.
...
In my dotfiles, I have a setup_miniconda.sh script that installs miniconda into per-CPython-version CONDA_ROOT and then creates a CONDA_ENVS_PATH for the condaenvs. It may be overkill because I could just specify a different python version for all of the conda envs in one CONDA_ENVS_PATH, but it keeps things relatively organized and easily diffable: CONDA_ROOT="~/-wrk/-conda37" CONDA_ENVS_PATH="~/-wrk/-ce37"
I run `_setup_conda 37; workon_conda|wec dotfiles` to work on the ~/-wrk/-ce37/dotfiles condaenv and set _WRD=~/-wrk/-ce37/dotfiles/src/dotfiles.
Similarly, for virtualenvwrapper virtualenvs, I run `WORKON_HOME=~/-wrk/-ve37 workon|we dotfiles` to set all of the venv cdaliases; i.e. then _WRD="~/-wrk/-ve37/dotfiles/src/dotfiles" and I can just type `cdwrd|cdw` to cd to the working directory. (Some of the other cdaliases are: {cdwrk, cdve|cdce, cdvirtualenv|cdv, cdsrc|cds}. So far, I have implemented cdalias support for bash, IPython, and vim)
One nice thing about defining _WRD is I can run `makew <tab>` and `gitw` to `cd $_WRD; make <tab>` and `git -C $_WRD` without having to change directory and then `cd -` to return to where I was.
So, for development, I use a combination of virtualenvwrapper, pipsi, conda, and some shell scripts in my dotfiles that I should get around to releasing and maintaining someday. https://westurner.github.io/dotfiles/venv
For publishing projects, I like environment.yml because of the REES support.
Is the era of the $100 graphing calculator coming to an end?
For $100, you can buy a Pinebook with an 11" or 14" screen, a multitouch trackpad, gigabytes of storage, WiFi, a keyboard without a numpad, and an ARM processor.
On this machine, you can create reproducible analyses with JupyterLab; do arithmetic with Python; work with multidimensional arrays with NumPy, SciPy, Pandas, xarray, Dask; do machine learning with Statsmodels, Scikit-learn, Dask-ML, TPOT; create books of these notebooks (containing code and notes (in Markdown, which easily transformed to HTML) and LaTeX equations) with jupyter-book, nbsphinx, git + BinderHub; store the revision history of your discoveries; publish what you've discovered and learned to public or private git repositories; and complete graded exercises with nbgrader.
But the task is to prepare for a world of mental arithmetic, no validation, no tests, no reference materials, and no search engines; and CAS (Computer Algebra Systems) systems like SymPy and Sage are not allowed.
On this machine, you can run write code, write papers, build spreadsheets and/or Jupyter notebooks, run physical simulations, explore the stars, and play games and watch videos. Videos like: Khan Academy videos and exercises that you can watch and do, with validation, until you've achieved mastery and move on to the next task on your todo.txt list.
But the task is to preserve your creativity and natural curiosity despite the compulsory education system's demands for quality control and allocative efficiency; in an environment where drama and popularity are the solutions to relatedness and acceptance needs.
I have three of these $100 calculators in my toolbox. It's been so long since I've powered them on that I'm concerned that the rechargeable AAA batteries are leaking battery acid.
For $100, you can buy an ARM notebook and install conda and conda-forge packages and build sweet visualizations to collaborate with colleagues on (with Seaborn (matplotlib), HoloViews, Altair, Plotly)
"You must buy a $100 calculator that only runs BASIC and ASM, and only use it for arithmetic so that we can measure you."
Hand tools are fun, but please don't waste any more of my compulsory time.
I would LOVE to live in a world where Jupyter notebooks are the defacto standard for post-primary education. It would take some vision and leadership to bring about that sort of change, but would far better prepare kids for the world they will be running one day.
Reinventing Home Directories
Full Disclosure: I am violently opposed to systemd and pulseaudio (before it was taken away from Poettering).
However, I think I'm seeing a pattern, Poettering identifies real issues and actually tries to fix them.
But it seems like his ambition is impeding his ability to consider those who do not want to approach the fix his way. It's never a measured approach, it /feels/ like a "lets write it now and figure out all the problems down the line and fix them" which, for core pieces of software is a dangerous mindset.
After watching the intro here I agree with him, the current state of things is not ideal, and nobody _wants_ to touch user account management on Linux. On MacOS for example, it's much more thought out, and I think Linux could do it better.
As much as I hate poetterings specific approaches (and, subsequent lockout) I have to give credit to him identifying these issues and actually trying to fix them, it's more than I do. Then again I am a lowly sysadmin type.
I like the tone of your comment, unlike many others here. But to your remark:
"But it seems like his ambition is impeding his ability to consider those who do not want to approach the fix his way. It's never a measured approach, it /feels/ like a "lets write it now and figure out all the problems down the line and fix them" which, for core pieces of software is a dangerous mindset."
I'd like to say that he forces nobody and so we should not complain about his creative endeavors. The market will decide, and you, will decide.
I disagree, you can argue the semantics of if it's him specifically or if it's another project that decides it will only target systemd/pulseaudio. But the fact that if I want a stable distro for server usage in 2019 I _must_ use a systemd distro speaks to the fact that I am indeed forced to use it.
Maybe that's just adoption though, typically the sysadmin community considers the "safe" options to be CentOS/RHEL or Debian Stable. Both of which default to systemd. (and no sysadmin is going to change the init system on production servers unless it presents some abject nightmare, short of eating 30% CPU/Mem-- it's core software, it wont be touched).
Projects like Devuan are simply not considered safe choices by the majority of the sysadmin community, and there's no alternative for RHEL-esque sites.
So, semantically: it's not poetterings fault. But at the same time, I'm still forced.
Sorry, but you'll need to find a word different from "forced"
What word would you recommend?
If I must do something then I am forced? no? -- and I'm not being facetious I'm genuinely curious.
If I wish to keep using production grade linux distributions (that my company has been using for 15+ years) then I must adopt systemd.
I guess I could convince my 25,000+ person organisation to change everything to a Debian based OS and subsequently move them to Debian, but that's not likely.
Why do you think that is? Why have production grade Linux distributions all chosen to adopt systemd?
With SysV init, how do you securely launch processes in cgroups, such that they'll consistently restart when the process happens to terminate, with stdout and stderr logged with consistent timestamps, with process dependency models that allow for faster boots due to parallelization?
(edit)
Journalctl is far better than `tail -f /var/log/starstar` and parsing all of those timestamps and inconsistently escaped logfile formats. There's no good way to modify everything in /etc/init.d in order to log to syslog-ng or rsyslog. Systemd and journalctl solve for that; for unified logging.
IMO, there's no question that systemd is the better way and I have zero nostalgia for spawning everything from a probably a shell specified in an /etc/init.d shebang without process restarting (and logging thereof), cgroups, and consistent logging.
> Why do you think that is? Why have production grade Linux distributions all chosen to adopt systemd?
Because RH funds a lot of software development, and Poettering works for them.
> Journalctl is far better than `tail -f /var/log/starstar`
Except when journald shits the bed and corrupts its own log file. Or when you want ship logs off-machine, yet journald does not support any standards-compliant transport so you end up have to log a second logging daemon (e.g., rsyslog)—in which case what's the damn point of the first? Or when you do a "service mysql start" and it dies, but there is no output anywhere of what went wrong, and you have to guess that you have a typo in your "my.cnf".
But other than that Mrs. Lincoln, how was the play?
> ... in an /etc/init.d shebang without process restarting (and logging thereof), cgroups, and consistent logging.
The problem is not systemd-as-init-replacement. The problem is systemd-as-kitchen-sink. udevd is an "essential" part of systemd? Really? It can't be it's own stand-alone project? Really?
When journald logfile corruption occurs, it's detected and it starts writing a new logfile.
When flatfile logfile corruption occurs, it's not detected and there are multiple logfile formats to contend with. And multiple haphazard logrotate configs.
Here's how to use a separate process to ship journald logs - from one file handle - to a remote logging service: https://unix.stackexchange.com/questions/394822/how-should-i...
While there is a systemd-journal-remote, it's not necessary for journald to try and replicate what's already solved and tested in rsyslog and syslog-ng.
It's quite a bit more work to add every new service to the syslog-ng or rsyslog configuration than to just ship one journald log.
Furthermore, service start/stop events are already in the same stream (with the same timestamp format) with the services' stdout and stderr.
Why hasn't anyone written fsck for corrupted journald recovery?
...
I have not needed to makedev and chown and chatted and chcon anything in very many years. When you accidentally newbishly delete something from a static /dev and rebooting doesn't work and you have no idea what the major minor is or was, it sucks bad.
When you're trying to boot a system on a different machine but it doesn't work because the NIC is in a different bus, it's really annoying to have to symlink /dev or modify /etc. With udevd, all you need to do is define a rule to map the busid device name to e.g. eth0. I can remember encountering the devfs race condition resulting in eth0 and eth1 being mapped to different devices on different boots; which was dangerous because firewall rules are applied to device names.
Udev has been in the kernel since 2.6.
"What problems does udev actually solve?" https://superuser.com/questions/686774/what-problems-does-ud...
With integrated udev and systemd, I have no reason to run a separate hotplugd with a different config format (again with no cgroup support) and a different logstream.
Perhaps ironically, here's a link to the presentation PDF that was posted yesterday: https://news.ycombinator.com/item?id=21036020
And my comments there:
> What a good idea.
> Here's the hyperlinkified link to the {systemd-homed.service, systemd-userdbd.service, homectl, userdbctl} sources from the PDF: https://github.com/poettering/systemd/tree/homed
> Hadn't heard of varlink: https://varlink.org/
> Is there a FIPS-like subset of the most-widely-available LUKS configs? Otherwise home directories won't work on systems that have a limited set of LUKS modules.
Serverless: slower and more expensive
It'd be interesting to see how much this same workload would cost with e.g. OpenFaaS on k8s with autoscaling to zero; but there also you'd need to include maintenance costs like OS and FaaS stack upgrades. https://docs.openfaas.com/architecture/autoscaling/
Entropy can be used to understand systems
Maximum entropy: https://en.wikipedia.org/wiki/Maximum_entropy
Here's a quote of a tweet about a (my own): comment on a schema:BlogPost: https://twitter.com/westurner/status/1048125281146421249:
> “When Bayes, Ockham, and Shannon come together to define machine learning” https://towardsdatascience.com/when-bayes-ockham-and-shannon...
> Comment: "How does this relate to the Principle of Maximum Entropy? How does Minimum Description Length relate to Kolmogorov Complexity?"
Thanks for sharing! Despite the fact that Shannon's "A Mathematical Theory of Communication" is so accessible, I find that most in our field (stats/ML) don't often think through information-theoretic tools in a "first principles way."
Yes, KL divergences show up everywhere, but they are not derived from scratch often enough. Maybe I'm stifled by my campus bubble though :)
New Query Language for Graph Databases to Become International Standard
Graph query languages are nice and all, but what about Linked Data here? Queries of schemaless graphs miss lots of data because without a schema this graph calls it "color" and that graph calls it "colour" and that graph calls it "色" or "カラー". (Of course this is also an issue even when there is a defined schema; but it's hardly possible to just happen to have comprehensible inter or even intra-organizational cohesion without e.g. RDFS and/or OWL and/or SHACL for describing (and changing) the shape of the data)
So, the task is then to compile schema-aware SPARQL to GQL or GraphQL or SQL or interminable recursive SQL queries or whatever it is.
For GraphQL, there's GraphQL-LD (which somewhat unfortunately contains a hashtag-indeterminate dash). I cite this in full here because it's very relevant to the GQL task at hand:
"GraphQL-LD: Linked Data Querying with GraphQL" (2018) https://comunica.github.io/Article-ISWC2018-Demo-GraphQlLD/
> GraphQL is a query language that has proven to be a popular among developers. In 2015, the GraphQL framework [3] was introduced by Facebook as an alternative way of querying data through interfaces. Since then, GraphQL has been gaining increasing attention among developers, partly due to its simplicity in usage, and its large collection of supporting tools. One major disadvantage of GraphQL compared to SPARQL is the fact that it has no notion of semantics, i.e., it requires an interface-specific schema. This therefore makes it difficult to combine GraphQL data that originates from different sources. This is then further complicated by the fact that GraphQL has no notion of global identifiers, which is possible in RDF through the use of URIs. Furthermore, GraphQL is however not as expressive as SPARQL, as GraphQL queries represent trees [4], and not full graphs as in SPARQL.
> In this work, we introduce GraphQL-LD, an approach for extending GraphQL queries with a JSON-LD context [5], so that they can be used to evaluate queries over RDF data. This results in a query language that is less expressive than SPARQL, but can still achieve many of the typical data retrieval tasks in applications. Our approach consists of an algorithm that translates GraphQL-LD queries to SPARQL algebra [6]. This allows such queries to be used as an alternative input to SPARQL engines, and thereby opens up the world of RDF data to the large amount of people that already know GraphQL. Furthermore, results can be translated into the GraphQL-prescribed shapes. The only additional requirement is their queries would now also need a JSON-LD context, which could be provided by external domain experts.
> In related work, HyperGraphQL [7] was introduced as a way to expose access to RDF sources through GraphQL queries and emit results as JSON-LD. The difference with our approach is that HyperGraphQL requires a service to be set up that acts as a intermediary between the GraphQL client and the RDF sources. Instead, our approach enables agents to directly query RDF sources by translating GraphQL queries client-side.
All of these RDFS vocabularies and OWL ontologies provide structure that minimizes the costs of merging and/or querying multiple datasets: https://lov.linkeddata.es/dataset/lov/
All of these schema.org/Dataset s in the "Linked Open Data Cloud" are easier to query than a schemaless graph: https://lod-cloud.net/ . Though one can query schemaless graphs with SPARQL, as well.
For reference, RDFLib has a bunch of RDF graph implementations over various key/value and SQL store backends. RDFLib-sqlachemy does query parametrization correctly in order to minimize the risk of query injection. FOR THE RECORD, SQL Injection is the CWE Top 25 #1 most prevalent security weakness; which is something that any new spec and implementation should really consider before launching anything other than an e.g. overly-verbose JSON-based query language that people end up bolting a micro-DSL onto. https://github.com/RDFLib/rdflib-sqlalchemy
Most practically, I frequently want to read a graph of objects into RAM; update, extend, and interlink; and then transactionally save the delta back to the store. This requires a few things: (1) an efficient binary serialization protocol like Apache Arrow (SIMD), Parquet, or any of the BSON binary JSONs; (2) a transactional local store that can be manually synchronized with the remote store until it's consistent.
SPARQL Update was somewhat of an out-of-scope afterthought. Here's SPARQL 1.1 Update: https://www.w3.org/TR/sparql11-update/
Here's SOLID, which could be implemented with SPARQL on GQL, too; though all the re-serialization really shouldn't be necessary for EAV triples with a named graph URI identifier: https://solidproject.org/
5 star data: PDF -> XLS -> CSV -> RDF (GQL, AFAIU (but with no URIs(!?))) -> LOD https://5stardata.info/en/
Linked Data tends to live in a semantic web world that has a lot of open world assumptions. While there are a few systems like this out there, there aren't many. More practically focused systems collapse this worldview down into a much simpler model, and property graphs suit just fine.
There's nothing wrong with enabling linked data use cases, but you don't need RDF+SPARQL+OWL and the like to do that.
The "semantic web stack" I think has been shown by time and implementation experience to be an elegant set of standards and solutions for problems that very few real world systems want to tackle. In the intervening 2 full generations of tech development that have happened since a lot of those standards were born, some of the underlying stuff too (most particularly XML and XML-NS) went from indispensable to just plain irritating.
> Linked Data tends to live in a semantic web world that has a lot of open world assumptions. While there are a few systems like this out there, there aren't many. More practically focused systems collapse this worldview down into a much simpler model, and property graphs suit just fine.
Data integration is cost prohibitive. In n years time, the task is "Let's move all of these data silos into a data lake housed in our singular data warehouse; and then synchronize and also copy data around to efficiently query it in one form or another"
Linked data enables data integration from day one: enables the linking of tragically-silo'd records within disparate databases
There are very very many systems that share linked data. Some only label some of the properties with URIs in templates. Some enable federated online querying.
When you develop a schema for only one application implementation, you're tragically limiting the future value of the data.
> There's nothing wrong with enabling linked data use cases, but you don't need RDF+SPARQL+OWL and the like to do that.
Can you name a property graph use case that cannot be solved with RDFS and SPARQL?
> The "semantic web stack" I think has been shown by time and implementation experience to be an elegant set of standards and solutions for problems that very few real world systems want to tackle.
TBH, I think the problem is that people don't understand the value in linking our data silos through URIs; and so they don't take the time to learn RDFS or JSON-LD (which is pretty simple and useful for very important things like SEO: search engine result cards come from linked data embedded in HTML attributes (RDFa, Microdata) or JSON-LD)
The action buttons to 'RSVP', 'Track Package', anf 'View Issue' on Gmail emails are schema.org JSON-LD.
Applications can use linked data in any part of the stack: the database, the messages on the message queue, in the UI.
You might take a look at all of the use cases that SOLID solves for and realize how much unnecessary re-work has gone into indexing structs and forms validation. These are all the same app with UIs for interlinked subclasses of https://schema.org/Thing with unique inferred properties and aggregations thereof.
> In the intervening 2 full generations of tech development that have happened since a lot of those standards were born, some of the underlying stuff too (most particularly XML and XML-NS) went from indispensable to just plain irritating.
Without XSD, for example, we have no portable way to share complex fractions.
There's a compact representation of JSON-LD that minimizes record schema overhead (which gzip or lzma generally handle anyway)
https://lod-cloud.net is not a trivial or insignificant amount of linked data: there's real value in structuring property graphs with standard semantics.
Are our brains URI-labeled graphs? Nope, and we spend a ton of time talking to share data. Eventually, it's "well let's just get a spreadsheet and define some columns" for these property graph objects. And then, the other teams' spreadsheets have very similar columns with different labels and no portable datatypes (instead of URIs)
> Can you name a property graph use case that cannot be solved with RDFS and SPARQL?
No - that's not the point. Of course you can do it with RDFS + SPARQL. For that matter you could do it with redis. Fully beside the point.
What's important is what the more fluent and easy way to do things is. People vote with their feet, and property graphs are demonstrably easier to work with for most use cases.
“Easier” is completely subjective, no way you can demonstrate that.
RDF solves a much larger problem than just graph data model and query. It addresses data interchange on the web scale, using URIs, zero-cost merge, Linked Data etc.
> “Easier” is completely subjective, no way you can demonstrate that.
I agree it's subjective. While there's no exact measurement for this sort of thing, the proxy measure people usually use is adoption; and if you look into for example Cypher vs. SPARQL adoption, Neo4j vs. RDF store adoption, people are basically voting with their feet.
From my personal experiences developing software with both, I've found property graphs much simpler and a better map for how people think of data.
It's true that RDF tries to solve data interchange on the web scale. That's what it was designed for. But the original design vision, in my view, hasn't come to fruition. There are bits and pieces that have been adopted to great effect (things like RDF microformats for tagging HTML docs) but nothing like what the vision was.
What was the vision?
The RDFJS "Comparison of RDFJS libraries" wiki page lists a number of implementations; though none for React or AngularJS yet, unfortunately. https://www.w3.org/community/rdfjs/wiki/Comparison_of_RDFJS_...
There's extra work to build general purpose frameworks for Linked Data. It may have been hard for any firm with limited resources to justify doing it the harder way (for collective returns)
Dokieli (SOLID (LDP,), WebID, W3C Web Annotations,) is a pretty cool - if deceptively simple-looking - showcase of what's possible with Linked Data; it just needs some CSS and a revenue model to pay for moderation. https://dokie.li/
> property graphs are demonstrably easier to work with for most use cases.
How do you see property graphs as distinct from RDF?
People build terrible apps without schema or validation and leave others to clean that up.
> How do you see property graphs as distinct from RDF?
This is the full answer: https://stackoverflow.com/a/30167732/2920686
I added an answer in context to the comments on the answer you've linked but didn't add a link from the comments to the answer. Here's that answer:
> (in reply to the comments on this answer: https://stackoverflow.com/a/30167732 )
> When an owl:inverseOf production rule is defined, the inverse property triple is inferred by the reasoner either when adding or updating the store, or when selecting from the store. This is a "materialized relation"
> Schema.org - an RDFS vocabulary - defines, for example, https://schema.org/isPartOf as the inverse property of hasPart. If both are specified, it's not necessary to run another graph pattern query to traverse a directed relation in the other direction. (:book1 schema:hasPart ?o), (?o schema:isPartOf :book1), (?s schema:hasPart :chapter2)
> It's certainly possible to use RDFS and OWL to describe schema for and within neo4j property graphs; but there's no reasoner to e.g. infer inverse properties or do schema validation.
> Is there any RDF graph that neo4j cannot store? RDF has datatypes and languages for objects: you'd need to reify properties where datatypes and/or languages are specified (and you'd be re-implementing well-defined semantics)
> Can every neo4j graph be represented with RDF? Yes.
> RDF is a representation for graphs for which there are very many store implementations that are optimized for various use cases like insert and query performance.
> Comparing neo4j to a particular triplestore (with reasoning support) might be a more useful comparison given that all neo4j graphs can be expressed as RDF.
And then, some time later, I realize that I want/need to: (3) apply production rules to do inference at INSERT/UPDATE/DELETE time or SELECT time (and indicate which properties were inferred (x is a :Shape and a :Square, so x is also a :Rectangle; x is a :Rectangle and :width and :height are defined, so x has an :area)); (4) run triggers (that execute code written in a different language) when data is inserted, updated, modified, or linked to; (5) asynchronously yield streaming results to message queue subscribers who were disconnected when the cached pages were updated
A Python Interpreter Written in Python
What an excellent 500 lines introduction to the byterun bytecode interpreter / virtual machine: https://github.com/nedbat/byterun
Also, proceeds from optional purchases of the AOSA books go to Amnesty International. https://aosabook.org/
Reinventing Home Directories – systemd-homed [pdf]
What a good idea.
Here's the hyperlinkified link to the {systemd-homed.service, systemd-userdbd.service, homectl, userdbctl} sources from the PDF: https://github.com/poettering/systemd/tree/homed
Hadn't heard of varlink: https://varlink.org/
Is there a FIPS-like subset of the most-widely-available LUKS configs? Otherwise home directories won't work on systems that have a limited set of LUKS modules.
Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM
There's also RustPython, a Rust implementation of CPython 3.5+: https://news.ycombinator.com/item?id=20686580
> https://github.com/RustPython/RustPython
Is this basically what Cython and PyPy are trying to do, but with Rust?
PyPy is a JIT compiler. RustPython is an interpreter.
And Cython is an AOT compiler for a superset of Python.
RustPython seems to be modestly aiming for a reimplementation of CPython.
Maybe "dialect" would be more accurate than "superset"? I don't think Cython is technically a superset of Python, since I think runtime metaprogramming features like __dict__ and monkey-patching are significantly altered or restricted?
Cython is a reification of the interpretation of a python program. Ie it converts the python code into the equivalent CPython API calls (which are all in C) thereby allowing the developer to interperse real C code. Anything you could do in python you could technically do in Cython, although it would be much more verbose.
Yeah, I was going to make a similar comment. It's a dialect of CPython, and certainly there are extensions required to make it usable. But I'm not sure it is a strict superset of the full Python language.
Convention wound lean towards calling it RPython
Why?
Craftsmanship–The Alternative to the 4 Hour Work Week
> To be successful over the course of a career requires the application and accumulation of expertise. This assumes that for any given undertaking you either provide expertise or you are just a bystander. It’s the experts that are the drivers — an expertise that is gained from a curiosity, and a mindset of treating one’s craft very seriously.
Solar and Wind Power So Cheap They’re Outgrowing Subsidies
USD$5.2 trillion was spent globally on fossil fuel subsidies in 2017
https://www.imf.org/en/Publications/WP/Issues/2019/05/02/Glo...
"hey fossil fuels, let's 1v1" sincerely, solar
(From Forbes: United States Spend Ten Times More On Fossil Fuel Subsidies Than Education)
A few things:
1) That's not what was spent, it's what this paper projected was spent.
2) I think this paper is defining subsidy in a way that we don't usually use that word. They're calling the costs associated with anthropogenic climate change and pollution "subsidies", most people would just call those things "costs."
Using the word "subsidy" implies that governments are actively taking tax dollars and giving it to fossil fuel companies and consumers. That doesn't appear to be what's going on for the most part.
Look at Figure 4, the bulk of the so-called "subsidies" are "global warming" and "local pollution" those aren't what most people would call subsidies, they're costs.
I'm saying this just to clarify things, personally I am supportive of massive tax increases on carbon and massive subsidies for renewables.
It’s an unconventional way to use “subsidy,” but I think it fits.
Imagine a garbage company. The government pays them so they can buy land where they dump their garbage. Obviously a subsidy.
Now, let’s say the government buys the land themselves then gives it to the company for dumping. No money changes hands but this is still pretty clearly a subsidy.
Instead of giving the company land, the government retains ownership, but lets the company dump there for free. Still a pretty clear subsidy.
Instead of buying the land, the government just takes it. Now no money is involved at all, but it’s still a subsidy.
Instead of taking the land, the government just declares that it’s legal for the garbage company to dump trash on other people’s land, and the owners just have to deal with it. This is quite different in the details from the original subsidy, but the overall effect is essentially the same.
Polluters are in a situation that’s exactly like this last scenario. They get to dump their trash on everyone’s property and don’t have to pay for the privilege. They’re being subsidized in an amount equal to whatever payment it would take to get everyone to willingly accept this trash.
This is wrong because subsidies aren't just externalized costs. They are defined as monetary gifts. You're extending the definition by analogy and arguing that a company externalizing its costs is the same thing as a subsidy, but a subsidy implies direct and deliberate support.
This is important for a couple of reasons: The main one is that it's confusing to redefine subsidy when you can just say "cost" and have people understand you. People reading this survey might see the $5.2T number and assume that that cost is in addition to whatever the cost of climate change is and will have to read the paper to understand otherwise. This is unnecessarily confusing even if one were to grant the logic of it.
In addition, when people discuss subsidies, they are often most interested in government policy. The purpose of a subsidy often is to increase a certain kind of business, so we might worry about unnecessarily funding the fossil fuels and thus encouraging climate change in a manner above and beyond simply allowing them to be used the way we've always done, but that's not quite what's going on here.
One thing to keep in mind is that it's a lot easier to measure a genuine government subsidy than an externality. So the distinction matters in that regard as well. Any measure of the cost of climate change is to at least some degree speculation, whereas any attempt to measure the direct amount of money given to fossil fuel companies can probably be much more exact.
Gifts in kind aren’t subsidies? If the government gives equipment or resources or labor, rather than money, it’s no longer a subsidy? That sure doesn’t fit how I understand the term. What would you call those?
"Gifts in kind aren’t subsidies?"
They can be but that's less common. The key difference is that a subsidy is direct and an active policy.
For example, when people talk about subsidies for renewables, they aren't talking about any externalized costs of manufacture, which do exist, they are talking about direct government gifts and tax breaks deliberately put in place to encourage investment in renewables. When people talk about subsidies given to fossil fuels the same is true, especially when they are being compared to renewables as is the case in this discussion.
Edit: removed word "decision" to clarify my meaning
Then it's a gift in kind and a subsidy.
When I litter, I get fined, but those companies are not when they litter e.g. carbon all over. When I dump chemicals into nature, I get fined for polluting the environment or even imprisoned outright, those companies do not e.g. when it's in the form of "emissions". Cars in my country get taxed directly or indirectly (through fuel) based in part on emissions, while a lot of commercial vehicles and fuels for these vehikles get a reduced rate or even excluded from taxation. But cars here are taxed far less so than in other European countries. etc.
The governments are clearly aware that pollution and dumping your garbage are things you should not do or at least minimize. They made laws against it, but actively decided to exclude certain business sectors and/or certain types of pollution, or actively decided not to regulate or tax certain types of pollution while regulating/taxing others.
"Cars in my country get taxed directly or indirectly (through fuel) based in part on emissions"
Unless these emmission taxes are calibrated to the cost of climate change then this argument is missing the point. Taxes and subsidies are often instituted in response to negative and positive externalities, but that doesn't change the fact that they are different things. This is important when trying to draw policy comparisons which is our situation here.
You seem to want to argue that subsidizing a business and not taxing them on an externality is somehow morally the same thing, and that's an entirely different discussion, but it doesn't mean that they are factually the same thing. There is a practical difference in terms of how things are measured and how policies are compared across industries and governments and that difference matters.
>Unless these emmission taxes are calibrated to the cost of climate change then this argument is missing the point.
The stated policy goal in part is to reduce emissions to met climate targets to fight climate change, so yes.
>You seem to want to argue that subsidizing a business and not taxing them on an externality is somehow morally the same thing
It is.
>but it doesn't mean that they are factually the same thing.
First of all, the meaning of words and political concepts are never factual.
But I'd still argue that at a high level they are the same, and both are subsidies. In both cases the government refuses money it would otherwise collect from different parties, thereby gifting those entities value you can put a price tag on.
Those decisions are active decisions NOT to do something (while doing something about the same thing or very similar things when it comes to other parts of the population), at least at this point.
The only distinction I'd make is between direct subsidies (the government forks over money) and indirect/implicit subsidies (the government decides not to make certain entities pay for certain things for which other entities have to pay the government).
">but it doesn't mean that they are factually the same thing.
First of all, the meaning of words and political concepts are never factual."
You're still missing the point. The point is that there is a distinction between a subsidy and an externality and that distinction is important. It's important for measurement reasons (The exact monetary amount a government spends on something is easier to measure than the indirect cost of a policy,) and for simple communication reasons. It makes no sense to talk about instituting a tax to cover a subsidy. You institute a tax to cover an externality. It also matters because there are ways of dealing with externalities other than taxes and subsidies and reducing the language makes this more confusing. It's especially confusing when the distinction is made in one discussion (about renewables) but not in the other (about fossil fuels.)
There is a term for using an unexpected definition for a word that already has a widely used definition during a discussion, that is a 'stipulative definition'. It's dishonest to do so without being clear upfront or in response to a discussion where the original definition is in use. This results in equivocation. Whether or not a subsidy is morally equivalent to an externality is a moot point if you're willing toss about with the language. My work involves financial reporting and if my employer asked for one set of numbers and I gave him another that I argued were 'morally equivalent' would pretty clearly be in the wrong, even if the point I was making about moral equivalence was correct.
No, you're splitting hairs.
There are direct and indirect subsidies. Indirect subsidies include externalities: external costs paid by everyone else (that the government should be incentivizing reductions in by requiring the folks causing them to pay)
Semantic digressions aside, they're earning while everyone else pays costs resultant from their operations (and from our apparent inability to allocate with e.g. long term security, health, and prosperity as primary objectives for the public sphere)
Handing money to polluters helps polluters, and failing to disincentivize externalities also helps polluters, but it's OK to call one thing a subsidy and the other thing poor governance.
"Subsidy" carries a connotation of purposeful action to help something. Subsidizing a bad thing is worse than merely allowing it to happen or looking the other way. It seems like you want to re-label things in "B" by the label for "A" to make it sound worse.
>"Subsidy" carries a connotation of purposeful action to help something.
Purpouseful action implies awareness and intention.
Externality implies not yet recognized.
Having 2 words for similar concepts does not mean they are different concepts. "heavy" and "massive" are two different words but they convey essentially the same thing.
I would agree that unrecognized externalities are not subsidies, say CO2 pollution before it was recognized, but then again since they were literally not recognized as externalities (yet).
But the very moment CO2 pollution is recognized, the previously unrecognized externality is to be instantly viewed as a subsidy.
History repeats itself, the first time as a tragedy, from then on as a farce.
[deleted]
Why do subsidies have to be an active policy decision?
You’re right, when people talk about fossil fuel subsidies, that’s thinking of government gifts and tax breaks. My point is that this is a deeply inadequate way to think about it. If you only look at these, and compare fossil fuels to renewables, you’ll come away thinking that fossil fuels are far cheaper than they actually are.
"If you only look at these, and compare fossil fuels to renewables, you’ll come away thinking that fossil fuels are far cheaper than they actually are."
In the case of this discussion, you're looking at it backwards. The subject of fossil fuel subsidies was brought up in comparison to subsidies for renewables which in this case were already defined as specific government programs. If it weren't for that, I wouldn't be being such a stickler. However given that we are comparing renewables to fossil fuels we need to be sure that we are measuring the same thing and to include externalities when measuring fossil fuels and not when measuring renewables is to be not measuring the same thing. (I actually got confused when I first saw the IMF link so I know my concern isn't hypothetical.)
Yes, I get that the case can be made that cost of using fossil fuels is more that the sum of direct gifts by the government to fossil fuel companies (fucking duh!) but that can be expressed without implying that governments gave $5.2T in tax dollars directly to fossil fuel companies in 2017. Where confusion is possible, it's best to make distinctions and clarify what is meant.
"subsidies" includes both direct and indirect subsidies.
We can measure direct subsidies by measuring real and effective tax rates.
We can measure indirect subsidies like healthcare costs paid by Medicare with subjective valuations of human life and rough estimates of the value of a person's health and contribution to growth in GDP, and future economic security.
But who has the time for this when we're busy paying to help folks who require disaster relief services from the government and NGOs (neither of which are preventing further escalations in costs)
Problem is that if the word "subsidies" is used exclusively to mean a specific and narrow concept around cash transfers when talking about renewables, but then you use the term to include a much broader set of scenarios when talking about fossil fuels, while directly comparing the derived numbers, then you're misusing the language in order to deceive, regardless of all else. You're communicating that the numbers represent the same calculations for both forms of energy, when you know that they actually don't.
Define the terms however you like, as long as you are careful to not to induce people to believe someone that is untrue.
Show HN: Python Tests That Write Themselves
This actually gave me another idea. What do you all think, I’d be up for trying to build it.
It would watch your program during execution and record each function calls input and output. And then create tests for each function using those inputs and outputs.
You could always go over the created tests manually and fix them or review them but if nothing else it could be a good start.
Instagram created 'MonkeyType' to do something similar for type annotations.
A lower barrier way of trying this out could be to run MonkeyType in combination with OP's hypothesis-auto (which uses type annotations to generate the tests).
pytype (Google) [1], PyAnnotate (Dropbox) [2], and MonkeyType (Instagram) [3] all do dynamic / runtime PEP-484 type annotation type inference [4]
[1] https://github.com/google/pytype
[2] https://github.com/dropbox/pyannotate
Most Americans see catastrophic weather events worsening
The stratifications on this are troubling.
> But there are wide differences in assessments by partisanship. Nine in 10 Democrats think weather disasters are more extreme, compared with about half of Republicans.
It's not a partisan issue: we all pay these costs.
> Majorities of adults across demographic groups think weather disasters are getting more severe, according to the poll. College-educated Americans are slightly more likely than those without a degree to say so, 79 percent versus 69 percent.
Weather disasters are getting more severe. It is objectively, quantitatively true that weather disasters are getting more frequent and more severe.
> Weather disasters are getting more severe. It is objectively, quantitatively true that weather disasters are getting more frequent and more severe.
Source? What definitions are being used for severity? How is the sample of events selected? Is there a statistically-significant effect or might it be random variation?
> Source? What definitions are being used for severity? How is the sample of events selected? Is there a statistically-significant effect or might it be random variation?
These are great questions that any good skeptic / data scientist should always be asking. Here are some summary opinions based upon meta analyses with varyingly stringent inclusion criteria.
( I had hoped that the other top-level post I posted here would develop into a discussion, but these excerpts seem to have bubbled up. https://news.ycombinator.com/item?id=20919368 )
"Scientific consensus on climate change" lists concurring, non-commital, and opposing groups of persons with and without conflicting interests: https://en.wikipedia.org/wiki/Scientific_consensus_on_climat...
USGCRP, "2017: Climate Science Special Report: Fourth National Climate Assessment, Volume I" [Wuebbles, D.J., D.W. Fahey, K.A. Hibbard, D.J. Dokken, B.C. Stewart, and T.K. Maycock (eds.)]. U.S. Global Change Research Program, Washington, DC, USA, 470 pp, doi: 10.7930/J0J964J6.
"Chapter 8: Droughts, Floods, and Wildfire" https://science2017.globalchange.gov/chapter/8/
"Chapter 9: Extreme Storms" https://science2017.globalchange.gov/chapter/9/
"Appendix A: Observational Datasets Used in Climate Studies" https://science2017.globalchange.gov/chapter/appendix-a/
The key findings in this report do list supporting evidence and degrees of confidence in predictions about the frequency and severity of severe weather events.
I'll now proceed to support the challenged claim that disaster severity and frequency are increasing by citing disaster relief cost charts which do not directly support the claim. Unlike your typical televised debate or congressional session, I have: visual aids, a computer, linked to the sources I've referenced. Finding the datasets ( https://schema.org/Dataset ) for these charts may be something that someone has time for while the costs to taxpayers and insurance holders are certainly increasing for a number of reasons.
"Taxpayer spending on U.S. disaster fund explodes amid climate change, population trends" (2019) has a nice chart displaying "Disaster-relief appropriations, 10-year rolling median" https://www.washingtonpost.com/us-policy/2019/04/22/taxpayer...
"2018's Billion Dollar Disasters in Context" includes a chart from NOAA: "Billion-Dollar Disaster Event Types by Year (CPI-Adjusted)" with the title embedded in the image text - which I searched for - and eventually found the source of: [1] https://www.climate.gov/news-features/blogs/beyond-data/2018...
[1] "Billion-Dollar Weather and Climate Disasters: Time Series" (1980-2019) https://www.ncdc.noaa.gov/billions/time-series
Thank you!
It seems these data are mostly counted in dollars of damage or dollars of relief, which is a proxy for the severity.
Would it be correct to say there is still some question about whether dollars are a good measure of severity?
EDIT: As I am browsing the data, it's hard to disentangle actual weather events from things like lava, fires, unsound building decisions, and just the politics of money moving around.
> It is objectively, quantitatively true that weather disasters are getting more frequent and more severe.
From the article, storms may be getting slightly more severe - but "more frequent" is incorrect:
> Scientific studies indicate a warming world has slightly stronger hurricanes, but they don’t show an increase in the number storms hitting land, Colorado State University hurricane researcher Phil Klotzbach said. He said the real climate change effect causing more damage is storm surge from rising seas, wetter storms dumping more rain and more people living in vulnerable areas.
Now, we all should do what we can to address it - but we all should also examine whether we're picking and choosing only scientific observations that support our feelings/pre-conceived notions while discarding the rest of the data.
The article seems to have focused on perceptions of persons who aren't concerned with taking an evidence-based look (at various types of storms: floods, cyclones (i.e. hurricanes), severe thunderstorms, windstorms. Regardless, costs are increasing. I've listed a few sources here: https://news.ycombinator.com/item?id=20925127
"2017: Climate Science Special Report: Fourth National Climate Assessment, Volume I" > "Chapter 9: Extreme Storms" lists a number of relevant Key Findings with supporting evidence (citations) and degrees of confidence: https://science2017.globalchange.gov/chapter/9/
> we all pay these costs
... if there actually are costs to pay. You seem to be sidestepping the actual question here.
"Billion-Dollar Weather and Climate Disasters: Time Series" (1980-2019) https://www.ncdc.noaa.gov/billions/time-series
> The stratifications on this are troubling.
They are, yet almost no one I know of (in leadership positions, or the general public) seems to think they're troubling enough to put any serious effort into determining with some level of certainty why there is so much stratification along political lines on so many important issues, this being only one of them. "Conservatives are uneducated" sounds about right so that's what we'll go with it seems, regardless of whether it's actually correct. That belief may be right, but it may not.
> It's not a partisan issue: we all pay these costs.
In one perspective, but as is usually the case, there are several perspectives involved. Another non-trivial perspective is that the solution requires non-partisan cooperation, and from that perspective partisanship is not only an issue, it is a crucially important issue, so we might be well served by putting some effort into understanding the true nature of it.
> Weather disasters are getting more severe. It is objectively, quantitatively true that weather disasters are getting more frequent and more severe.
For certain definitions of "more", "severe", "objectively", "true", and "frequent".
So if the problem isn't simply that "conservatives are uneducated", what else might it be? I think the answer lies in the incredibly complex manner in which people perceive the world in general, what information they consume, how they perceive that information, and how they integrate it into their personal internal overall worldview. People think they think in facts, but they actually think in stories, and in turn this affects how they consume new information.
Take this simple article as an example, and notice the variety of perspectives we see already with just a few (14 at the time of my writing) comments posted in this thread. Notice how people aren't discussing just the content of the article, but rather including related ideas from past information they have consumed and stored (the mechanics of which we do not understand) within the mental model of the world they hold in their brain. An even better example of this behavior can be seen in the recent discussion on the CDC report on vaping: https://news.ycombinator.com/item?id=20915520 Observe all the complex perspectives based on the very same "facts" in the article. Observe all the assertions of related "facts", that are actually only opinions. Observe all the mind reading.
If you start looking for this phenomenon you will notice it everywhere, and if you pay closer attention you may also notice that certain topics are particularly prone to devolving into narrative (rather than fact) based conversation. The usual suspects are obvious: religion, gender, sexuality, etc, but smoking/vaping seems to me like somewhat of an interesting outlier. The former examples are very closely tied to personal identity, but while smoking/vaping shares an identity attribute to some degree, I feel like there's some other unseen psychological issue in play that results in such high polarization of beliefs.
My guess is you and I see very different things despite consuming the very same article. I lean conservative/libertarian (generally speaking), and I am deeply distrustful of government (for extremely good reasons I believe), so I know for a fact that my interpretation of the article is going to be heavily distorted by that. Any logical inconsistency, ambiguousness, disingenuousness, technical dishonesty, or anything else along those lines is going to get red flagged in my mind, whereas others will read it in a much more forgiving fashion. And in an article on a different political hot topic, we will switch our behaviors.
In such threads, I think it would be extremely interesting for people with opposing views to post excerpts of the parts that "catch your attention", with an explanation of why. This is kind of what happens anyway, but I'm thinking with a completely different motive: rather than quoting excerpts with commentary to argue your ~political side of the issue with the goal of "winning the argument", take an unemotional, more abstract view of your personal cognitive processing of the article, and post commentary on ~why/how you believe you feel you consider that important on a psychological level. Psychological self-analysis is famously difficult, but even with moderate success I suspect some very interesting things would rise to the surface.
> My guess is you and I see very different things despite consuming the very same article. I lean conservative/libertarian (generally speaking),
HN specifically avoids politics. In context to the in-scope article, when you say "conservative/libertarian" do you mean: fiscally conservative (haven't seen a deficit hawk in decades other than "Read my lips. No new taxes" followed by responsibly raising taxes), socially libertarian (Liberty as a fundamental right; if you're not violating the rights of others the government is not obligated or even granted the right to intervene at all), or conservative as in imposing your particular traditional standard of moral values which you believe are particular to a particular side of the aisle?
Or, do you mean that you're libertarian in regards to the need and the right to regulate business and industry in the interest of consumers ("laissez faire")? I'm certainly not the only person to observe that lack of regulation results in smog-filled cities due to un-costed 'externalities' in a blind pursuit of optimization for short-term profit.
At issue here, I think, is whether we think we can avert future escalations of costs by banding together to address climate change now; and how best to achieve the Paris Agreement targets that we set for ourselves (despite partisan denial, delusion, and indifference to increasing YoY costs [1]) https://en.wikipedia.org/wiki/Paris_Agreement
I'm personally and financially far more concerned about the long-term costs of climate change than a limited number of special interests who can very easily diversify and/or divest to take advantage of the exact same opportunities.
> and I am deeply distrustful of government (for extremely good reasons I believe), so I know for a fact that my interpretation of the article is going to be heavily distorted by that. Any logical inconsistency, ambiguousness, disingenuousness, technical dishonesty, or anything else along those lines is going to get red flagged in my mind, whereas others will read it in a much more forgiving fashion. And in an article on a different political hot topic, we will switch our behaviors.
While governments (and militaries (TODO)) do contribute substantially to emissions and resultant climate change, I think it unnecessary to qualify that unregulated decisions by industry should be the primary focus here. Industry has done far more to cause climate change than governments (which can more efficiently provide certain services useful to all citizens)
> In such threads, I think it would be extremely interesting for people with opposing views to post excerpts of the parts that "catch your attention", with an explanation of why. This is kind of what happens anyway, but I'm thinking with a completely different motive: rather than quoting excerpts with commentary to argue your ~political side of the issue with the goal of "winning the argument", take an unemotional, more abstract view of your personal cognitive processing of the article,
These people aren't doing jack about the problem because they haven't reviewed this chart: "Billion-Dollar Weather and Climate Disasters: Time Series" (1980-2019) https://www.ncdc.noaa.gov/billions/time-series
Maybe they want insurance payouts, which result in higher premiums. Maybe the people who built in those locations should be paying the costs.
> and post commentary on ~why/how you believe you feel you consider that important on a psychological level. Psychological self-analysis is famously difficult, but even with moderate success I suspect some very interesting things would rise to the surface.*
They don't even care because they refuse to accept that it's a problem.
The article was ineffectual at addressing the very real problem.
From https://news.ycombinator.com/item?id=20925127 :
> ( I had hoped that the other top-level post I posted here would develop into a discussion, but these excerpts seem to have bubbled up. https://news.ycombinator.com/item?id=20919368 )
In this observational study of perceptions, college education was less predictive than party affiliation.
Maybe reframing this as a short-term money problem [1] would result in compassion for people who are suffering billions of dollars of loss every year.
> HN specifically avoids politics.
I'm not saying this to be argumentative, but I suspect that is once again merely your perception. HN avoids (dang aggressively shuts down, to the detriment of the world imho, because if smart people can't find a way to discuss these things objectively, how do we expect the average person on the street to) political discussions that have a caustic odor, but there's plenty of political shit talking on HN.
> "conservative/libertarian" do you mean: fiscally conservative
Yes, but the real kind, not the phonies we've had for decades.
> socially libertarian (Liberty as a fundamental right; if you're not violating the rights of others the government is not obligated or even granted the right to intervene at all)
Yes.
> or conservative as in imposing your particular traditional standard of moral values which you believe are particular to a particular side of the aisle
To a degree, but here I think you're dealing with an interpretation of reality more than reality itself (not meant as an insult; it's how humans are). But yes, somewhat, and this is a whole other interesting and important conversation, that I think society should be having in a more serious / less polarized way.
> Or, do you mean that you're libertarian in regards to the need and the right to regulate business and industry in the interest of consumers ("laissez faire")?
I used to be very laissez faire, but in many specific areas I am now the exact opposite - if I was in charge, corporations would be in for a very rude awakening. But laissez faire is my default stance until facts suggest it is excessively harmful to the greater good.
> At issue here, I think, is whether we think we can avert future escalations of costs by banding together to address climate change now
100% agree. But if we willfully ignore the realities of human nature/psychology, I predict we'll never be able to even remotely band together, especially in the hyper-weaponized-meme world we now find ourselves in. Averting future escalations is the larger goal, but it is completely dependent upon cooperation, which is dependent on communication & perception. I believe perception is where we are failing most.
> I'm personally and financially far more concerned about the long-term costs of climate change than a limited number of special interests who can very easily diversify and/or divest to take advantage of the exact same opportunities.
Me too.
>> and I am deeply distrustful of government (for extremely good reasons I believe), so I know for a fact that my interpretation of the article is going to be heavily distorted by that. Any logical inconsistency, ambiguousness, disingenuousness, technical dishonesty, or anything else along those lines is going to get red flagged in my mind, whereas others will read it in a much more forgiving fashion. And in an article on a different political hot topic, we will switch our behaviors.
> While governments (and militaries (TODO)) do contribute substantially to emissions and resultant climate change, I think it unnecessary to qualify that unregulated decisions by industry should be the primary focus here. Industry has done far more to cause climate change than governments (which can more efficiently provide certain services useful to all citizens)
I think you misunderstood. Here I'm not talking about government/military damage to the environment (although that's a very big deal, the hypocrisy of which reinforces my distrust even more), I'm saying that I don't trust what they're up to at all. With a few exceptions (Bernie Sanders, etc), I am very distrustful of the true honesty and sincerity of all politicians regardless of affiliation. I suppose this is less that they're fundamentally dishonest, but rather the nature of our system is such that you have to be dishonest.
>> In such threads, I think it would be extremely interesting for people with opposing views to post excerpts of the parts that "catch your attention", with an explanation of why. This is kind of what happens anyway, but I'm thinking with a completely different motive: rather than quoting excerpts with commentary to argue your ~political side of the issue with the goal of "winning the argument", take an unemotional, more abstract view of your personal cognitive processing of the article,
> These people aren't doing jack about the problem because they haven't reviewed this chart: "Billion-Dollar Weather and Climate Disasters: Time Series" (1980-2019) https://www.ncdc.noaa.gov/billions/time-series Maybe they want insurance payouts, which result in higher premiums. Maybe the people who built in those locations should be paying the costs.
I'm afraid you've completely misunderstood. I was referring to a meta discussion, on human psychology and the nature of perception - the distinctly different way in which you and I consume, perceive, store, integrate, and recall the information in an article, not the information itself. I believe this is where the solution to these problems is hiding.
> They don't even care because they refuse to accept that it's a problem.
Here you are acting as if you are able to read the minds of other people. Intuition, which is kind of a dimensional compression of perceived reality, has evolved in humans because it is extremely useful. However, when dealing with high complexity, it can be dangerous.
> ( I had hoped that the other top-level post I posted here would develop into a discussion, but these excerpts seem to have bubbled up. https://news.ycombinator.com/item?id=20919368 )
Oh, you're not wrong on those facts I'm sure, but people don't think in facts. They think they do, but they don't actually. If we want to do something about these and other problems, you have to look for the invisible blockages, and some of them are within you and I, despite the sincerity of our intentions.
Unfortunately, even genuinely smart people (like much of the HN crowd) seem to be extremely resistant to even considering this notion. I suspect the problem is that although a person may be highly educated, at the subconscious level they are still highly tribal. But I believe if we can get people to start to realize these things, to see the world in this abstract form, then much of the rest might fall into place far easier than the current apparent polarization of beliefs would suggest.
> It is objectively, quantitatively true that weather disasters are getting more frequent and more severe.
There is no data to support such an assertion. It's confirmation bias stemming from the belief that climate change will lead to such an effect.
> "2017: Climate Science Special Report: Fourth National Climate Assessment, Volume I" > "Chapter 9: Extreme Storms" lists a number of relevant Key Findings with supporting evidence (citations) and degrees of confidence: https://science2017.globalchange.gov/chapter/9/
How about a link to a chart indicating frequency and severity of severe weather events?
The Paris Agreement is predicated upon the link between human actions, climate change, and severe weather events. 195 countries have signed the Paris Agreement with consensus that what we're doing is causing climate change.
Here are some climate-relevant poll questions:
Do you think the costs of disaster relief will continue to increase due to frequency and severity of severe weather events?
Does it make sense to spend more on avoiding further climate change now rather than even more on disaster relief later?
How can you help climate refugees? Do you donate to DoD and National Guards? Do you donate to NGOs? How can we get better at handling more frequent and more severe disasters?
Emergent Tool Use from Multi-Agent Interaction
Inkscape 1.0 Beta 1
Where Dollar Bills Come From
Monetary Policy Is the Root Cause of the Millennials’ Struggle
Volatility works out for people who save (who park capital in liquid assets that aren't doing work in order to have wheat for the eventual famine). These guys. They save, short like heck when the market is falling, and swoop in to save the day. What a great time to be selling 0% loans.
Personal Savings Rate (PSR) stratified by greatest generation and not greatest generation is also relevant. Are relatively fixed living expenses higher now? Yes. Is my generation just blowing what they could invest into interest-bearing investments on unnecessary stuff from Amazon? Yes. And expensive meals and drinks.
How have corporate profits and wages changed?
In their day, you put you gosh-danged money aside. For later. So that you have money later.
And that is why you should buy my book, entitled: "Invest in things with long term returns: don't buy shtuff you don't f need, save for tomorrow; and other financial advice"
Which brings me to: the cost of college textbooks and a college education in terms of average hourly wages.
By the way, over the longer term, index funds are likely to outperform funds. Gold may be likely to outperform the stock market. And, over the recent term -- this is for all you suckers out there -- cryptocurrencies have outperformed all stock and commodities markets. How much total wealth is being created on an annual basis here?
Payday loans have something like 300% APY.
How does 2% inflation affect trade when other central banking cabals haven't chosen the same target? "Devaluation"! "Treachery"!
Non-root containers, Kubernetes CVE-2019-11245 and why you should care
> At the same time, all the current implementations of rootless containers rely on user namespaces at their core. Not to be confused with what is referred to as non-root containers in this article, rootless containers are containers that can be run and managed by unprivileged users on the host. While Docker and other runtimes require a daemon running as root, rootless containers can be run by any user without additional capabilities.
non-root / rootless
How do black holes destroy information and why is that a problem?
man, watch the PBS YouTube channel SpaceTime. They're short, informative, and very well done. they dedicate a handful of episodes to getting you ready for Hawking radiation.
"Why Quantum Information is Never Destroyed" re: determinism and T-Symmetry ("time-reversal symmetry") by PBS SpaceTime https://youtu.be/HF-9Dy6iB_4
Classical information is 'collapsed' quantum information, so that would mean that classical information is never lost either.
There appear to be multiple solutions for Navier-Stokes; i.e. somewhat chaotic.
If white holes are on the other side of black holes, Hawking radiation would not account for the entirety of the collected energy/information. Is our visible universe within a white hole? Is everything that's ever been embedded in the sidewall of a black hole shredder?
Maybe even recordings of dinosaurs walking; or is that lemurs walking in reverse?
Do 1/n, 1/∞, and n/∞ approach a symbolic limit where scalars should not be discarded; with piecewise operators?
Banned C standard library functions in Git source code
FWIW, here's awesome-static-analysis > Programming Languages > C/C++: https://github.com/mre/awesome-static-analysis/blob/master/R...
These tools have lists of functions not to use. Most of them — at least the security-focused ones — likely also include: strcpy, strcat, strncpy, strncat, sprints, and vsprintf just like banned.h
Ask HN: What's the hardest thing to secure in a web-app?
"OWASP Top 10 Most Critical Web Application Security Risks" https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Proje...
> A1:2017-Injection, A2:2017-Broken Authentication, A3:2017-Sensitive Data Exposure, A4:2017-XML External Entities (XXE), A5:2017-Broken Access Control, A6:2017-Security Misconfiguration, A7:2017-Cross-Site Scripting (XSS), A8:2017-Insecure Deserialization, A9:2017-Using Components with Known Vulnerabilities, A10:2017-Insufficient Logging&Monitoring
"OWASP Top 10 compared to SANS CWE 25" https://www.templarbit.com/blog/2018/02/08/owasp-top-10-vs-s...
Crystal growers who sparked a revolution in graphene electronics
> This seven-metre-tall machine can squeeze carbon into diamonds
OT but, is this a thing now? Diamonds can be entangled.
Don't know what you mean by entangled. They mention forming diamonds using the heat and pressure of the press, which is a well known technique to alter the crystal lattice between carbon atoms.
Does it take more energy than mining for diamonds?
> Quantum Entanglement Links 2 Diamonds: Usually a finicky phenomenon limited to tiny, ultracold objects, entanglement has now been achieved for macroscopic diamonds at room temperature (2011) https://www.scientificamerican.com/article/room-temperature-...
It takes less human suffering.
Things to Know About GNU Readline
I map <up> to history-search-backward in my .inputrc; so I can type 'sudo ' and press <up> to cycle through everything starting with sudo:
# <up> -- history search backward (match current input)
"\e[A": history-search-backward
# <down> -- history search forward (match current input)
"\e[B": history-search-forward
I do the exact same thing. This will probably be considered sacrilege by some but I also remap tab completion to cycle through all matches instead of stopping at the first ambiguous character:
# Set up tab and Shift-tab to cycle through completion options
"\e[Z": "\e-1\t"
TAB: menu-complete
# Alt-S is normal style TAB complete
"\es": complete
> I also remap tab completion to cycle through all matches instead of stopping at the first ambiguous character
Windows PowerShell does this and I love it. I don’t understand why people wouldn’t want this
Because instead of helping me enter a long filename by its distinctive parts, it forces flat mental read-decide-skip loop based on first few letters, which is slow and cannot be narrowed down without typing out additional letters by hand. It is like selecting from a popup menu through a 1-line window.
If it presented a real popup for file selection, combining both worlds (tab to longest match, untab to undo, up-down to navigate), that would be great. But it doesn’t.
> which is slow and cannot be narrowed down without typing out additional letters by hand.
In a PowerShell environment specifically, you usually end up having to backspace a dozen or more characters completed from a long command name before you can resume narrowing the search. It's less of a problem on a Unix shell where commands tend to be short.
Interestingly, that's one of the things I hate about powershell (and dos)
zsh works like that too.
I did this a lot, too, until I started using fzf and mapped <Control-r> to fuzzy history search. It is really useful, you might like it!
I came here to post exactly this! Very useful.
Two more quick tips
1. Undo is ^_ (ctrl+shift+-)
2. To learn more tricks...
bindkey -p> Undo is ^_ (ctrl+shift+-)
I use ctrl-/. I don't recall whether it's a default binding or not, but at least you don't have to press the shift key :-)
Both are valid undo bindings in emacs so this is probably the reason that they are both supported. (I think this is an arrogant of a time where the keys were indistinguishable (and I think maybe they still are mostly indistinguishable to terminal apps))
Is this macro from the article dangerous because it doesn't quote the argument?
Control-j: "\C-a$(\C-e)"
The macro means:
\C-a: beginning of line
$: self-insert
(: self-insert
\C-e: end of line
): self-insert
grep 'blah blah' |some_file
$(grep 'blah blah' some_file)|
Show HN: Termpage – Build a webpage that behaves like a terminal
This looks useful.
FWIW, you can build a curses-style terminal GUI with Urwid (in Python) and use that through the web. AFAIU, it requires Apache; but it's built on Tornado (which is now built on Asyncio) so something more lightweight than Apache on a Pi should definitely be doable. Termpage with like a Go or Rust REST API may still be more lightweight, but more work.
Vimer - Avoid multiple instances of GVim with gvim –remote[-tab]-silent wrapper
I have a shell script I named 'e' (for edit) that does basically this. If VIRTUAL_ENV_NAME is set (by virtualenvwrapper), e opens a new tab in that gui vim remote if gvim or macvim are on PATH, or just in a console vim if not. https://github.com/westurner/dotfiles/blob/develop/scripts/e
'editwrd'/'ewrd'/'ew' does tab-completion relative to whatever $_WRD (working directory) is set to (e.g. by venv) and calls 'e' with that full path: https://github.com/westurner/dotfiles/blob/develop/scripts/_...
It's unfortunately not platform portable like vimer, though.
Electric Dump Truck Produces More Energy Than It Uses
Ask HN: Let's make an open source/free SaaS platform to tackle school forms
I have 4 kids. I am filling out all the start of school forms for each kid. I have to fill out these same forms each year. Are you doing the same thing? Let's make this year the last year we are manually filling out forms -- let's build a SaaS platform for school forms. Community built, open-sourced, free.
Brief sketch of the idea: survey monkey + docusign, but with a 100 pre-built templates for K-12 school situations. Medical emergency form. Carpool form. Field trip permission form. Backend gives schools an easy way to customize and track forms. Forms are emailed to parents and filled out online. Parent's information is saved so that any new form is pre-filled in with as much known info as possible.
Anyone feeling the same pain? Anyone want to join with me and do it?
Technically, a checkbox may qualify as a digital signature; however, identification / authentication and storage integrity are fairly challengeable (just as a written signature on a piece of paper with a date written on it is challengeable)
Given that notarization is not required for parental consent forms, I'm not sure what sort of server security expense is justified or feasible.
How much does processing all of the paper forms cost each school? Per-student?
In terms of storing digital record of authorization, a private set of per-student OpenBadges with each OpenBadge issued by the school would be easy enough. W3C Verified Claims (and Linked Data Signatures) are the latest standards for this sort of thing.
We could evaluate our current standards for chain of custody in regards to the level of trust we place in commercial e-signature platforms.
The school could send home a sheet with a QR code and a shorturl, but that would be more expensive than running hundreds of copies of the same sheet of paper.
The school could require a parent or guardian's email address for each student in the SIS Student Information System and email unique links to prefilled forms requesting authorization(s).
Just as with e-Voting, assuring that the person who checks a checkbox or tries to scribble their signature with a mouse or touchscreen is the authorized individual may be more difficult than verifying that a given written signature is that of the parent or guardian authorized to authorize.
AFAIU, Google Forms for School can include the logged-in user's username; but parents don't have school domain accounts with Google Apps for Education or Google Classroom.
How would the solution integrate with schools' existing SIS (Student Information Systems)? Upload a CSV of (student, {student info}, {guardian email (s)})? This is private information that deserves security, which costs money.
Which users can log-in for the school and/or district to check the state of the permission / authorization requests and PII personally-identifiable information.
While cryptographic signatures may be overkill as a substitute for permission slips, FWIW, a timestamp within a cryptographically-signed document only indicates what the local clock was set to at the time. Blockchains have relatively indisputable timestamps ("certainly no later than the time that the tx made it into a block"), but blockchains don't solve for proving the key-person relation at a given point in time.
And also, my parent or guardian said you can take me on field trips if you want. https://backpack.openbadges.org/
Ask HN: Is there a CRUD front end for databases (especially SQLite)?
I'm currently looking for a program (a simple executable) that "opens" an SQLite database and (via introspection of the schema) without any further configuration allows simple CRUD operations on the database.
Yes, there is DB Browser and a gazillion other database administration frontends, but it should really be limited to CRUD operations. No changing the table, the schema, the indexes. Simple UI.
For users that have no idea about SQL or databases.
Is there anything like that already done and ready to use?
There are lots of apps that do database introspection. Some also generate forms on the fly, but eventually it's necessary to: specify a forms widget for a particular field because SQL schema only describes the data and not the UI; and specify security authorization restrictions on who can create, read, update, or delete data.
And then you want to write arbitrary queries to filter on columns that aren't indexed; but it's really dangerous to allow clients to run arbitrary SQL queries because there basically are no row/object-level database permissions (the application must enforce row-level permissions).
Datasette is a great tool for read-only database introspection and queries of SQLite databases. https://github.com/simonw/datasette
Sandman2 generates a REST API for an arbitrary database. https://github.com/jeffknupp/sandman2
You can generate Django models and then write admin.py files for each model/table that you want to expose in the django.contrib.admin interface.
There are a number of apps for providing a GraphQL API given introspection of a database that occurs at every startup or at runtime; but that doesn't solve for row-level permissions (or web forms)
If you have an OpenAPI spec for the REST API that runs atop The database, you can generate forms ("scaffolding") from the OpenAPI spec and then customize those with form widgets; optionally with something like json-schema.
It's not safe to allow introspected CRUD like e.g. phpMyAdmin for anything but development. If there are no e.g. foreign-key constraints specified in the SQL schema,a blindly-introspected UI very easily results in database corruption due to invalid foreign key references (because the SQL schema doesn't specify what table.column a foreign key references).
Django models, for example, unify SQL schema and forms UI in models.py; admin.py is optional but really useful for scaffolding (such as when you're doing manual testing because you haven't yet written automated tests) https://docs.djangoproject.com/en/2.2/ref/contrib/admin/#mod...
California approves solar-powered EV charging network and electric school buses
> The press release from the company said, “heavy-duty vehicles produce more particulate matter than all of the state’s power plants combined”.
> […] for instance why only “10 school buses”?
IARC has recognized diesel exhaust as carcinogenic (lung cancer) since 2012.
Are there other electric school bus programs in the US?
(edit)
https://www.trucks.com/2019/03/22/can-electric-school-buses-...
> Most school systems don’t have sufficient capital to finance the high initial costs of electric bus purchases and charging infrastructure development, he said.
> In the U.S., the school bus market is about 33,000 to 35,000 vehicles per year – about six times more than transit buses.
You May Be Better Off Picking Stocks at Random, Study Finds
How is this not the same conclusion as other studies that find that index funds out perform others?
In addition to diversification that reduces risk of overexposure to down sectors or typically over-performing assets, index funds have survivorship bias: underperforming assets are replaced by assets that meet the fund's criteria.
Yes its sad that every one jumps on index funds which can be a good idea but its not the right choice 100% of the time.
Root: CERN's scientific data analysis framework for C++
Root has some features that are very unique and powerful.
It’s used in particle physics today mostly because it allows to do performant out-of-memory, on-disk Data Processing.
With frameworks like Python pandas, you always end up having to manually partition your data if it doesn’t fit in memory. And of course, it’s C++, so by default the data analysis code is pretty performant. This makes a difference when you can iterate your analysis in one hour instead of 20.
That being said, when I last worked with it, Root was a scrambled mess with terrible interfaces and way to many fringe features, e.g. around plotting, that are better handled by Python nowadays. It even has a C++ command line!!!
I wrote a blog post back then how I thought it could be fixed: https://www.konstantinschubert.com/2016/06/18/root8-what-roo...
> With frameworks like Python pandas, you always end up having to manually partition your data if it doesn’t fit in memory.
"Pandas Docs > Pandas Ecosystem > Out of Core" lists a number of solutions for working with datasets that don't fit into RAM: Blaze, Dask, Dask-ML (dask-distributed; Scikit-Learn, XGBoost, TensorFlow), Koalas, Odo, Ray, Vaex https://pandas-docs.github.io/pandas-docs-travis/ecosystem.h...
The dask API is very similar to the pandas API.
Are there any plans for ROOT to gain support for Apache Parquet, and/or Apache Arrow zero-copy reads and SIMD support, and/or https://RAPIDS.ai (Arrow, numba, Dask, pandas, scikit-learn, XGboost, spark, CUDA-X GPU acceleration, HPC)? https://arrow.apache.org/
https://root.cern.ch/root-has-its-jupyter-kernel (2015)
> Yet another milestone of the integration plan of ROOT with the Jupyter technology has been reached: ROOT now offers a Jupyter kernel! You can try it already now.
> ROOT is the 54th entry in this list and this is pretty cool. Now not only the PyROOT, the ROOT Python bindings, are integrated with notebooks but it's also possible to express your data mining in C++ within a notebook, taking advantage of all the powerful features of ROOT - plotting (now also interactive thanks to (Javascript ROOT](https://root.cern.ch/js/)), multivariate analysis, linear algebra, I/O and reflection: all available within a notebook.
Does this work with JupyterLab now? (edit) Here's the JupyterLab extension developer guide: https://jupyterlab.readthedocs.io/en/stable/developer/extens... (edit) here's the gh issue: https://github.com/root-project/jsroot/issues/166
...
ROOT is now installable with conda: `conda install -c conda-forge root metakernel jupyterlab # notebook`
For c/c++ in Jupyter, see xeus-cling https://github.com/QuantStack/xeus-cling
Coincidentally, cling (wrapped by xeus-cling) is also a product from CERN.
Many of the CERN researchers are pretty deep into C++.
It was there that I got my template meta-programming baptism, back in 2002, when gcc was still trying to cope with template heavy code.
And curiously, also where I got my first safety heavy code reviews of C++ best practices.
I don't believe those a coincidences, but more collaborations and the right people working together :-)
MesaPy: A Memory-Safe Python Implementation based on PyPy (2018)
I gave this a spin a while back but, unfortunately didn’t get very far since the software and dependencies have grown long in the byte. Since then, I’ve found RustPython [0] which is progressing toward feature parity with CPython but entirely written in Rust (!). A side benefit is that it compiles to Web Assembly, so if you could sandbox it without too much extra overhead.
> Since then, I’ve found RustPython [0] which is progressing toward feature parity with CPython but entirely written in Rust (!). A side benefit is that it compiles to Web Assembly, so if you could sandbox it without too much extra overhead.
It's now possible to run JupyterLab entirely within a browser with jyve (JupyterLab + pyodide) https://github.com/iodide-project/pyodide/issues/431
Pyodide:
> Pyodide brings the Python runtime to the browser via WebAssembly, along with the Python scientific stack including NumPy, Pandas, Matplotlib, parts of SciPy, and NetworkX. The packages directory lists over 35 packages which are currently available.
Is the RustPython WASM build more performant or otherwise preferable to brython or pyodide?
Ask HN: Configuration Management for Personal Computer?
Hello HN,
Every couple of years I find myself facing the same old tired routine: migrating my stuff off some laptop or desktop to a new one, usually combined with an OS upgrade. Is there anything like the kind of luxuries we now consider normal on the server side (IaaS; Terraform; maybe Ansible) that can be used to manage your PC and that would make re-imaging it as easy as it is on the server side?
Ansible is worth the extra few minutes, IMHO.
+ (minimal) Bootstrap System playbook
+ Complete System playbook (that references group_vars and host_vars)
+ Per-machine playbooks stored alongside the ansible inventory, group_vars, and host_vars in a separate repo (for machine-specific kernel modules and e.g. touchpad config)
+ User playbook that calls my bootstrap dotfiles shell script
+ Bootstrap dotfiles shell script, which creates symlinks and optionally installs virtualenv+virtualenvwrapper, gitflow and hubflow, and some things with pipsi. https://github.com/westurner/dotfiles/blob/develop/scripts/b...
+ setup_miniconda.sh that creates a CONDA_ROOT and CONDA_ENVS_PATH for each version of CPython (currently py27-py37)
Over the years, I've worked with Bash, Fabric, Puppet, SaltStack, and now Ansible + Bash
I log shell commands with a script called usrlog.sh that creates a $USER and per-virtualenv tab-delimited logfiles with unique per-terminal-session identifiers and ISO8601 timestamps; so it's really easy to just grep for the apt/yum/dnf commands that I ran ad-hoc when I should've just taken a second to create an Ansible role with `ansible-galaxy init ansible-role-name ` and referenced that in a consolidated system playbook with a `when` clause. https://westurner.github.io/dotfiles/usrlog.html#usrlog
A couple weeks ago I added an old i386 netbook to my master Ansible inventory and system playbook and VScode wouldn't install because VScode Linux is x86-64 only and the machine doesn't have enough RAM; so I created when clauses to exclude VScode and extensions on that box (with host_vars). Gvim with my dotvim works great there too though. Someday I'll merge my dotvim with SpaceVim and give SpaceMacs a try; `git clone; make install` works great, but vim-enhanced/vim-full needs to be installed with the system package manager first so that the vimscript plugin installer works and so that the vim binary gets updated when I update all.
I've tested plenty of Ansible server configs with molecule (in docker containers), but haven't yet taken the time to do a full workstation build with e.g. KVM or VirtualBox or write tests with testinfra. It should be easy enough to just run Ansible as a provisioner in a Vagrantfile or a Packer JSON config. VirtualBox supports multi-monitor VMs and makes USB passthrough easy, but lately Docker is enough for everything but Windows (with a PowerShell script that installs NuGet packages with chocolatey) and MacOS (with a few setup scripts that download and install .dmg's and brew) VMs. Someday I'll write or adapt Ansible roles for Windows and Mac, too.
I still configure browser profiles by hand; but it's pretty easy because I just saved all the links in my tools doc: https://westurner.github.io/tools/#browser-extensions
Someday, I'll do bookmarks sync correctly with e.g. Chromium and Firefox; which'll require extending westurner/pbm to support Firefox SQLite or a rewrite in JS with the WebExtension bookmarks API.
A few times, I've decided to write docs for my dotfiles and configuration management policies like someone else is actually going to use them; it seemed like a good exercise at the time, but invariably I have to figure out what the ultimate command sequence was and put that in a shell script (or a Makefile, which adds a dependency on GNU make that's often worth it)
Clonezilla is great and free, but things get out of date fast in a golden master image. It's actually possible to PXE boot clonezilla with Cobbler, but, AFAICT, there's no good way to secure e.g. per-machine disk or other config with PXE. Apt-cacher-ng can proxy-cache-mirror yum repos, too. Pulp requires a bit of RAM but looks like a solid package caching system. I haven't yet tested how well Squid works as a package cache when all of the machines are simultaneously downloading the exact same packages before a canary system (e.g. in a VM) has populated the package cache.
I'm still learning to do as much as possible with Docker containers and Dockerfiles or REES (Reproducible Execution Environment Specifications) -compatible dependency configs that work with e.g. repo2docker and https://mybinder.org/ (BinderHub)
GitHub Actions now supports CI/CD, free for public repositories
This is great news for developers. The trend has been to combine version control and CI for years now. For a timeline see https://about.gitlab.com/2019/08/08/built-in-ci-cd-version-c...
This is bad news for the CI providers that depend on GitHub, in particular CircleCI. Luckily for them (or maybe they saw this coming) they recently raised a series D https://circleci.com/blog/we-raised-a-56m-series-d-what-s-ne... and are already looking to add support for more platforms. It is hard to depend on a marketplace when it starts competing with you, from planning (Waffle.io), to dependency scanning (Gemnasium acquired by us), to CI (Travis CI layoff where especially sad).
It is interesting that a lot of the things GitHub is shipping is already part of Azure DevOps https://docs.microsoft.com/en-us/azure/architecture/example-... The overlap between Azure DevOps and GitHub seems to increase instead of to be reduced. I wonder what the integration story is and what will happen to Azure DevOps.
It's a horrible trend. CI should not be tied to version control. I mean we all have to deal with it now, but I'd much rather have my CI agnostic and not have config files for it checked into the repo.
I've browsed through the article you linked to, one of the subtitles was "Realizing the future of DevOps is a single application". Also a horrible idea: I think it locks developers into a certain workflow which is hard to escape. You have an issue with your setup you can't figure out - happened to me with Gitlab CI - sorry, you're out of luck. Every application is different, DevOps processes is something to be carefully crafted for each particular case with many considerations: large/small company, platform, development cycle, people preferred workflow etc. What I like to do is to have small well tested parts constitute my devops. It's a bad idea to adopt something just because everyone is doing this.
To sum it up, code should be separate from testing, deployment etc. On our team, I make sure developers don't have to think about devops. They know how to deploy and test and they know the workflow and commands. But that's about it.
I'm an operations guy, but I think I have a different perspective. The developers I work with don't have to think about CI/CD, but the configuration still lives in the repo, I'm just a contributer to that repo like they are.
Having CI configuration separate from the code sounds like a nightmare when a code change requires CI configurations to be updated. A new version of code requires a new dependency for instance, there needs to be a way to tie the CI configuration change with a commit that introduced that dependency. That comes automatically when they're in the same repo.
Having CI configuration inside the codebase also sounds like a nightmare when changes to the CI or deployment environment require configuration changes or when multiple CI/deployment environments exist.
For example as a use case: Software has dozens of tagged releases; organization moves from deploying on AWS to deploying in a Kubernetes cluster (requiring at least one change to the deployment configuration). Now, to deploy any of the old tagged releases, every release now has to be updated with the new configuration. This gets messy because there are two different orthogonal sets of versions involved. First, the code being developed has versions and second, the environments for testing, integration, and deployment also change over time and have versions to be controlled.
Even more broadly, consider multiple organizations using the same software package. They will each almost certainly have their own CI infrastructure, so there is no one "CI configuration" that could ever be checked into the repository along with the code without each user having to maintain their own forks/patchsets of the repo with all the pain that entails.
You can create a separate repo with your own CI config that pulls in the code you want to test; and thus ignore the code's CI config file. When something breaks, you'd then need to determine in which repo something changed: in the CI config repo, or the code repo. And then, you have CI events attached to PRs in the CI config repository.
IMHO it makes sense to have CI config version controlled in the same repo as the code. Unless there's a good tool for bisecting across multiple repos and subrepos?
The Fed is getting into the Real-Time payments business
This system will need to interface with other domestic and international settlement and payments networks.
There is thus an opportunity for standards, a need for federation, and a need to make it easy for big players to offer liquidity.
As far as I understand, e.g. Ripple and Stellar solve basically exactly the 24x7x365 RTGS problem that FedNow intends to solve; and, they allow all sorts of assets to be plugged into the network. Could FedNow just use a different UNL (Unique Node List) with participating banks operating trusted validators and/or offering liquidity ("liquidity provisioning")?
Notably, Ripple is specifically positioned to do international interbank real time gross settlement (RTGS) and remittances. Ripple could integrate with FedNow directly. Most efficiently, if it complies with KYC/AML requirements, FedNow could operate an XRP Ledger. Or, each bank could operate XRP Ledgers. https://xrpl.org/become-an-xrp-ledger-gateway.html
Getting thousands of banks to comply with an evolving API / EDI spec is no small task. Blockchain solutions require API compliance, have solutions for governance where there are a number of stakeholders seeking to reach consensus, and lack single points of failure.
Here's to hoping that we've learned something about decentralizing distributed systems for resiliency.
>> In contrast, the XRP Ledger requires 80 percent of validators on the entire network, over a two-week period, to continuously support a change before it is applied. Of the approximately 150 validators today, Ripple runs only 10. Unlike Bitcoin and Ethereum — where one miner could have 51 percent of the hashing power — each Ripple validator only has one vote in support of an exchange or ordering a transaction. https://news.ycombinator.com/item?id=19195050
So, you want to get banks onboard with only one s'coin USD stablecoin; but you don't want to deal with exchanges or FOREX or anything because that's a different thing? And, this is not just yet another ACH with lower clearance time?
> Interledger Architecture
https://interledger.org/rfcs/0001-interledger-architecture/
> Interledger provides for secure payments across multiple assets on different ledgers. The architecture consists of a conceptual model for interledger payments, a mechanism for securing payments, and a suite of protocols that implement this design.
> The Interledger Protocol (ILP) is the core of the Interledger protocol suite. Colloquially, the whole Interledger stack is sometimes referred to as "ILP". Technically, however, the Interledger Protocol is only one layer in the stack.
> Interledger is not a blockchain, a token, nor a central service. Interledger is a standard way of bridging financial systems. The Interledger architecture is heavily inspired by the Internet architecture described in RFC 1122, RFC 1123 and RFC 1009.
[...]
> You can envision the Interledger as a graph where the points are individual nodes and the edges are accounts between two parties. Parties with only one account can send or receive through the party on the other side of that account. Parties with two or more accounts are connectors, who can facilitate payments to or from anyone they're connected to.
> Connectors provide a service of forwarding packets and relaying money, and they take on some risk when they do so. In exchange, connectors can charge fees and derive a profit from these services. In the open network of the Interledger, connectors are expected to compete among one another to offer the best balance of speed, reliability, coverage, and cost.
Why should we prefer an immutable, cryptographically-signed blockchain solution over SQL/BigTable/MQ for FedNow?
Blockchain and payments standards: https://news.ycombinator.com/item?id=19813340
... Here's the notice and request for comment PDF: "Docket No. OP – 1670: Federal Reserve Actions to Support Interbank Settlement of Faster Payments" https://www.federalreserve.gov/newsevents/pressreleases/file...
"Federal Reserve announces plan to develop a new round-the-clock real-time payment and settlement service to support faster payments" https://www.federalreserve.gov/newsevents/pressreleases/othe...
A Giant Asteroid of Gold Won’t Make Us Richer
> this example shows that real wealth doesn’t actually come from golden hoards. It comes from the productive activities of human beings creating things that other human beings desire.
Value, Price, and Wealth
???
I'd suggest a different triad: cost, price, value.
https://old.reddit.com/r/dredmorbius/comments/48rd02/cost_va...
Good call. I don't know where I was going with that. Cost, price, value, and wealth.
Are there better examples for illustrating the differences between these kind of distinct terms?
Less convertible collectibles like coins and baseball cards (that require energy for exchange) have (over time t): costs of production, marketing, and distribution; retail sales price; market price; and 'value' which is abstract relative (opportunity cost in terms of fiat currency (which is somehow distinct from price at time t (possibly due to 'speculative information')))
Wealth comes from relationships, margins between costs and prices, long term planning, […]
Are there better examples for illustrating the differences between these kind of distinct terms?
For concepts as intrinsically fundamental to economics as these are, the agreement and understanding of what they are, even amomg economists, is surprisingly poor. It's not even clear whether or not "wealth" refers to a flow or stock -- Adam Smith uses the term both ways. And much contemporary mainstream 'wealth creation" discussion addresses accounting profit rather than economic wealth. Or broader terms such aas ecological wealth (or natural capital). There's some progress, and Steve Keen has been synthesizing much of it recently, but the terms fare poorly.
A key issue is that "price‘ and "exchange value" are ofteen conflated, creating confusiin with use/ownership value.
Addressing your terms, "cost" and "price", and typically "value", indicate some metric of exchange or opportunity cost (or benefit). Whilst "wealth", as typically used, tends to relate to some store or accumulation. In electrical terms (a potentially, so to speak, useful analogue) the difference between voltage and charge, with current representing some other property, possibly material flows of goods or energy.
The whole question of media for exchange (currency, and the like), and durable forms of financial wealth (land, art, collectibles) is another interesting one, with discussionnby Ricardo and Jevons quite interesting -- both useful and flawed.
And don't even get me started on the near total discounting of accumulated natural capital, say, the 100-300 million year factor of time embodied in fossil fuels. The reasons and rationales for excluding that being fascinating (Ricardo, Tolstoy, Gray, Hotelling, Boulding, Soddy, Georgescu-Roegen, Daly, Keen).
You are correct that all value (and hence wealth) is relative, and hence relational.
TL;DR: Not that I'm aware.
Abusing the PHP Query String Parser to Bypass IDS, IPS, and WAF
Hrm... the solution is to keep your PHP codebase up to date...?
The solution is to throw out your WAF.
Possible solutions:
(1) Change all underscores in WAF rule URL attribute names to the appropriate non-greedy regex. Though I'm not sure about the regex the article suggests: '.' only matches one character, AFAIU.
(2) Add a config parameter to PHP that turns off the magical url parameter name mangling that no webapp should ever depend on ( and have it default to off because if you rely on this 'feature' you should have to change a setting in php.ini anyway )
Ask HN: Scripts/commands for extracting URL article text? (links -dump but)
I'd like to have a Unix script that basically generates a text file named, with the page title, with the article text neatly formatted.
This seems to me to be something that would be so commonly desired by people that it would've been done and done and done a hundred times over by now, but I haven't found the magic search terms to dig up people's creations.
I imagine it starts with "links -dump", but then there's using the title as the filename, and removing the padded left margin, wrapping the text, and removing all the excess linkage.
I'm a beginner-amateur when it comes to shell scripting, python, etc. - I can Google well and usually understand script or program logic but don't have terms memorized.
Is this exotic enough that people haven't done it, or as I suspect does this already exist and I'm just not finding it? Much obliged for any help.
Just for the record in case anyone digs this up on a later Google search, install the newspaper, unidecode, and re python libraries (pip3 install), then:
from sys import argv
from unidecode import unidecode
from newspaper import Article
import re
script, arturl = argv
url = arturl
article=Article(url)
article.download()
article.parse()
title2 = unidecode(article.title)
fname2 = title2.lower()
fname2 = re.sub(r"[^\w\s]", '', fname2)
fname2 = re.sub(r"\s+", '-', fname2)
text2 = unidecode(article.text)
text2 = re.sub(r'\n\s*\n', '\n\n', text2)
f = open( '~/Desktop/' + str(fname2) + '.txt', 'w' )
f.write( str(title2) + '\n\n' )
f.write( str(text2) + '\n' )
f.close()
#!/bin/bash
/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.7/bin/python3 ~/bin/url2txt.py $1
#!/bin/bash
while IFS='' read -r l || [ -n "$l" ]; do
~/bin/u2t "$l"
done < $1
There could be collisions where `fname2` is the same for different pages; resulting in unintentionally overwriting. A couple possible solutions: generate a random string and append it to the filename, set fname2 to a hash of the URL, replace unsafe filename characters like '/' and/or '\' and/or '\n' with e.g. underscores. IIRC, URLs can be longer than the max filename length of many filesystems, so hashes as filenames are the safest solution. You can generate an index of the fetched URLs and store it with JSON or e.g. SQLite (with Records and/or SQLAlchemy, for example).
If or when you want to parallelize (to do multiple requests at once because most of the time is spent waiting for responses from the network) write-contention for the index may be an issue that SQLite solves for better than a flatfile locking mechanism like creating and deleting an index.json.lock. requests3 and aiohttp-requests support asyncio. requests3 supports HTTP/2 and connection pooling.
SQLite can probably handle storing the text of as many pages as you throw at it with the added benefit of full-text search. Datasette is a really cool interface for sqlite databases of all sorts. https://datasette.readthedocs.io/en/stable/ecosystem.html#to...
...
Apache Nutch + ElasticSearch / Lucene / Solr are production-proven crawling and search applications: https://en.m.wikipedia.org/wiki/Apache_Nutch
> I imagine it starts with "links -dump", but then there's using the title as the filename,
The title tag may exceed the filename length limit, be the same for nested pages, or contain newlines that must be escaped.
These might be helpful for your use case:
"Newspaper3k: Article scraping & curation" https://github.com/codelucas/newspaper
lazyNLP "Library to scrape and clean web pages to create massive datasets" https://github.com/chiphuyen/lazynlp/blob/master/README.md#s...
scrapinghub/extruct https://github.com/scrapinghub/extruct
> extruct is a library for extracting embedded metadata from HTML markup.
> It also has a built-in HTTP server to test its output as JSON.
> Currently, extruct supports:
> - W3C's HTML Microdata
> - embedded JSON-LD
> - Microformat via mf2py
> - Facebook's Open Graph
> - (experimental) RDFa via rdflib
NPR's Guide to Hypothesis-Driven Design for Editorial Projects
HDD – Hypothesis-Driven Development – Research, Plan, Prototype, Develop, Launch, Review.
The article lists (and links to!) "Lean UX" [1] and Google Ventures' Design Sprint Methodology as inspirations.
[1] "Lean UX: Applying Lean Principles to Improve User Experience" http://shop.oreilly.com/product/0636920021827.do
[2] https://www.gv.com/sprint/
"How To Write A Technical Paper" [3][4] has: (Related Work, System Model, Problem Statement), (Your Solution), (Analysis), (Simulation, Experimentation), (Conclusion)
Gryphon: An open-source framework for algorithmic trading in cryptocurrency
Hey folks, I'm the primary maintainer of Gryphon. The backstory here is: I was one of the founders of Tinker, the trading company that built Gryphon as our in-house trading infrastructure. We operated 2014-18, starting with just a simple arbitrage bot, and slowly grew the operation until our trades were peaking above 20% of daily trading volume on the big exchanges.
The company has since wound down (founders moved on to other projects), but I always thought the code deserved to be shared, so we've open sourced it. Someone trying to do what we did will probably save 1.5 years of engineering effort if they build on Gryphon vs. make their own. As far as I know there isn't anything out there like this, in any market (not just cryptocurrencies).
Hope you guys like it!
> As far as I know there isn't anything out there like this, in any market (not just cryptocurrencies).
How does Gryphon compare to Catalyst (Zipline)? https://github.com/enigmampc/catalyst
They list a few example algorithms: https://enigma.co/catalyst/example-algos.html
"Ask HN: Why would anyone share trading algorithms and compare by performance?" https://news.ycombinator.com/item?id=15802834 (pyfolio, popular [Zipline] algos shared through Quantopian)
"Superalgos and the Trading Singularity" https://news.ycombinator.com/item?id=19109333 (awesome-quant,)
Looks great!
Though looking at the code -> https://github.com/garethdmm/gryphon/tree/master/gryphon/lib...
Seems quite hard to extend it with new exchanges.
Shouldn't be, you just need to write a wrapper for the exchange API with the interface defined in 'gryphon.lib.exchange.exchange_api_wrapper'. I'll add an article to the docs with more about that soon.
Would CCXT be useful here? https://github.com/ccxt/ccxt
> The ccxt library currently supports the following 135 cryptocurrency exchange markets and trading APIs:
It's possible CCXT could be used to easily wrap other exchanges into gryphon. I'm not familiar with the library so hard to guess if it would be a net win or not.
Thanks for taking the time to open source this.
So is its use case limited to arb? Or are other HFT strategies supported?
It's perfectly general: arb, market making, signal trading, ml, etc. Whatever strategy class you're thinking of, you can probably implement it on Gryphon.
Can you please explain to me how a tool written in python can be used for HFT or market making?
I’m asking because we generally used ASICS and c++ in the past, or more recently rust. Even GPUs are often difficult because they introduce milliseconds of latency.
If you want to restrict the definition of HFT to only sub-millisecond strategies you're correct. But then, all HFT is impossible in crypto, since with web request latency and rate limits, it would be very difficult to get tick speeds even in the 10s of milliseconds. It's fine if you want to call this "algo trading" instead of HFT, but I think a common understanding of the term would include Gryphon's capabilities.
In any case Gryphon uses Cython to compile itself down to C, which isn't quite as good as writing in native C but is a good chunk of the way there.
> In any case Gryphon uses Cython to compile itself down to C, which isn't quite as good as writing in native C but is a good chunk of the way there.
Would there be any advantage to asyncio with uvloop (also written in Cython (on libuv like Node) like Pandas)? https://github.com/MagicStack/uvloop
IDK how many e.g. signals routines benefit from asyncio yet.
Wow. That’s pretty cool.
Been toying with the idea of algo trading on stock market. Nice to have a reference work
I've been playing around with using lstm recurrent nets to find patterns in forex trades, with no real expectation of anything other than learning about recurrent nets (and td convolutional nets). I was able to access 15 years of historical tick data. I would imagine lack of historical pricing data would be an issue for any machine learning approach to crypto trading. Even with 15 years of daily prices I only have ~5500 samples per major currency pair. I've toyed with learning off hourly prices rather than daily, and I've also thought about creating more samples by shifting prices up or down, since perhaps the general patterns would be the same.
Ouch. Out of all the possible subjects to learn NNs on, you have picked by far the most difficult possible. Seriously. If you think of an analogue to rocketry, with the easiest being launching fireworks from a bottle and the other being a mission to Mars, you have picked a Moon landing.
I don’t even know where to begin. Financial data has an extremely low signal to noise ratio and it is fraught with pitfalls. It is highly non-normal, heteroscedastic, non-stationary and frequently changes behavioural regimes. It is irregular, the information content is itself irregular and the prices sold by vendors often have difficult to detect issues that will taint your results until you actually start trading and realise that a fundamental assumption was wrong. You may train a model on one period, and find that the market behaviour has changed and your model is rubbish. Cross validation and backtesting on black box algorithms with heavy parameter tuning is a field of study on it’s own with so many issues that endless papers have been written on each specific nuance.
Successfully building ML models for trading is an extremely difficult discipline that requires a deep understanding of the markets, the idiosyncrasies of market data, statistics and programming. Most quant shops who run successful ML Algos (they are quite rare) have dedicated data teams whose entire remit is to source and clean data. The saying of rubbish in, rubbish out is very true. Even data providers like Reuter’s or Bloomberg frequently have crap data. We pay nearly 500k a year to Reuters, and find errors in their tick data every week. Data like spot forex is a special beast because the market is decentralized. There is no exchange which could provide an authoritative price feed. Trades have been rolled back in the past and if your data feed does not reflect this, you are effectively analysing junk data.
I don’t even want to get started about the fact that trying to train an RNN on 5500 observations is folly. Did you treat the data in any way? The common way to regularise market data for ML is to resample it to information bars. This is not going to work on a daily basis, so you should start off with actual tick data.
Nearly every starry eyed junior quant goes in with the notion that you can just run some fancy ML models on some market data and you’ll get a star trading algo. That a small handful of statistical tests will tell you whether your results are meaningful, whether your data has autocorrelation or mean reverting properties. In reality, ML models are very difficult to train on financial data. Most statistical forecasting tools fail to find relationships and blindly training models on past data very rarely results in more than spurious performance in a back test.
I don’t want to discourage you by any means, but I’d start off with something easier than what you are proposing. Finance firms have entire teams dedicated to what you are trying to do and even they often fail to find anything.
Thanks for the great feedback. I have no expectations for this other than the learning, and it's already been successful on that front. Just seemed like a fun thing to poke at when most other hobbyists seem to be doing image analysis and language modeling. I've crawled a couple of forums and I get that there are a lot of people out there who think they can readily use these techniques to make money. I doubt very much that this will be the outcome in my case :).
Where I am now I am just trying to figure out how to treat the data, whether to normalize or stationarize and how to encode inputs, etc. The reason that I am working with daily prices is that the fantasy output of this would be a model that can inform a one day grid trading strategy. It may very well be that daily prices won't work for this.
Whether there's anything like an equilibrium in cryptoasset markets where there are no underlying fundamentals is debatable. While there's no book price, PoW coin prices might be rationally describable in terms of (average_estimated cost of energy + cost per GH/s + 'speculative value')
A proxy for energy costs, chip costs, and speculative information
Are there standard symbols for this?
Can cryptoasset market returns be predicted with quantum harmonic oscillators as well? What NN topology can learn a quantum harmonic model? https://news.ycombinator.com/item?id=19214650
"The Carbon Footprint of Bitcoin" (2019) defines a number of symbols that could be standard in [crypto]economics texts. Figure 2 shows the "profitable efficiency" (which says nothing of investor confidence and speculative information and how we maybe overvalue teh security (in 2007-2009)). Figure 5 lists upper and lower estimates for the BTC network's electricity use. https://www.cell.com/joule/fulltext/S2542-4351(19)30255-7
Here's a cautionary dialogue about correlative and causal models that may also be relevant to a cryptoasset price NN learning experiment: https://news.ycombinator.com/item?id=20163734
Wind-Powered Car Travels Downwind Faster Than the Wind
NOAA upgrades the U.S. global weather forecast model
> Working with other scientists, Lin developed a model to represent how flowing air carries these substances. The new model divided the atmosphere into cells or boxes and used computer code based on the laws of physics to simulate how air and chemical substances move through each cell and around the globe.
> The model paid close attention to conserving energy, mass and momentum in the atmosphere in each box. This precision resulted in dramatic improvements in the accuracy and realism of the atmospheric chemistry.
Global Forecast System > Future https://en.wikipedia.org/wiki/Global_Forecast_System#Future
A plan to change how Harvard teaches economics
This isn't an economics class. It's a public policy class. Here are the course topics:[1]
- Part I: Equality of Opportunity
- Part II: Education
- Part III: Racial Disparities
- Part IV: Health
- Part V: Criminal Justice
- Part VI: Climate Change
- Part VII: Tax Policy
- Part VIII: Economic Development and Institutional Change
This really belongs in Harvard's "JFK School of Government", not economics.
Possible topics for a modern economics intro class:
- Instability and equilibrium, or why markets oscillate.
- From zero to one, the tendency to and effects of monopoly and near-monopoly.
- Externalities, their uses and discontents.
- Debt vs. equity vs. what tax policy rewards
- Scarce resources that don't map to money - attention and time.
- Finance as a system decoupled from productive activity
[1] https://opportunityinsights.org/wp-content/uploads/2019/05/E...
Is there a single book that covers all that? That would be awesome.
Schools like Harvard don’t teach from a book.
Wrong side of the Atlantic. I didn’t have to buy a single textbook my entire undergrad because we had libraries and lecture notes. Required textbooks are far more a US thing than most other countries.
Harvard absolutely has courses taught from a book. I’m sure Mankiw requires his textbook for Harvard’s intro econ course. He wrote it, he thinks it’s good.
And he's made $42 million in royalties on the book. It's almost endearing how economists claim they are somehow immune to incentives, until you pause to consider they are primarily employed as apologists for the continuation of rent-seeking policies that entrench the rich and mighty.
> apologists for the continuation of rent-seeking policies that entrench the rich and mighty.
This.
"THE IMF CONFIRMS THAT 'TRICKLE-DOWN' ECONOMICS IS, INDEED, A JOKE" https://psmag.com/economics/trickle-down-economics-is-indeed...
> INCREASING THE INCOME SHARE TO THE BOTTOM 20 PERCENT OF CITIZENS BY A MERE ONE PERCENT RESULTS IN A 0.38 PERCENTAGE POINT JUMP IN GDP GROWTH.
> The IMF report, authored by five economists, presents a scathing rejection of the trickle-down approach, arguing that the monetary philosophy has been used as a justification for growing income inequality over the past several decades. "Income distribution matters for growth," they write. "Specifically, if the income share of the top 20 percent increases, then GDP growth actually declined over the medium term, suggesting that the benefits do not trickle down."
"Causes and Consequences of Income Inequality: A Global Perspective" (2015) https://scholar.google.com/scholar?hl=en&as_sdt=0%2C43&q=%22...
I'll add that we tend to overlook the level of government spending during periods trickle-down economics and confound. Change in government spending (somewhat unfortunately regardless of revenues) is a relevant factor.
Let's make this economy great again? How about you identify the decade(s) you're referring to and I'll show you the tax revenue (on income and now capital gains), federal debt per capital, and the growth in GDP.
The difference big data/data science/empirical study and 'classical' economics (by which I'd include any system of economics that seeks to explain human behavior via an underlying metatheory) is that a primarily empirical approach obscures the necessary underlying theory present in any experiment where you're trying to fit data to a curve.
For example, when you run a science experiment and you plot the data, you may find that you're looking at a line. While this is an interesting finding, it has zero predictive value for anything other than the exact situation you've collected data for. In order to formulate scientific law, you first must (a) believe that such a thing exists and (b) have some theory as to what shape the curve ought to fit. For example, a naive look at physics using an 'empirical' approach might incorrectly conclude that force is mass times acceleration. While moderately useful for many problems, this offers little predictive power in the general case. In order to actually formulate a law that can be of predictive value, you have to first consider various other laws and axioms (such as the constant speed of light for force), at which point -- by deduction, without any need of empirism -- you determine that this is wrong, and you need another kind of equation to fit your data to.
I don't know if the simplistic demand curves drawn in the original text book are correct or not. However, at least those are based on a particular set of assumptions that can be validated or not. The kind of empiricism put forth by Mr Chetty does not offer this at all.
All this is to say that, while data is useful for validation, it is not useful for prediction. The last thing we need is a black-box machine learning model to make major economic decisions off of. What we do need is proper models that are then validated, which don't necessarily need 'big data.'
> All this is to say that, while data is useful for validation, it is not useful for prediction. The last thing we need is a black-box machine learning model to make major economic decisions off of. What we do need is proper models that are then validated, which don't necessarily need 'big data.'
Hand-wavy theory - predicated upon physical-world models of equillibrium which are themselves classical and incomplete - without validation is preferable to empirical models? Please.
Estimating the predictive power of some LaTeX equations is a different task than measuring error of a trained model.
If the model does not fit all of the big data, the error term is higher; regardless of whether the model was pulled out of a hat in front of a captive audience or deduced though inference from actual data fed through an unbiased analysis pipeline.
If the 'black-box predictive model' has lower error for all available data, the task is then to reverse the model! Not to argue for unvalidated theory.
Here are a few discussions regarding validating economic models, some excellent open econometric lectures (as notebooks that are unfortunately not in an easily-testable programmatic form), the lack of responsible validation, and some tools and datasets that may be useful for validating hand-wavy classical economic theories:
"When does the concept of equilibrium work in economics?" https://news.ycombinator.com/item?id=19214650
> "Lectures in Quantitative Economics as Python and Julia Notebooks" https://news.ycombinator.com/item?id=19083479 (data sources (pandas-datareader, pandaSDMX), tools, latex2sympy)
That's just an equation in a PDF.
(edit) Here's another useful thread: "Ask HN: Data analysis workflow?" https://news.ycombinator.com/item?id=18798244
Most of the interesting economic questions are inference problems, not prediction problems The question is not "what is the best guess of y[i] given these values of x[i]'s", but what would y[i] have been for this very individual i (or country in macro-economics) if we could have wound back the clock and change the values of x[i]'s for this individual. The methods that economists know and use may not be the best, but the standard ML prediction methods do not address the same questions, and data scientists without a social / economic / medical background are often not even aware of the distinction.
Economists and social scientists try to do non-experimental causal inference. Maybe they're not good at it, maybe the very problem is unsolvable, but it's not because they don't know how Random Forests or RNNs work. Economists already know that students from single parent families do worse at school than from married families. If the problem is just to predict individual student results, number of parents in the household is certainly a good predictor. The problem facing economists is, would encouraging marriage or discouraging diveroce improve student results? Nothing in PyTorch or Tensorflow will help with the answer..
Backtesting algorithmic trading algorithms is fairly simple: what actions would the model have taken given the available data at that time, and how would those trading decisions have affected the single objective dependent variable. Backtesting, paper trading, live trading.
Medicine (and also social sciences) is indeed more complex; but classification and prediction are still the basis for making treatment recommendations, for example.
Still, the task really is the same. A NN (like those that Torch, Theano, TensorFlow, and PyTorch produce; now with the ONNX standard for neural network model interchange) learns complex relations and really doesn't care about causality: minimize the error term. Recent progress in reducing the size of NN models e.g. for offline natural language classification on mobile devices has centered around identifying redundant neuronal connections ("from 100GB to just 0.5GB"). Reversing a NN into a far less complex symbolic model (with variable names) is not a new objective. NNs are being applied for feature selection, XGBoost wins many Kaggle competitions, and combinations thereof appear to be promising.
Actually testing second-order effects of evidence-based economic policy recommendations is certainly a complex highly-multivariate task (with unfortunate ideological digression that presumes a higher-order understanding based upon seeming truisms that are not at all validated given, in many instances, any data). A causal model may not be necessary or even reasonably explainable; and what objective dependent variables should we optimize for? Short term growth or long-term prosperity with environmental sustainability?
... "Please highly weight voluntary sustainability reporting metrics along with fundamentals" when making investments and policy decisions?
Were/are the World3 models causal? Many of their predictions have subsequently been validated. Are those policy recommendations (e.g. in "The Limits to Growth") even more applicable today, or do we need to add more labeled data and "Restart and Run All"?
...
From https://research.stlouisfed.org/useraccount/fredcast/faq/ :
> FREDcast™ is an interactive forecasting game in which players make forecasts for four economic releases: GDP, inflation, employment, and unemployment. All forecasts are for the current month—or current quarter in the case of GDP. Forecasts must be submitted by the 20th of the current month. For real GDP growth, players submit a forecast for current-quarter GDP each month during the current quarter. Forecasts for each of the four variables are scored for accuracy, and a total monthly score is obtained from these scores. Scores for each monthly forecast are based on the magnitude of the forecast error. These monthly scores are weighted over time and accumulated to give an overall performance.
> Higher scores reflect greater accuracy over time. Past months' performances are downweighted so that more-recent performance plays a larger part in the scoring.
The #GobalGoals Targets and Indicators may be our best set of variables to optimize for from 2015 through 2030; I suppose all of them are economic.
Using predictive models for policy is not new, in fact it was the standard approach long before more inferential models, and the famed Lucas critique precisely targets a primitive approach similar to what you are proposing.
The issue is the following: In economics, one is interested in an underlying parameter of a complex equilibrium system (or, if you wish, a non-equilibrium complex system of multi-agentic behavior). This may be, for example, some pricing parameter for a given firm - say - how your sold units react to setting a price.
Economics faces two basic issues:
First, any predictive model (like a NN or simple regression) that takes price as an input, will not correctly estimate the sensitivity of revenue to price. It is actually usually the case, that the inference is reversed.
A model where price is input, and sold units or revenue is output (or vice-versa) will predict (you can check that using pretty much any dataset of prices and outputs) that higher prices lead to higher outputs, because that is the association in the data. Of course we know that in truth, prices and outputs are co-determined. They are simultaneous phenomena, and regressing one on the other is not sufficient to "causally identify" the correct effect.
This is independent of how sophisticated your model is otherwise. Fitting a better non-linear representation does not help.
The solution is of course to reduce down these "endogenous" phenomena to their basic ingredients. Say you have cost data, and some demand parameters. Then, using a regression model (or NN) to predict the vector of endogenous outcome variables will work, and roughly give you the right inference.
Then, as a firm, you are able to use these (more) exogenous predictive variables to find your correct pricing.
This is not new, pops up everywhere in social science, is the basis of a gigantic literature called econometrics, and really has nothing to do with how you do the prediction.
The only thing that NN add are better predictions (better fitting) and the ability to deal with more data. As this inferential problem shows, using more (and more fine-grained) data is indeed crucial to predicting what a firm should do.
BUT, it is crucial to understand and reason about the underlying causality FIRST, because otherwise even the most sophisticated statistical approach will simply give you wrong results.
Secondly, the counterfactual data for economic issues is usually very scarce. The approach taken by machine learning is problematic, not only because of potentially wrong inference, but also because two points in time may simply not be based on comparable data-generating processes.
In fact, this is exactly the blindness that led to people missing the financial crisis. Of course, with enough data, and long enough samples, one should expect to be become pretty good at predicting economic outcomes. But experience has shown that in economics, these data are simply too scarce. The unobserved variation between two quarters, two years, two countries, two firms (etc.) is simply very large and has fat tails. This leads to spontaneous breakdowns of such predicitive models.
Taking these two issues together, we see that better non-linear function approximation is not the solution to our problems. Instead, it is a methodological improvement that must be used in conjunction with what we have learned about causality.
Indeed the literature moves into a different direction. Good economic science nowadays means to identify effects via natural experiments and other exogenous shifts that can plausibly show causality.
Of course such experiments are more rare, and more difficult, the larger the scale becomes. Which is why Macroeconomics is arguably the "worst science" in economics, while things like auctions and microstructure of markets are actually surprisingly good science (nowadays).
Doors are wide open for ML techniques, but really only to the point that they are useful in operationalizing more and better data.
Anyone trying to understand economic phenomena needs to be keenly aware of how inference can be done, which requires an understanding (or an approach to) - that is, a theory - of the underlying mechanisms.
Yes, some combination of variables/features grouped and connected with operators that correlate to an optima (some of which are parameters we can specify) that occurs immediately or after a period of lag during which other variables of the given complex system are dangerously assumed to remain constant.
> In fact, this is exactly the blindness that led to people missing the financial crisis
ML was not necessary to recognize the yield curve inversion as a strongly predictive signal correlating to subsequent contraction.
An NN can certainly learn to predict according to the presence or magnitude of a yield curve inversion and which combinations of other features.
- [ ] Exercise: Learning this and other predictive signals by cherry-picking data and hand-optimizing features may be an extremely appropriate exercise.
"This field is different because it's nonlinear, very complex, there are unquantified and/or uncollected human factors, and temporal"
Maybe we're not in agreement about whether AI and ML can do causal inference just as well if not better than humans manipulating symbols with human cognition and physical world intuition. The time is nigh!
In general, while skepticism and caution are appropriate, many fields suffer from a degree of hubris which prevents them from truly embracing stronger AI in their problem domain. (A human person cannot mutate symbol trees and validate with shuffled and split test data all night long)
> Anyone trying to understand economic phenomena needs to be keenly aware of how inference can be done, which requires an understanding (or an approach to) - that is, a theory - of the underlying mechanisms.
I read this as "must be biased by the literature and willing to disregard an unacceptable error term"; but also caution against rationalizing blind findings which can easily be rationalized as logical due to any number of cognitive biases.
Compared to AI, we're not too rigorous about inductive or deductive inference; we simply store generalizations about human behavior and predict according to syntheses of activations in our human NNs.
If you're suggesting that the information theory that underlies AI and ML is insufficient to learn what we humans have learned in a few hundred years of observing and attempting to optimize, I must disagree (regardless of the hardness or softness of the given complex field). Beyond a few combinations/scenarios, our puny little brains are no match for our department's new willing AI scientist.
> ML was not necessary to recognize the yield curve inversion as a strongly predictive signal correlating to subsequent contraction.
> An NN can certainly learn to predict according to the presence or magnitude of a yield curve inversion and which combinations of other features.
> - [ ] Exercise: Learning this and other predictive signals by cherry-picking data and hand-optimizing features may be an extremely appropriate exercise.
If the financial crisis has not yet occurred, how will the NN learn a relationship that does not exist in the data?
The exercise of cherry picking data and hand-optimizing is equivalent to applying theory to your statistical problem. It is what is required if you lack data points - using ML or otherwise. Nevertheless, we (as in humans) are bad at it. Speaking of the financial crisis. It was not AI's that picked up on it, it was some guys applying sophisticated and deep understanding of causal relationships. And that so few people did this, shows how bad we humans are at doing this implicitly and automatically by just looking at data!
> Maybe we're not in agreement about whether AI and ML can do causal inference just as well if not better than humans manipulating symbols with human cognition and physical world intuition. The time is nigh! In general, while skepticism and caution are appropriate, many fields suffer from a degree of hubris which prevents them from truly embracing stronger AI in their problem domain. (A human person cannot mutate symbol trees and validate with shuffled and split test data all night long)
ML and AI certainly can do causal inference. But then you have to do causal inference. Again, prediction on historical data is not equivalent to causal analysis, and neither is backtesting or validation. At the end of the day, AI and ML improves on predictions, but the distinction of causal analysis is a qualitative one.
> I read this as "must be biased by the literature and willing to disregard an unacceptable error term"; but also caution against rationalizing blind findings which can easily be rationalized as logical due to any number of cognitive biases.
No. My point is that for causal analysis, you have to leverage assumptions that are beyond your data set. Where these come from is besides the point. You will always employ a theory, implicitly or explicitly.
The major issue is not the we use theories, but rather that we might do it implicitly, hiding the assumptions about the DGP that allows causal inference. This is where humans are bad. Theories are just theories. With precise assumptions giving us causal identification, we are in a good position to argue where we stand.
If we just run algorithms without really understand what is going on, we are just repeating the mistakes from the last forty years!
> If you're suggesting that the information theory that underlies AI and ML is insufficient to learn what we humans have learned in a few hundred years of observing and attempting to optimize, I must disagree (regardless of the hardness or softness of the given complex field). Beyond a few combinations/scenarios, our puny little brains are no match for our department's new willing AI scientist.
All the information theory I have seen in any of the Machine Learning textbooks I have picked up is methodologically equivalent to statistics. In particular, the standard textbooks (Elements, Murhpy etc.) treatment of information theory would only allow causal identification under the exact same conditions that the statistics literature treats.
I fail to see the difference, or what AI in particular adds. The issue of causal inference is a "hot topic" in many fields, including AI, but the underlying philosophical issues are not exactly new. This includes information theory.
You seem to think that ML has somehow solved this problem. From my reading of these books, I certainly disagree. Causal inference is certainly POSSIBLE - just as in statistics, but ML doesn't give it to you for free!
In particular, note the following issue: To show causal identification, you need to make assumptions on your DGP (exogenous variation, timing, graphical causal relations ... whatever). Even if these assumptions are very implicit, they do exist. Just by looking at data, and running a model, you do not get causal inference. It can not be done "within" the system/model.
If you bake these things into your AI, then it, too makes these assumptions. There really is no difference. For example, I could imagine an AI that can identify likely exogenous variations in the data and use them to predict counterfactuals. That's probably not too far off, if it doesn't exist already. But, this is still based on the assumption that these variations are, indeed exogenous, which can never be proven within the DGP.
In contrast, I find that most "AI scientists" care very much about prediction, and very little about causal inference. I don't mean this subfield doesn't exist. But it is a subfield. In contrast, for many non-AI scientists, causal inference IS the fundamental question, and prediction is only an afterthought. ML in practice involved doing correct experiments (AB testing), at best. It will sooner or later also adopt all other causal inference techniques. But, my point stands, I have yet to see what ML adds. Enlighten me!
AI, ML and stats will merge, if they haven't already. The distinction will disappear. I believe the issues will not. I employ a lot of AI/ML techniques in my scientific work. Never have they solved the underlying issue of causal inference for me!
> AI, ML and stats will merge, if they haven't already. The distinction will disappear. I believe the issues will not.
All tools are misapplied; including economics professionals and their advice.
Here's a beautiful Venn diagram of "Colliding Web Sciences" which includes economics as a partially independent category: https://www.google.com/search?q=colliding+web+sciences&tbm=i...
A causal model is a predictive model. We must validate the error of a causal model.
Why are theoretic models hand-wavy? "That's just because noise, the model is correct." No, such a model is insufficient to predict changes in dependent variables when in the presence of noise; which is always the case. How does validating a causal model differ from validating a predictive model with historical and future data?
Yield-curve inversion as a signal can be learned by human and artificial NNs. Period. There are a few false positives in historical data: indeed, describe the variance due to "noise" by searching for additional causal and correlative relations in additional datasets.
I searched for "python causal inference" and found a few resources on the first page of search results: https://www.google.com/search?q=python+causal+inference
CausalInference: https://pypi.org/project/CausalInference/
DoWhy: https://github.com/microsoft/dowhy
CausalImpact (Python port of the R package): https://github.com/dafiti/causalimpact
"What is the best Python package for causal inference?" https://www.quora.com/What-is-the-best-Python-package-for-ca...
Search: graphical model "information theory" [causal] https://www.google.com/search?q=graphical+model+%22informati...
Search: opencog causal inference https://www.google.com/search?q=opencog+causal+inference (MOSES, PLN,)
If you were to write a pseudocode algorithm for an econometric researcher's process of causal inference (and also their cognitive processes (as executed in a NN with a topology)), how would that read?
(Edit) Something about the sufficiency of RL (Reinforcement Learning) for controlling cybernetic systems. https://en.wikipedia.org/wiki/Cybernetics
What's the point of dumping a bunch of Google results here? At least half the results are about implementations of pretty traditional etatistical / econometric inference techniques. The Rudin causal inference framework requires either randomized controlled trials or for propensity score models an essentially unverifiable separate model step.
Google's CausalImpact model, despite having been featured on Google's AI blog, is a statistical/econometric model (essentially the same as https://www.jstor.org/stable/2981553). It leaves it up to the user to find and designate a set of control variables, which has to be designated by the user to be unaffected by the treatment. This is not done algorithmically, and has very little to do with RNNs, Random Forests or regression regularization.
> If you were to write a pseudocode algorithm for an econometric researcher's process of causal inference (and also their cognitive processes (as executed in a NN with a topology)), how would that read?
[1] Set up a proper RCT, that is randomly assign the treatment to different subjects [2] Calculate the outcome diffences between the treated and untreated
For A/B testing your website, the work division between [1] and [2] might be 50-50, or at least at similar order of magnitudes.
For the questions that academic economists wrstle with, say, estimate the effect of increasing school funding / decreasing class size, the effect of shifts between tax deductions vs tax credits vs changing tax rates or bands, or of the different outcome on GDP growth and unemployment of monetary vs fiscal expansion [1] would be 99.9999% of the work, or completely impossible.
Faced with the impracticallity/impossiblility of proper experiments, academic micro-economists have typically resorted to Instrumental Variable regressions. AFAICT finding (or rather, convincing the audience that you have) a proper instrument is not very amendable to automation or data mining.
In academic macro-economics (and hence at Serious Institutions such as central banks and the IMF), the most popular approaches to building causal models in the last 3 or 4 decades have probably been 1) making a bunch of unrealistic assumpsions of the behaviour individual agents (microfoundations/DSGE models) 2) making a bunch of uninterpretable and unverifyable technical assumptions on the parameters in a generic dynamic stochastic vector process fitted to macro-aggregates (Structural VAR with "identifying restrictions") 3) manually grouping different events in different countries from different periods in history as "similar enough" to support your pet theory: lowering interest rates can lead to a) high inflation, high unemployment (USA 1970s), b) high inflation, low unemployment (Japan 1970s), b) low inflation, high unemployment (EU 2010s) c) low inflation, low unemployment (USA, Japan past 2010s)
I really don't see how a RL would help with any of this. Care to come up with something concrete?
> What's the point of dumping a bunch of Google results here? At least half the results are about implementations of pretty traditional etatistical / econometric inference techniques.
Here are some tools for causal inference (and a process for finding projects to contribute to instead of arguing about insufficiency of AI/ML for our very special problem domain here). At least one AGI implementation doesn't need to do causal inference in order to predict the outcomes of actions in a noisy field.
Weather forecasting models don't / don't need to do causal inference.
> A/B testing
Is multi-armed bandit feasible for the domain? Or, in practice, are there too many concurrent changes in variables to have any sort of a controlled experiment. Then, aren't you trying to do causal inference with mostly observational data.
> I really don't see how a RL would help with any of this. Care to come up with something concrete?
The practice of developing models and continuing on with them when they seem to fit and citations or impact reinforce is very much entirely an exercise in RL. This is a control system with a feedback loop. A "Cybernetic system". It's not unique. It's not too hard for symbolic or neural AI/ML. Stronger AI can or could do [causal] inference.
I am at loss at what you want to say to me, but let me reiterate:
Any learning model by itself is a statistical model. Statistical models are never automatically causal models, albeit causal models are statistical models.
Several causal models can be observationally equivalent to a single statistical model, but the substantive (inferential) implications on doing "an intervention" on the DGP differ.
It is therefore not enough to validate and run on a model on data. Several causal models WILL validate on the same data, but their implications are drastically different. The data ALONE provides you no way to differentiate (we say, identify) the correct causal model without any further restrictions.
By extension, it is impossible for any ML mechanism to predict unobserved interventions without being a causal model.
ML and AI models CAN be causal models, which is the case if they are based on further assumptions about the DGP. For example, they may be graphical models, SCM/SEM etc. These restrictions can be derived algorithmically, based on all sorts of data, tuning, coding and whatever. It really doesn't change the distinction between causal and statistical analysis.
The way these models become causal is based on assumptions that constitute a theory in the scientific sense. These theories can then of course also be validated. But this is not based on learning from historical data alone. You always have to impose sufficient restrictions on your model (e.g. the DGP) to make such causal inference.
This is not new, but for your benefit, I basically transferred the above from an AI/ML book on causal analysis.
AI/ML can do causal analysis, because its statistics. AI/ML are not separate from these issues, do not solve these issues ex-ante, are not "better" than other techniques except on the dimensions that they are better as statistical techniques, AND, most importantly, causal application necessarily implies a theory.
Whether this is implicit or explicit is up to the researcher, but there are dangers associated with implicit causal reasoning.
And as Pearl wrote (who is not a fan of econometrics by any means!), the issue of causal inference was FIRST raised by econometricians BASED on combining the structure of economic models with statistical inference. In the 1940's.
I mean I get the appeal to trash talk social sciences, but when it comes to causal inference, you probably picked exactly the wrong one.
You are free to disregard economic theory. But you can not claim to do causal analysis without any theory. Doing so implicitly is dangerous. Furthermore, you are wrong in the sense that economic theory has put causal inference issues at the forefront of econometric research, and is therefore good for science even if you dislike those theories.
And by the way, I can come up with a good number of (drastic, hypothetical) policy interventions that would break your inference about a market crash - an inference you only were able to make once you saw such a market crash at least once.
If this dependence is broken, your non-causal model will no longer work, because the relationship between yield curve and market crash is not a physical constant fact. What you did to make it a causal inference is implicitly assume a theory about how markets work (e.g. - as they do right now -) and that it will stay this way. Actually, you did a lot more, but that's enough.
Now, you and me, we can both agree that your model with yield curves is good enough. We could even agree that you would have found it before the financial crashes, and are a billionaire. But the commonality we agree upon is a context that defines a theory.
Some alien that has been analyzing financial systems all across the universe may disagree, saying that your statistical model is in fact highly sensitive to Earth's political, societal and natural context.
Such is the difficulty of causal analysis.
> By extension, it is impossible for any ML mechanism to predict unobserved interventions without being a causal model.
In lieu of a causal model, when I ask an economist what they think is going to happen and they aren't aware of any historical data - there is no observational data collected following the given combination of variables we'd call an event or an intervention - is it causal inference that they're doing in their head? (With their NN)
> Now, you and me, we can both agree that your model with yield curves is good enough.
Yield curves alone are insufficient due to the rate of false positives. (See: ROC curves for model evalutation just like everyone else)
> We could even agree that you would have found it before the financial crashes,
The given signal was disregarded as a false positive by the appointed individuals at the time; why?
> Some alien that has been analyzing financial systems all across the universe may disagree,
You're going to run out of clean water and energy, and people will be willing to pay for unhealthy sugar water and energy-inefficient transaction networks with a perception of greater security.
That we need Martian scientist as an approach is, IMHO, necessary because of our learned biases; where we've inferred relations that have been reinforced which cloud our assessment of new and novel solutions.
> Such is the difficulty of causal analysis.
What a helpful discussion. Thanks for explaining all of this to me.
Now, I need to go write my own definitions for counterfactual and DGP and include graphical models in there somewhere.
A further hint, here is a great book about causal analysis from a ML/AI perspective
https://mitpress.mit.edu/books/elements-causal-inference
I feel like you will benefit from reading this!
It's free!
> In lieu of a causal model, when I ask an economist what they think is going to happen and they aren't aware of any historical data - there is no observational data collected following the given combination of variables we'd call an event or an intervention - is it causal inference that they're doing in their head? (With their NN)
It's up for debate if NN's represent what is going on in our heads. But let's for a moment assume it is so.
Then indeed, an economist leverages a big set of data and assumptions about causal connections to speculate how this intervention would change the DGP (the modules in the causal model) and therefore how the result would change.
An AI could potentially do the same (if that is really what we humans do), but so far, we certainly lack the ability to program such a general AI. The reason is, in part, because we have difficulty creating causal AI models even for specialized problems. In that sense, humans are much more sophisticated right now.
It is important to note that such a hypothetical AI would create a theory, based on all sorts of data, analogies, prior research and so forth, just like economists do.
It does not really matter if a scientist, or an AI, does the theorizing. The distinction is between causal and non-causal analysis.
The value of formal theory is to lay down assumptions and tautological statements that leave no doubt about what the theory is. If we see that the theory is wrong, because we disagree on the assumptions, this is actually very good and speaks for the theory. Lot's of social sciences is plagued by "general theories" that can never really shown to be false ex ante. And given that theories can never be empirically "proven", only validated in the statistical sense, this leads to a many parallel theories of doubtful value. Take a gander into sociology if you want to see this in action.
Secondly, and this is very important, is that we learn from models. This is not often recognized. What we learn from writing down models is how mechanics or modules interact. These interactions, highly logical, are USUALLY much less doubtful than the prior assumptions. For example, if price and revenues are equilibrium phenomena, we LEARN from the model that we CAN NOT estimate them with a standard regression model!
This is exactly what lead to causal analysis in this case, because earlier we would literally regress price on quantity or production on price etc. and be happy about it. But the results were often even in the entirely wrong direction!
Instead, looking at the theory, we understood the mechanical intricacies of the process we supposedly modeled, and saw that we estimated something completely different than what we interpreted. Causal analysis, among other things, tackles this issue by asking "what it is really that we estimate here?".
> Hand-wavy theory - predicated upon physical-world models of equillibrium which are themselves classical and incomplete - without validation is preferable to empirical models? Please.
My friend, you are strawmanning.
I said,
> What we do need is proper models that are then validated, which don't necessarily need 'big data.'
Which agrees with you. I said we need both and, not one or the other.
> If the model does not fit all of the big data, the error term is higher; regardless of whether the model was pulled out of a hat in front of a captive audience or deduced though inference from actual data fed through an unbiased analysis pipeline.
Big data without a model is still only valid for the scenario the data were collected in
> If the 'black-box predictive model' has lower error for all available data, the task is then to reverse the model! Not to argue for unvalidated theory.
Certainly, but we should simultaneously recognize that any model so conceived is still only valid in the situations the data were collected in, which makes the not necessarily useful for the future. You could turn such an equation into an economic philosophy, but you'd have to do a lot more, non metric, work.
How can you possibly be arguing that we should not be testing models with all available data?
All models are limited by the data they're trained from; regardless of whether they are derived through rigorous, standardized, unbiased analysis or though laudable divine inspiration.
From https://news.ycombinator.com/item?id=19084622 :
> pandas-datareader can pull data from e.g. FRED, Eurostat, Quandl, World Bank: https://pandas-datareader.readthedocs.io/en/latest/remote_da...
> pandaSDMX can pull SDMX data from e.g. ECB, Eurostat, ILO, IMF, OECD, UNSD, UNESCO, World Bank; with requests-cache for caching data requests: https://pandasdmx.readthedocs.io/en/latest/#supported-data-p...
> All models are limited by the data they're trained from; regardless of whether they are derived through rigorous, standardized, unbiased analysis or though laudable divine inspiration.
Some of the data we have isn't training data. Purely data-driven models tend to be ensnared by Goodhart's law.
For example, suppose we're issuing 30-year term loans and we have some data that shows that people with things like country club memberships and foie gras on their credit card statements have a higher tendency not to miss payments. So we use that information to make our determination.
But people are aware we're doing this and the same data is externally available, so now people start to waste resources on extravagant luxuries in order to qualify for a loan or a low interest rate, and that only makes it more likely that they ultimately default. However, that consequence doesn't become part of the data set until years have passed and the defaults actually occur, and in the meantime we're using flawed reasoning to issue loans. When we finally figure that out after ten years, the new data we use then will have some fresh different criteria for people to game, because the data is always from the past rather than the future.
We've already seen the kind of damage this can do. Politicians see data that college-educated people are better off so subsidize college loans, only to discover that the signal from having a degree that caused it to result in such gainful employment is diluted as it becomes more common, and subsidizing loans results in price inflation, and making a degree a prerequisite for jobs that shouldn't require it creates incentives for degree mills that pump out credentials but not real education.
To get out of this we have to consider not only what people have done in the past but how they are likely to respond to a given policy change, for which we have no historical data prior to when the policy is enacted, and so we need to make those predictions based on logic in addition to data or we go astray.
> To get out of this we have to consider not only what people have done in the past but how they are likely to respond to a given policy change, for which we have no historical data prior to when the policy is enacted, and so we need to make those predictions based on logic in addition to data or we go astray.
"Pete, it's a fool who looks for logic in the chambers of the human heart."
Logically, we might have said "prohibition will reduce substance abuse harms" but the actual data indicates that margins increased. Then, we look at the success of Portugal's decriminalization efforts and cannot at all validate our logical models.
Similarly, we might've logically claimed that "deregulation of the financial industry will help everyone" or "lowering taxes will help everyone" and the data does not support.
So, while I share the concerns about Responsible AI and encoding biases (and second-order effects of making policy recommendations according to non-causal models without critically, logically thinking first) I am very skeptical about our ability to deduce causal relations without e.g. blind, randomized, longitudinal, interventional studies (which are unfortunately basically impossible to do with [economic] policy because there is no "ceteris paribus")
https://personalmba.com/second-order-effects/
"Causal Inference Book" https://news.ycombinator.com/item?id=17504366
> https://www.hsph.harvard.edu/miguel-hernan/causal-inference-...
> Causal inference (Causal reasoning) https://en.wikipedia.org/wiki/Causal_inference ( https://en.wikipedia.org/wiki/Causal_reasoning )
The virtue of logic isn't that your model is always correct or that it should be adhered to without modification despite contrary evidence, it's that it allows you to have one to begin with. It's a method of choosing which experiments to conduct. If you think prohibition will reduce substance abuse but then you try it and it doesn't, well, you were wrong, so end prohibition.
This is also a strong argument for "laboratories of democracy" and local control -- if everybody agrees what to do then there is no dispute, but if they don't then let each local region have their own choice, and then we get to see what happens. It allows more experiments to be run at once. Then in the worst case the damage of doing the wrong thing is limited to a smaller area than having the same wrong policy be set nationally or internationally, and in the best case different choices are good in different ways and we get more local diversity.
> If you think prohibition will reduce substance abuse but then you try it and it doesn't, well, you were wrong, so end prohibition.
Maybe we're at a local optima, though. Maybe this is a sign that we should just double down, surge on in there and get the job done by continuing to do the same thing and expecting different results. Maybe it's not the spec but the implementation.
Recommend a play according to all available data, and logic.
> This is also a strong argument for "laboratories of democracy" and local control -- if everybody agrees what to do then there is no dispute, but if they don't then let each local region have their own choice, and then we get to see what happens. It allows more experiments to be run at once. Then in the worst case the damage of doing the wrong thing is limited to a smaller area than having the same wrong policy be set nationally or internationally, and in the best case different choices are good in different ways and we get more local diversity.
"Adjusting for other factors," the analysis began.
- [ ] Exercise / procedure to be coded: Brainstorm and identify [non-independent] features that may create a more predictive model (a model with a lower error term). Search for confounding variables outside of the given data.
The New York Times course to teach its reporters data skills is now open-source
It's more work to verify all formulas that reference unnamed variables in a spreadsheet than to review the code inputs and outputs in a notebook.
"Teaching Pandas and Jupyter to Northwestern journalism students" [in DC] https://www.californiacivicdata.org/2017/06/07/dc-python-not...
> http://www.firstpythonnotebook.org/
You can also develop d3.js visualizations — just like NYT — with jupyter notebooks and whichever language(s).
"Data-Driven Journalism" ("ddj") https://en.wikipedia.org/wiki/Data-driven_journalism
http://datadrivenjournalism.net/
"The Data Journalism Handbook 1" https://datajournalism.com/read/handbook/one
"The Data Journalism Handbook 2" https://datajournalism.com/read/handbook/two
While there are a number of ScholarlyArticle journals that can publish notebooks, I'm not aware of any newspapers that are prepared to publish notebooks as NewsArticles. It's pretty easy to `jupyter convert --to html` and `--to markdown` or just 'Save as'
Regarding expressing facts as verifiable claims with structured data in HTML and/or blockchains: "Fact Checks" https://news.ycombinator.com/item?id=15529140
Does this course recommend linking to every source dataset and/or including full citations (with DOI) in the article? Does this course recommend getting a free DOI for the published revision of an e.g. GitHub project repository (containing data, and notebooks and/or the article text) with Zenodo?
No Kings: How Do You Make Good Decisions Efficiently in a Flat Organization?
Group decision-making > Formal systems: https://en.wikipedia.org/wiki/Group_decision-making#Formal_s...
> Consensus decision-making, Voting-based methods, Delphi method, Dotmocracy
Consensus decision-making: https://en.wikipedia.org/wiki/Consensus_decision-making
There's a field that some people are all calling "Collaboration Engineering". I learned about this from a university course in Collaboration.
6 Patterns of Collaboration [GRCOEB] — Generate, Reduce, Clarify, Organize, Evaluate, Build Consensus
7 Layers of Collaboration [GPrAPTeToS] — Goals, Products, Activities, Patterns of Collaboration, Techniques, Tools, Scripts
The group decision making processes described in the article may already be defined with the thinkLets design pattern language.
A person could argue against humming for various unspecified reasons.
I'll just CC this here from my notes, which everyone can read here [1]:
“Collaboration Engineering: Foundations and Opportunities” de Vreede (2009) http://aisel.aisnet.org/jais/vol10/iss3/7/
“A Seven-Layer Model of Collaboration: Separation of Concerns for Designers of Collaboration Systems” Briggs (2009) http://aisel.aisnet.org/icis2009/26/
Six Patterns of Collaboration “Defining Key Concepts for Collaboration Engineering” Briggs (2006) http://aisel.aisnet.org/amcis2006/17/
“ThinkLets: Achieving Predictable, Repeatable Patterns of Group Interaction with Group Support Systems (GSS)” http://www.academia.edu/259943/ThinkLets_Achieving_Predictab...
https://scholar.google.com/scholar?q=thinklets
[1] https://wrdrd.github.io/docs/consulting/team-building#collab...
4 Years of College, $0 in Debt: How Some Countries Make Education Affordable
It at least makes sense to pay for doctors and nurses to go to school, right? If you want to care for others and you do the work to earn satisfactory grades, I think that investing in your education would have positive ROI.
We had plans here in the US to pay for two years of community college for whoever ("America's College Promise"). IDK what happened to that? We should have called it #ObamaCollege so that everyone could attack corporate welfare and bad investments with no ROI.
New York has the Excelsior scholarship for CUNY and SUNY. Tennessee pays for college with lottery proceeds. Are there other state-level efforts to fund higher education in the US such that students can finish school debt-free or close to it?
There are MOOCs (online courses) which are worth credit hours for the percentage of people that commit to finishing the course. https://www.classcentral.com/
Khan Academy has free SAT, MCAT, NCLEX-RN, GMAT, and LSAT test prep and primary and supplementary learning resources. https://www.khanacademy.org/test-prep
Free education: https://en.wikipedia.org/wiki/Free_education
Ask HN: What jobs can a software engineer take to tackle climate change?
I'm a software engineer with a diverse background in backend, frontend development.
How do I find jobs related to tackling global warming and climate change in Europe for an English speaker?
Open to ideas and thoughts.
> I'm a software engineer with a diverse background in backend, frontend development.
> How do I find jobs related to tackling global warming and climate change in Europe for an English speaker?
While not directly answering the question, here are some ideas for purchasing, donating, creating new positions, and hiring people that care:
Write more efficient code. Write more efficient compilers. Optimize interpretation and compilation so that the code written by people with domain knowledge who aren't that great at programming who are trying to solve other important problems is more efficient.
Push for PPAs (Power Purchase Agreements) that offset energy use. Push for directly sourcing clean energy.
Use services that at least have 100% PPAs for the energy they use: services that run on clean energy sources.
Choose green datacenters.
- [ ] Add the capability for cloud resource schedulers like Kubernetes and Terraform to prefer or require clean energy datacenters.
Choose to work with companies that voluntarily choose to do sustainability reporting.
Work to help develop (and popularize) blockchain solutions that are more energy efficient and that have equal or better security assurances as less efficient chains.
Advocate for clean energy. Donate to NGOs working for our environment and for clean energy.
Invest in clean energy. There are a number of clean energy ETFs, for example. Better energy storage is a good investment.
Push for certified green buildings and datacenters.
- [ ] We should create some sort of a badge and structured data (JSONLD, RDFa, Microdata) for site headers and/or footers that lets consumers know that we're working toward '200% green' so that we can vote with our money.
Do not vote for people who are rolling back regulations that protect our environment. Pay an organization that pays lobbyists to work the system: that's the game.
Help explain why it's both environment-rational and cost-rational to align with national and international environmental sustainability and clean energy objectives.
Argue that we should make external costs internal in order that markets will optimize for what we actually want.
Thermodynamics is part of the physics curriculum for many software engineering and computer science degrees.
There are a number of existing solutions that solve for energy inefficiency due to unreclaimed waste heat.
"Thermodynamics of Computation Wiki" https://news.ycombinator.com/item?id=18146854
"Why Do Computers Use So Much Energy?" https://news.ycombinator.com/item?id=18139654
YC's request for startups: Government 2.0
There's money to be earned in solving for the #GlobalGoals Goals, Targets, and Indicators:
The Global Goals
1. No Poverty
2. Zero Hunger
3. Good Health & Well-Being
4. Quality Education
5. Gender Equality
6. Clean Water & Sanitation
7. Affordable & Clean Energy
8. Decent Work & Economic Growth
9. Industry, Innovation & Infrastructure
10. Reduced Inequalities
11. Sustainable Cities and Communities
12. Responsible Consumption & Production
13. Climate Action
14. Life Below Water
15. Life on Land
16. Peace and Justice & Strong Institutions
17. Partnerships for the Goals
Almost 40% of Americans Would Struggle to Cover a $400 Emergency
I always wonder what proportion of that group is due to insufficient income, and what proportion is due to terrible financial literacy.
> I always wonder what proportion of that group is due to insufficient income
According to the Social Security Administration [1]:
2017 Average net compensation: 48,251.57
2017 Median net compensation: 31,561.49
The FPL (Federal Poverty Level) income numbers for Medicaid and the Children's Health Insurance Program (CHIP) eligibility [2]:
>> $12,140 for individuals, $16,460 for a family of 2, $20,780 for a family of 3, $25,100 for a family of 4, $29,420 for a family of 5, $33,740 for a family of 6, $38,060 for a family of 7, $42,380 for a family of 8
Wages are not keeping up with corporate profits. That can't all be due to automation.
The minimum wage is only one factor linked to price inflation. We can raise wages and still keep inflation down to an ideal range.
Maybe it's that we don't understand what it's like to live on $12K or $32K a year (without healthcare due to lack of Medicaid expansion; due to our collective failure to instill charity as a virtue and getting people back on their feet as a good investment). How could we learn (or remember!) about what it's like to be in this position (without zero-interest bank loans to bail us out)?
> and what proportion is due to terrible financial literacy.
The r/personalfinance wiki is one good resource for personal finance. From [3]:
>> Personal Finance (budgets, interest, growth, inflation, retirement)
Personal Finance https://en.wikipedia.org/wiki/Personal_finance
Khan Academy > College, careers, and more > Personal finance https://www.khanacademy.org/college-careers-more/personal-fi...
"CS 007: Personal Finance For Engineers" https://cs007.blog
https://reddit.com/r/personalfinance/wiki
... How can we make personal finance a required middle and high school curriculum component? [4]
"What are some ways that you can save money in order to meet or exceed inflation?"
Dave Ramsey's 7 Baby Steps to financial freedom [5] seem like good advice? Is the debt snowball method ideal for minimizing interest payments?
[1] https://www.ssa.gov/OACT/COLA/central.html
[2] https://www.healthcare.gov/glossary/federal-poverty-level-fp...
[3] "Ask HN: How can you save money while living on poverty level?" https://news.ycombinator.com/item?id=18894582
[4] "Consumer science (a.k.a. home economics) as a college major" https://news.ycombinator.com/item?id=17894632
Congress should grow the Digital Services budget, it more than pays for itself
> The U.S. Digital Service isn’t perfect, but it is clearly working. The team estimates that for every $1 million invested in USDS that the government will avoid spending $5 million and save thousands of labor hours. Over a five-year period, the team’s efforts will save $1.1 billion, redirect almost 2,000 labor years towards higher value work, and generate over 400 percent return on investment. Most importantly, USDS will continue to deliver better government services for the American people, including Veterans who deserve better.
> In the private sector, these kinds of numbers would not lead to a 50 percent cut in budget. Instead, you’d clearly invest further with that kind of return. Considering the ambitious goals set out in the President’s Management Agenda, the Trump Administration should double down on better support for the public, our troops, and our veterans. The best way to do that is clearly through investments like USDS.
Why would you halve the budget of a team that's yielding a more than 400% ROI (in terms of cost savings)?
https://en.wikipedia.org/wiki/United_States_Digital_Service
400% ROI to whom?
More data makes hiding corruption harder, so I'm not sure if it really gives 400% to the decision makers.
USDS reports 400% ROI in savings to the taxpayers who fund the government with tax revenue (instead of kicking the can down the road with debt financing) and improvements in customer service quality.
https://www.usaspending.gov (Federal Funding Accountability and Transparency Act of 2006 (Obama, McCain, Carper, Coburn)) has more fine-grained spending data, but not credit-free immutable distributed ledger transaction IDs, quantitative ROI stats, or performance.gov and #globalgoals goal alignment. We'd need a metadata field on spending bills to link to performance.gov and SDG Goals, Targets, and Indicators.
"Transparency and Accountability"
IIRC, here on HN, I've mentioned a number of times -- and quoted in the full from -- the 13 plays of the USDS Digital Services Playbook; all of which are applicable to and should probably be required reading for all government IT and govtech: https://playbook.cio.gov/
There are forms with workflow states that need human review sometimes. USDS helps with getting those processes online in order to reduce costs, increase cost-efficiency, and increase quality of service.
The Trillion-Dollar Annual Interest Payment
> Given the recent actions of Congress, and the years of prior inaction in changing the nation’s fiscal path, the U.S. government’s annual interest payment will eclipse annual defense spending in only six years. By 2025, annual interest costs on the national debt will reach $724 billion, while annual defense spending will reach $706 billion. To put that into perspective, in the 2018 fiscal year, the U.S. government spent $325 billion in interest payments and spent $622 billion in defense (Exhibit 2).
Why would you cut taxes and debt finance our nation's future?
Oak, a Free and Open Certificate Transparency Log
Great use case for blockchain technology
CT logs are already chained
> Great use case for blockchain technology
>> CT logs are already chained
Trillian is a centralized Merkle tree: it doesn't support native replication (AFAIU?) and there is a still a password that can delete or recreate the chain (though we can track for any such inappropriate or errant modifications (due to e.g. solar flares) by manually replicating and verifying every entry in the chain, or trusting that everything before whatever we consider to be a known hash (that could be colliding) is unmodified (since the last time we never verified those entries)).
According to the trillian README, trillian depends upon MySQL/MariaDB and thus internal/private replication is as good as the SQL replication model (which doesn't have a distributed consensus algorithm like e.g. paxos).
A Merkle tree alone is not a blockchain; though it provides more assurance of data integrity than a regular tree, verifying that the whole chain of hashes actually is good and distributed replication without configuring e.g. SSL certs are primary features of blockchains.
There are multiple certificate issuers, multiple logs, and multiple log verifiers. With no single point of failure, that doesn't sound centralized to me?
Which components of the system are we discussing?
PKI is necessarily centralized: certs depend upon CA certs which can depend upon CA certs. If any CA is compromised (e.g. by theft or brute force (which is inestimably infeasible given current ASIC resources' preference for legit income)) that CA can sign any CRL. A CT log and a CT log verifier can help us discover that a redundant and so possibly unauthorized cert has been issued for a given domain listed in an x.509 cert CN/SAN.
The CT log itself - trillian, for Google and now LetsEncrypt, too - though, runs on MySQL; which has one root password.
The system of multiple independent, redundant CT logs is built upon databases that depend upon presumably manually configured replication keys.
Does my browser call a remote log verifier API over (hopefully pinned with a better fingerprint than MD5) HTTPS?
There are multiple issuers, so from an availability point of view, if one is down, you could choose another. They submit to at least two logs, so if one log is unavailable you could read the other one. This is a form of decentralization.
Now, from a security point of view, it only takes breaking into one issuer to issue bad certificates. But maybe classifying everything as either centralized or decentralized is too simple?
Centralized and decentralized are overloaded terms. We could argue that every system that depends upon DNS is a centralized (and thus has a single point of failure).
We could describe replication models as centralized or decentralized. Master/master SQL replication is still not decentralized (regardless of whether there are multiple A records or multiple static IPs configured in the client).
With PKI, we choose the convenience of trusting a CA bundle over having to manually check every cert fingerprint.
Whether a particular chain is centralized or decentralized is often bandied about. When there are a few mining pools that effectively choose which changes are accepted, that's not decentralized either.
That there are multiple redundant independent CT logs is a good thing.
How do I, as a concerned user, securely download (and securely mirror?) one or all of the CT logs and verify that none of the record hashes don't depend upon the previous hash? If the browser relies upon a centralized API for checking hash fingerprints, how is that decentralized?
Looks like there is a bit here about how to get started: https://security.stackexchange.com/questions/167366/how-can-...
Most people aren't going to do it, but I think that's not really the point, any more than every user needs to review Linux kernel patches. But I wonder if there are enough "eyes" on this and how would we check?
Death rates from energy production per TWh
Apparently the deaths are justified because energy.
Are the subsidies and taxes (incentives and penalties) rational in light of the relative harms of each form of energy?
"Study: U.S. Fossil Fuel Subsidies Exceed Pentagon Spending" https://www.rollingstone.com/politics/politics-news/fossil-f...
> The IMF found that direct and indirect subsidies for coal, oil and gas in the U.S. reached $649 billion in 2015. Pentagon spending that same year was $599 billion.
> The study defines “subsidy” very broadly, as many economists do. It accounts for the “differences between actual consumer fuel prices and how much consumers would pay if prices fully reflected supply costs plus the taxes needed to reflect environmental costs” and other damage, including premature deaths from air pollution.
IDK whether they've included the costs of responding to requests for help with natural disasters that are more probable due to climate change caused by these "externalties" / "external costs" of fossil fuels.
Energy saves lives so some risk probably is justified.
Why isn't the market choosing the least harmful, least lethal energy sources? Energy is for the most part entirely substitutable: switching costs for consumers like hospitals are basically zero.
(Everyone is free to invest in clean energy at any time)
Switching costs for the entire society are far from zero.
100% Renewable Energy https://en.wikipedia.org/wiki/100%25_renewable_energy
> The main barriers to the widespread implementation of large-scale renewable energy and low-carbon energy strategies are political rather than technological. According to the 2013 Post Carbon Pathways report, which reviewed many international studies, the key roadblocks are: climate change denial, the fossil fuels lobby, political inaction, unsustainable energy consumption, outdated energy infrastructure, and financial constraints.
We need to make the external costs of energy production internal in order to create incentives to prevent these fossil fuel deaths and other costs.
Use links not keys to represent relationships in APIs
A thing may be identified by a URI (/person/123) for which there are zero or more URL routes (/person/123, /v1/person/123). Each additional route complicates caching; redirects are cheap for the server but slower for clients.
JSONLD does define a standard way to indicate that a value is a link: @id (which can be specified in a/an @context) https://www.w3.org/TR/json-ld11/
One additional downside to storing URIs instead of bare references is that it's more complicated to validate a URI template than a simple regex like \d+ or [abcdef\=\d+]+
And on a more fundamental standpoint I get that disk space is cheap theses days. But you just doubled, if not worse, the storage space required for a key for a vague reason.
It may never make any difference on a small dataset, where storage was anyway unaware of differences between integer and text. But it would be hiding in the dark. And maybe in a few year a new recruit will have the outstanding idea to convert text-link to bigint to save some space...
No Python in Red Hat Linux 8?
/usr/bin/python can point to either /usr/bin/python3 or (as PEP 394 currently recommends) /usr/bin/python2
$ alternatives --config python
JMAP: A modern, open email protocol
What are the optimizations in JMAP that make it faster than, say, Solid? Solid is built on a bunch of W3C Web, Security, and Linked Data Standards; LDP: Linked Data Protocol, JSON-LD: JSON Linked Data, WebID-TLS, REST, WebSockets, LDN: Linked Data Notifications. [1][2] Different worlds, I suppose.
There's no reason you couldn't represent RFC5322 data with RDF as JSONLD. There's now a way to do streaming JSON-LD.
LDP does paging and querying.
Solid supports pubsub with WebSockets and LDN. It may or may not (yet?) be as efficient for synchronization as JMAP, but it's definitely designed for all types of objects with linked data web standards; and client APIs can just parse JSON-LD.
[1] https://github.com/solid/information#solid-specifications
[2] https://github.com/solid/solid-spec/issues/123 "WebSockets and HTTP/2" SSE (Server-Side Events)
JMAP: JSON Meta Application Protocol https://en.wikipedia.org/wiki/JSON_Meta_Application_Protocol
Is there a OpenAPI Specification for JMAP? There are a bunch of tools for Swagger / OpenAPIs: DRY interactive API docs, server implementations, code generators: https://swagger.io/tools/open-source/ https://openapi.tools/
Does JMAP support labels; such that I don't need to download a message and an attachment and mark it as read twice like labels over IMAP?
How does this integrate with webauthn; is that a different layer?
(edit) Other email things: openpgpjs; Web Key Directory /.well-known/openpgpkey/*; if there's no webserver on the MX domain, you can use the ACME DNS challenge to get free 3-month certs from LetsEncrypt.
I would say your comment, and the first reply, both demonstrate quite effectively why JMAP is probably a better choice for email.
> It may or may not (yet?) be as efficient for synchronization as JMAP, but it's definitely designed for all types of objects
If we hypothetically allow for equal adoption & mindshare of both, and assume both are non-terrible designs, I'd guess the one designed for "all types of objects" is less likely to ever be as efficient as the one designed with a single use-case in mind.
And narrow focus is not only good for optimising specific use-cases, it's also good for adoption as people immediately understand what your protocol is for and how to use it when it has a single purpose and a single source of truth for reference spec, rather than a series of disparate links and vague all-encompassing use-cases.
Solid has brilliant people behind it, but it's too broad, too ambitious, and very much lacks focus, and that will impair adoption because it isn't the "one solution" for anyone's "one problem".
--
To take another perspective on this, there are other commenters in this thread bemoaning the loss of non-HTTP-based protocols. Funnily enough, HTTP itself is a broadly used, broadly useful protocol than can be used for pretty much anything (and had TBL behind it also). The big difference was that Tim wasn't proposing that HTTP be the solution to all our internet problems and needs in 1989—it was just for hypertext documents. It's only now, post-adoption, that it is used for so much more than that.
> If we hypothetically allow for equal adoption & mindshare of both, and assume both are non-terrible designs, I'd guess the one designed for "all types of objects" is less likely to ever be as efficient as the one designed with a single use-case in mind.
This is a generalization that is not supported by any data.
Standards enable competing solutions. Competing solutions often result in performance gains and efficiency.
Hopefully, there will be performant implementations and we won't need to reinvent the wheel in order to synchronize and send notifications for email, contacts, and calendars.
> There's no reason you couldn't represent RFC5322 data with RDF as JSONLD. There's now a way to do streaming JSON-LD.
Is there a reason you'd want to? I clicked all your links but I still have no idea what Solid is.
To eliminate the need for domain-specific parser implementations on both server and client, make it easy to index and search this structured data, and to link things with URIs and URLs like other web applications that also make lots of copies.
Solid is a platform for decentralized linked data storage and retrieval with access controls, notifications, WebID + OAuth/OpenID. The Wikipedia link and spec documents have a more complete description that could be retrieved and stored locally.
Grid Optimization Competition
From "California grid data is live – solar developers take note" https://news.ycombinator.com/item?id=18855820 :
>> It looks like California is at least two generations of technology ahead of other states. Let’s hope the rest of us catch up, so that we have a grid that can make an asset out of every building, every battery, and every solar system.
> +1. Are there any other states with similar grid data available for optimization; or any plans to require or voluntarily offer such a useful capability?
How do these competitions and the live actual data from California-only (so far; AFAIU) compare?
Are there standards for this grid data yet? Without standards, how generalizable are the competition solutions to real-world data?
Blockchain's present opportunity: data interchange standardization
What are the current standards efforts for blockchain data interchange?
W3C JSON-LD, ld-signatures + lds-merkleproof2017 (normalize the data before signing it so that the signature is representation-independent (JSONLD, RDFa, RDF, n-triples)), W3C DID Decentralized Identifiers, W3C Verifiable Claims, Blockcerts.org
W3C Credentials Community Group: https://w3c-ccg.github.io/community/work_items.html#draft-sp... (DID, Multihash (IETF), [...])
"Blockchain Credential Resources; a gist" https://gist.github.com/westurner/4345987bb29fca700f52163c33...
Specifically for payments:
https://www.w3.org/TR/?title=payment (the W3C Payment Request API standardizes browser UI payment/checkout workflows)
ILP: Interledger Protocol https://interledger.org/rfcs/0027-interledger-protocol-4/
> W3C JSON-LD
https://www.w3.org/TR/json-ld/ (JSON-LD 1.0)
https://www.w3.org/TR/json-ld11/ (JSON-LD 1.1)
> ld-signatures + lds-merkleproof2017 (normalize the data before signing it so that the signature is representation-independent (JSONLD, RDFa, RDF, n-triples))
https://w3c-dvcg.github.io/ld-signatures/
https://w3c-dvcg.github.io/lds-merkleproof2017/ (2017 Merkle Proof Linked Data Signature Suite)
> W3C DID Decentralized Identifiers
https://w3c-ccg.github.io/did-primer/
>> A Decentralized Identifier (DID) is a new type of identifier that is globally unique, resolveable with high availability, and cryptographically verifiable. DIDs are typically associated with cryptographic material, such as public keys, and service endpoints, for establishing secure communication channels. DIDs are useful for any application that benefits from self-administered, cryptographically verifiable identifiers such as personal identifiers, organizational identifiers, and identifiers for Internet of Things scenarios. For example, current commercial deployments of W3C Verifiable Credentials heavily utilize Decentralized Identifiers to identify people, organizations, and things and to achieve a number of security and privacy-protecting guarantees.
> W3C Verifiable Claims
https://github.com/w3c/verifiable-claims
https://w3c.github.io/vc-data-model/ (Data Model)
https://w3c.github.io/vc-use-cases/ (Use Cases: Education, Healthcare, Professional Credentials, Legal Identity,)
Ask HN: Value of “Shares of Stock options” when joining a startup
I got an offer from a US start-up (well +25 employees) which has an office in EU where I would join them.
The offer's base salary is good (ie. higher than average for senior positions for that location) but I intend to negotiate it further, as I have possible other options. patio11's negotiation guide was a great read in that regard.
However, I'm relocating from a non-EU/US country, and I don't have a single idea about the financial systems, stock markets, and how to evaluate "15k shares of stock-options" or what "Stock Option and Grant Plan" means, I'm asking you fellow HNers about this part.
Do I just treat them as worthless and focus on base salary (as some internet sources suggest) or is there a formula to evaluate what they would be worth in say 2 years for instance ?
There are a number of options/equity calculators:
https://tldroptions.io/ ("~65% of companies will never exit", "~15% of companies will have low exits*", "~20% of companies will make you money")
https://comp.data.frontapp.com/ "Compensation and Equity Calculator"
http://optionsworth.com/ "What are my options worth?"
http://foundrs.com/ "Co-Founder Equity Calculator"
CMU Computer Systems: Self-Grading Lab Assignments (2018)
These look fun; in particular the "Attack Lab".
Dockerfiles might be helpful and easy to keep updated. Alpine Linux or just busybox are probably sufficient?
The instructor set could extend FROM the assignment image and run a few tests with e.g. testinfra (pytest)
You can also test code written in C with gtest.
I haven't read through all of the materials: are there suggested (automated) fuzzing tools? Does OSS-Fuzz solve?
Are there references to CWE and/or the SEI CERT C Coding Standard rules? https://wiki.sei.cmu.edu/confluence/plugins/servlet/mobile?c...
"How could we have changed our development process to catch these bugs/vulns before release?"
"If we have 100% [...] test coverage, would that mean we've prevented these vulns?"
What about 200%?
What on earth would 200% test coverage mean?
Coverage for all the code that was written, and all the code that has yet to be typed, of course.
All thanks to quantum computing :)
Show HN: Debugging-Friendly Tracebacks for Python
pytest also has helpful tracebacks; though only for test runs.
With nose-progressive, you can specify --progressive-editor or update the .noserc so that traceback filepaths are prefixed with your preferred editor command.
vim-unstack parses paths from stack traces / tracebacks (for a number of languages including Python) and opens each in a split at that line number. https://github.com/mattboehm/vim-unstack
Here's the Python regex from my hackish pytb2paths.sh script:
'\s+File "(?P<file>.*)", line (?P<lineno>\d+), in (?P<modulestr>.*)$'
Why isn't 1 a prime number?
Funny true story about this: One time a man came up to me and handed me his phone, and asked me to call his mom. He was about to pass out and he asked me not call an ambulance (God bless America) , he appeared to have a concussion. By this time there was about 10-15 people around him and he could barely talk. I asked him what his phone code was and instead of just giving it to me he said "it's the first four prime numbers". Immediately, about five people shout "1,2,3,5". I am no longer holding the phone, because I handed to someone else to make sure it was okay. Sure enough, I was in a mathematical proofs class and we had just discussed this topic. So, I say "one is not a prime number". Of course, we get the phone unlocked in the second try with "2,3,5,7" and the guys mom is on the way. Everyone thought I was a genius a hero.
You can also dial emergency contacts without unlocking the phone. They are accessible from the medical ID page on iOS, I assume Android has similar.
> You can also dial emergency contacts without unlocking the phone. They are accessible from the medical ID page on iOS, I assume Android has similar.
You can set a Lock Screen Message by searching for "Lock Screen Message" in the Android Settings.
You can also create an "ICE (In Case of Emergency)" contact.
How do we know when we’ve fallen in love? (2016)
Where is the feeling felt? Is there an associated color or a shape or a kinesiologic position?
"Love styles" https://en.wikipedia.org/wiki/Love_styles
"Greek words for love" https://en.wikipedia.org/wiki/Greek_words_for_love
Rare and strange ICD-10 codes
These are too funny: https://www.empr.com/home/features/the-strangest-and-most-ob...
Choice selection:
V91.07X - Burn Due to Water Skis on Fire(?!)
W61.42XA. Struck By Turkey, initial encounter. If a duck is involved, there's a code for that too. (W61.62X)
R46.1. Bizarre Personal Appearance.
My favorite: T63.012D: Toxic effect of rattlesnake venom, intentional self-harm, subsequent encounter
So basically you have a case where someone not only intentionally wanted to harm themselves with rattlesnake venom once, but at least twice!
> So basically you have a case where someone not only intentionally wanted to harm themselves with rattlesnake venom once, but at least twice!
No, you misunderstand the terminology. "Subsequent encounter" means with the doctor not with the rattlesnake. AKA followup care during or after recovery.
> No, you misunderstand the terminology. "Subsequent encounter" means with the doctor not with the rattlesnake
You can reference ICD codes with the schema.org/code property of schema.org/MedicalEntity and subclasses. https://schema.org/docs/meddocs.html
"Subsequent encounter" is poorly defined. IMHO, there should be a code for this.
> "Subsequent encounter" is poorly defined.
"Poorly defined" is poorly defined. Explanations of when to use the D make perfect sense to me.
"The 7th character for “subsequent encounter” is to be used for all encounters after the patient has received active treatment of the condition and is receiving routine care for the condition during the healing or recovery phase. Examples of subsequent encounters include cast change or removal, x-ray to check healing status of a fracture, removal of external or internal fixation device, medication adjustments, and other aftercare and follow-up visits following active treatment of the injury or condition. Encounters for rehabilitation, such as physical and occupational therapy, are another example of the use of the “subsequent encounter” 7th character. For aftercare following an injury, the acute injury code should be assigned with the 7th character for subsequent encounter."
This parses the ICD 10 CM with lxml: https://github.com/westurner/pycd10api/blob/master/pycd10api...
Python Requests III
So what's the difference to the predecessor?
asyncio, HTTP/2, connection pooling, timeouts, Python 3.6+
README > "Feature Support" https://github.com/kennethreitz/requests3/blob/master/README...
https://github.com/aio-libs/aiohttp already supports asyncio. Why do I need this for asyncio?
Anyway, I see that Issues are disabled for the repo. Is that the new way to develop? /s
Tools built with aiohttp: https://libraries.io/pypi/aiohttp/usage
Tools built with requests: https://libraries.io/pypi/requests/usage
Post-surgical deaths in Scotland drop by a third, attributed to a checklist
My former roommate is a pilot. When I first met him, I noticed that he uses checklists for just about everything, even the most basic everyday tasks.
After some time, I decided to apply that same mentality to my own life. Both in private and work situations.
I get it now. Checklists reduce cognitive load tremendously well, even for basic tasks. As an example: I have a checklist for when I need to travel, it contains stuff like what to pack, asking someone to feed my cat, check windows are closed, dishwasher empty, heating turned down, etc. Before the checklist, I would always be worried I forgot something, now I can relax.
Also, checklists are a great way to improve processes. Basically a way to debug your life. For instance: I once forgot to empty the trash bin before a long trip, I added that to my checklist and haven't had a smelly surprise ever since ;)
This is described in the book called "the checklist manifesto". Very good book by the way.
https://www.amazon.com/Checklist-Manifesto-How-Things-Right/...
The medical community uses checklists. They've been using checklists for a while. A number of studies found that they only help for a few months, while they're new.
That said, "the medical community" is not a homogeneous monolith, and you can absolutely find regional variation in what checklists are used for, how detailed they are, how closely they're followed, how people are accountable for keeping to them, etc.
"The Checklist Manifesto" chose to overlook the studies about how transient the benefit of checklists is.
Are these paper based? Any checklist apps out there for our daily use?
GitHub and GitLab support task checklists in Markdown and also project boards which add and remove labels like 'ready' and 'in progress' when cards are moved between board columns; like kanban:
- [ ] not complete
- [x] completed
Other tools support additional per-task workflow states:
- [o] open
- [x (2019-04-17)] completed on date
I worked on a large hospital internal software project where the task was to build a system for reusable checklists editable through the web that prints them out in duplicate or triplicate at nearby printers. People really liked having the tangible paper copy.
"The Checklist Manifesto" by Atul Gawande was published while I worked there. TIL pilots have been using checklists for process control in order to reduce error for many years.
Evernote, RememberTheMilk, Google Tasks, and Google Keep all support checklists. Asana and Gitea and TaskWarrior support task dependencies.
A person could carry around a Hipster PDA with Bullet Journal style tasks lists with checkboxes; printed from a GTD service with an API and a @media print CSS stylesheet: https://en.wikipedia.org/wiki/Hipster_PDA
I'm not aware of very many tools that support authoring reusable checklists with structured data elements and data validation.
...
There are a number of configuration management systems like Puppet, Chef, Salt, and Ansible that build a graph of completable and verifiable tasks and then depth-first traverse said graph (either with hash randomization resulting in sometimes different traversals or with source order as an implicit ordering)
Resource scheduling systems like operating systems and conference room schedulers can take ~task priority into account when optimally ordering tasks given available resources; like triage.
Scheduling algorithms: https://news.ycombinator.com/item?id=15267146
TodoMVC catalogs Todo list implementations with very many MV* JS Frameworks: http://todomvc.com
Reusable Checklists could be done with a simple text document that you duplicate every time you need it no ?
For sure. Though many tools don't read .txt (or .md/.markdown) files.
GitHub and GitLab support (multiple) Issue and Pull Request templates:
Default: /.github/ISSUE_TEMPLATE.md || Configure in web interface
/.github/ISSUE_TEMPLATE/Name.md || /.gitlab/issue_templates/Name.md
Default: /.github/PULL_REQUEST_TEMPLATE.md || Configure in web interface
/.github/PULL_REQUEST_TEMPLATE/Name.md || /.gitlab/merge_request_templates/Name.md
There are template templates in awesome-github-templates [1] and checklist template templates in github-issue-templates [2].
I really want slack to adopt these as well...
I'll often be on call with customer and create a checklist on MacOS Notes on the fly. Then will copy paste that in slack or github for simple tracking.
Mattermost supports threaded replies and Markdown with checklist checkboxes
You can post GitHub/GitLab project updates to a Slack/Mattermost channel with webhooks (and search for and display GH/GL issues with /slash commands); though issue edits and checkbox state changes aren't (yet?) included in the events that channels receive.
Apply to Y Combinator
The next Startup School course begins in July: https://www.startupschool.org/
> Learn how to start a startup with YC’s free 10-week online course.
> Here is a list of the top 100 Y Combinator companies by valuation (why valuation?), as of October 2018. We also included YC’s top 12 exits.
Here's the list of the 1,900 Y Combinator companies through Winter 2019 (W19) https://www.ycombinator.com/companies/
"Startup Playbook" by Sam Altman (YC Founder) and Illustrated by Gregory Koberger is also a good read: https://playbook.samaltman.com/
Trunk-Based Development vs. Git Flow
One major advantage of the gitflow/hubflow git workflows is that there is a standard way of merging across branches. For example, a 'hotfix' branch is merged into the stable master branch and also develop with one standard command; there's no need to re-explain and train new devs on how the branches were supposed to work here. I even copied the diagram(s) into my notes: https://westurner.github.io/tools/#hubflow
IMHO, `git log` on the stable master branch containing each and every tagged release is preferable to having multiple open release branches.
Requiring tests to pass before a PR gets merged is a good policy that's independent of the trunk or gitflow workflow decision.
Ask HN: Anyone else write the commit message before they start coding?
I feel like I just learned how to use Git: writing the message first thing has made me a lot more productive. I'm wondering if anyone else does this; I know test driven development is a thing, where people write tests before code, and this seems like a logical extension.
Ask HN: Datalog as the only language for web programming, logic and database
Can Datalog be used as the only language which we can use for writing server-side web application, complex domain business logic and database querying?
Are there any efforts made in this direction.
To quote myself from a post the other day https://news.ycombinator.com/item?id=19407170 :
> PyDatalog does Datalog (which is ~Prolog, but similar and very capable) logic programming with SQLAlchemy (and database indexes) and apparently NoSQL support. https://sites.google.com/site/pydatalog/
> Datalog: https://en.wikipedia.org/wiki/Datalog
> ... TBH, IDK about logic programming and bad facts. Resilience to incorrect and incredible information is - I suppose - a desirable feature of any learning system that reevaluates its learnings as additional and contradictory information makes its way into the datastores.
I'm not sure that Datalog is really necessary for most CRUD operations; SQLAlchemy and the SQLAlchemy ORM are generally sufficient for standard database querying CRUD.
The cortex is a neural network of neural networks
Is there a program like codeacademy but for learning sysadmin?
if not, anyone wanna build one?
A few sysadmin and devops curriculum resources; though none but Beaker and Molecule are interactive with any sort of testing AFAIU:
"System Administrator" https://en.wikipedia.org/wiki/System_administrator
"Software Configuration Management" (SCM) https://en.wikipedia.org/wiki/Software_configuration_managem...
"DevOps" https://en.wikipedia.org/wiki/DevOps
"OpsSchool Curriculum" http://www.opsschool.org
- Soft Skills 101, 201
- Labs Exercises
- Free. Contribute
awesome-sysadmin > configuration-management https://github.com/kahun/awesome-sysadmin/blob/master/README...
- This could list reusable module collections such as Puppet Forge and Ansible Galaxy;
- And module testing tools like Puppet Beaker and Ansible Molecule (that can use Vagrant or Docker to test a [set of] machines)
https://github.com/stack72/ops-books
- I'd add "Time Management for System Administrators" (2005)
https://landing.google.com/sre/books/
- There's now a "Site Reliability Workbook" to go along with the Google SRE book. Both are free online.
https://response.pagerduty.com
- The PagerDuty Incident Response Documentation is also free online.
- OpsGenie has a free plan also with incident response alerting and on-call management.
There are a number of awesome-devops lists.
Minikube and microk8s package Kubernetes into a nice bundle of distributed systems components that'll run on Lin, Mac, Win. You can convert docker-compose.yml configs to Kubernetes pods when you decide that it should've been HA with a load balancer SPOF and x.509 certs and a DRP (Disaster Recovery Plan) from the start!
Maybe You Don't Need Kubernetes
The argument that Kubernetes adds complexity is, in my opinion, bogus. Kubernetes is a "define once, forget about it" type of infrastructure. You define the state you want your infrastructure to and Kubernetes takes care of maintaining that state. Tools like Ansible and Puppet, as great as they are, do not guarantee your infrastructure will end up in the state you defined and you easily end up with broken services. The only complexity in kubernetes is the fact that it forces you to think and carefully design your infra in a way people aren't used to, yet. More upfront, careful thinking isn't complexity. It can only benefit you in the long run.
There is, however, a learning curve to Kubernetes, but it isn't this sharp. It does require you to sit down and read the doc for 8 hours, but that a small price to pay.
A few month back I wrote a blog post[1] that, through walking through the few different infrastructures my company experimented with over the years, surfaces many reasons one would want to use [a managed] Kubernetes. (For a shorter read, you can probably start at reading at [2])
[1]: https://boxunix.com/post/bare_metal_to_kube
[2]: https://boxunix.com/post/bare_metal_to_kube/#_hardware_infra...
It's pretty common for new technologies to advertise themselves as "adopt, and forget about it", but in my experience it's unheard of that any actually deliver on this promise.
Any technology you adopt today is a technology you're going to have to troubleshoot tomorrow. (I don't think the 15,000 Kubernetes questions on StackOverflow are all from initial setup.) I can't remember the last [application / service / file format / website / language / anything related to computer software] that was so simple and reliable that I wasn't searching the internet for answers (and banging my head against the wall because of) the very next month. It was probably something on my C=64.
As Kernighan said back in the 1970's, "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?" I've never used Kubernetes, but I've read some articles about it and watched some videos, and despite the nonstop bragging about its simplicity (red flag #1), I'm not sure I can figure out how to deploy with it. I'm fairly certain I wouldn't have any hope of fixing it when it breaks next month.
Hearing testimonials only from people who say "it doesn't break!" is red flag #2. No technology works perfectly for everyone, so I want to hear from the people who had to troubleshoot it, not the people who think it's all sunshine and rainbows. And those people are not kind, and make it sound like the cost is way more than just "8 hours reading the docs" -- in fact, the docs are often called out as part of the problem.
> As Kernighan said back in the 1970's, "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?"
What a great quote. Thanks
Some more quotes taken from http://typicalprogrammer.com/what-does-code-readability-mean that you might find interesting.
Any fool can write code that a computer can understand. Good programmers write code that humans can understand. – Martin Fowler
Just because people tell you it can’t be done, that doesn’t necessarily mean that it can’t be done. It just means that they can’t do it. – Anders Hejlsberg
The true test of intelligence is not how much we know how to do, but how to behave when we don’t know what to do. – John Holt
Controlling complexity is the essence of computer programming. – Brian W. Kernighan
The most important property of a program is whether it accomplishes the intention of its user. – C.A.R. Hoare
No one in the brief history of computing has ever written a piece of perfect software. It’s unlikely that you’ll be the first. – Andy Hunt
> Tools like Ansible and Puppet, as great as they are, do not guarantee your infrastructure will end up in the state you defined and you easily end up with broken services.
False dilemma. Ansible and Puppet are great tools for configuring kubernetes, kubernetes worker nodes, and building container images.
Kubernetes does not solve for host OS maintenance; though there are a number of host OS projects which remove most of what they consider to be unnecessary services, there's still need to upgrade kubernetes nodes and move pods out of the way first (which can be done with e.g. Puppet or Ansible).
As well, it may not be appropriate for monitoring to depend upon kubernetes; there again you have nodes to manage with an SCM tool.
Quantum Machine Appears to Defy Universe’s Push for Disorder
Maybe an example or related to time crystals?
"Scar (physics)": https://en.wikipedia.org/wiki/Scar_(physics)
> Scars are unexpected in the sense that stationary classical distributions at the same energy are completely uniform in space with no special concentrations along periodic orbits, and quantum chaos theory of energy spectra gave no hint of their existence
Pytype checks and infers types for your Python code
How does pytype compare with the PyAnnotate [1] and MonkeyType [2] dynamic / runtime PEP-484 type annotation type inference tools?
How I'm able to take notes in mathematics lectures using LaTeX and Vim
As a feat in itself this is definitely very impressive, but I wonder if it's really worth anyone's time to spend precious lecture time with your mind fully occupied in the mechanical task of taking notes rather than actually absorbing and engaging with the content. Especially for an extremely content-dense subject like Mathematics, where you need all your concentration just to process what you are reading and follow along the logic.
There is absolutely no dearth of study material on the internet. It's one thing to take notes as part of your study process. There it helps solidify your understanding. But surely taking notes on the fly when you have been barely introduced to the subject isn't going to help with that.
> mechanical task of taking notes rather than actually absorbing and engaging with the content
The mechanical task of taking notes is one of the most important parts of actually absorbing the material. It is not an either-or. Hearing/seeing the information, processing it in a way that makes sense to you individually, and then mechanically writing it down in a legible manner is one of the main methods that your brain learns. It's one of the primary reasons that taking notes is important in the first place. This is referred to as the "encoding hypothesis" [1].
There are actually even studies [2] that show that tools that assist in more efficient note taking, such as taking notes via typing rather than by hand, are actually detrimental to absorbing information, as it makes it easier for you to effectively pass the information directly from your ears to your computer without actually doing the processing that is required when writing notes by hand. This is why many universities prefer (or even require) notes to be taken by hand, and disallow laptops in class.
1: https://www.sciencedirect.com/science/article/pii/0361476X78...
2: https://www.npr.org/2016/04/17/474525392/attention-students-...
> The mechanical task of taking notes is one of the most important parts of actually absorbing the material. It is not an either-or. Hearing/seeing the information, processing it in a way that makes sense to you individually, and then mechanically writing it down in a legible manner is one of the main methods that your brain learns. It's one of the primary reasons that taking notes is important in the first place. This is referred to as the "encoding hypothesis" [1].
There's almost certainly an advantage to learning to think about math using a publishable symbol set like LaTeX.
We learn by reinforcement; with feedback loops that may take until weeks later in a typical university course.
> There are actually even studies [2] that show that tools that assist in more efficient note taking, such as taking notes via typing rather than by hand, are actually detrimental to absorbing information, as it makes it easier for you to effectively pass the information directly from your ears to your computer without actually doing the processing that is required when writing notes by hand.
Handwriting notes is impractical for some people due to e.g. injury and illegibility.
The linked study regarding retention and handwritten versus typed notes has been debunked with references that are referenced elsewhere in comments on this post. There have been a few studies with insufficient controls (lack of randomization, for one) which have been widely repeated by educators who want to be given attention.
Doodling has been shown to increase information retention. Maybe doodling as a control really would be appropriate.
Banning laptops from lectures is not respectful of students with injury and illegible handwriting. Asking people to put their phones on silent (so they can still make and take emergency calls) and refrain from distracting other students with irrelevant content on their computers is reasonable and considerate.
(What a cool approach to math note-taking. I feel a bit inferior because I haven't committed to learning that valuable, helpful skill and so that's stupid and you're just wasting your time because that's not even necessary when all you need to do is retain the information you've paid for for the next few months at most. If course, once you get on the job, you'll never always be using that tool and e.g. latex2sympy to actually apply that theory to solving a problem that people are willing to pay for. So, thanks for the tips and kudos, idiot)
LHCb discovers matter-antimatter asymmetry in charm quarks
So, does this disprove all of supersymmetry? https://en.wikipedia.org/wiki/Supersymmetry
No, supersymmetry and charge-parity symmetry are different.
Ah, thanks.
"CPT Symmetry" https://en.wikipedia.org/wiki/CPT_symmetry
"CP Violations" https://en.wikipedia.org/wiki/CP_violation
"Charm quark" https://en.wikipedia.org/wiki/Charm_quark :
> The antiparticle of the charm quark is the charm antiquark (sometimes called anticharm quark or simply anticharm), which differs from it only in that some of its properties have equal magnitude but opposite sign.
React Router v5
Using react router on one of my personal projects (~ 30k loc) was probably one of my largest regrets. At every turn it seemed designed to do the thing I wouldn’t expect, or have arbitrary restrictions that made my life tougher.
Some examples:
* there's no relative routes https://github.com/ReactTraining/react-router/issues/2172
* there's no way to refresh the page https://github.com/ReactTraining/react-router/issues/1982 ("that's your responsibility, not ours")
* The scroll position will stick when you navigate to a new route, causing you to need to create a custom component wrapper to manage scrolling https://github.com/ReactTraining/react-router/issues/3950
* React router's <link> do not allow you to link outside of the current site https://github.com/ReactTraining/react-router/issues/1147 so you have to make an in those cases. This doesn’t sound bad, but it’s particularly frustrating when dealing with dynamic or user generated content. Why can’t they handle this simple case?
> there's no relative routes
> React router's <link> do not allow you to link outside of the current site
Both valid points. Not show stoppers though. And not enough to make me regret using the library. The solution to the external links issue is a one-liner.
> there's no way to refresh the page https://github.com/ReactTraining/react-router/issues/1982 ("that's your responsibility, not ours")
That is your responsibility. Why do you need the routing library to handle page refreshing for you?
> the scroll position will stick when you navigate to a new route
Making the page scroll to the top when navigating to a new page is trivial. I would 100% rather have this problem instead of the opposite: scroll jumping to the top when I don't want it to. That's so much harder to fix.
> Navigating to the same page would not actually reload the page, it would just trigger a componentDidMount() on all components in the page again, which led me to have a lot of bugs when I did some initialization in my constructor
That's exactly what it's supposed to do. It's a client side routing solution. (I'm also pretty sure that it doesn't remount)
From the issues that you've had with the library, it seems like client side routing is not actually what you're looking for. If you regret using it so much, may I ask what the alternative would be?
IMO a project either needs client side routing, or it doesn't. If it does, then React Router is the obvious choice. Otherwise, of course, don't use it and save yourself from unnecessary complexity.
> The solution to the external links issue is a one-liner.
It is most definitely not a one-liner.
> Making the page scroll to the top when navigating to a new page is trivial.
So do it for me. There’s no reason a routing library should break default behavior.
> That's exactly what it's supposed to do. It's a client side routing solution. (I'm also pretty sure that it doesn't remount)
Then it’s “supposed” to have inconsistent behavior. Navigating anywhere else in my site will call constructors to all my components. Navigating to the same page won’t.
> From the issues that you've had with the library, it seems like client side routing is not actually what you're looking for.
My site was a music site. I wanted clientside routing so I could keep music playing while you navigated between links. Seemed like a slam dunk for clientside routing to me.
> may I ask what the alternative would be?
Quite honestly I would go onto GitHub and look for any routing solution that didn’t have multiple closed unfixed issues with hundreds of thumbs up.
> So do it for me.
I think this will work:
Put this wherever you put your reusable functions:
`const scrollToTop = () => document.getElementById('root').scrollIntoView();`
And put this in the components / functions you want the scrollToTop effect to work on:
`useEffect(() => { scrollToTop() }, []);`
And put this in your CSS:
`html { scroll-behavior: smooth }`
(Edit to add: Not saying React Router is something you should / shouldn’t use. Just wanted to share that code in case it helps unblock anyone.)
Accidentally downvoted on mobile (and upvoted two others). Thanks for this.
"Scroll Restoration" https://reacttraining.com/react-router/web/guides/scroll-res...
Experimental rejection of observer-independence in the quantum world
Objective truth!? A question for epistemologists to decide.
How could they record their high entropy (?) solipsistic observations in an immutable datastore in such as way as to have probably zero knowledge of the other party's observations?
Anyways, that's why I only read the title and the abstract.
Wigner's friend experiment: https://en.wikipedia.org/wiki/Wigner%27s_friend
Show HN: A simple Prolog Interpreter written in a few lines of Python 3
Cool tests! PyDatalog does Datalog (which is ~Prolog, but similar and very capable) logic programming with SQLAlchemy (and database indexes) and apparently NoSQL support. https://sites.google.com/site/pydatalog/
Datalog: https://en.wikipedia.org/wiki/Datalog
... TBH, IDK about logic programming and bad facts. Resilience to incorrect and incredible information is - I suppose - a desirable feature of any learning system that reevaluates its learnings as additional and contradictory information makes its way into the datastores.
Thanks for the feedback :) I'll definitely check out Datalog - I didn't realize they had logic programming integrated with SQLAlchemy, so it definitely sounds interesting!
How to earn your macroeconomics and finance white belt as a software developer
Thanks for the wealth of resources in this post. Here are a few more:
"Python for Finance: Analyze Big Financial Data" (2014, 2018) https://g.co/kgs/qkY8J6 ... https://pyalgo.tpq.io also includes the "Finance with Python" course and this book as a PDF and Jupyter notebooks.
Quantopian put out a call for the best Value Investing algos (implemented in quantopian/zipline) awhile back. This post links to those and other value investing resources: https://westurner.github.io/hnlog/#comment-19181453 (Ctrl-F "econo")
"Lectures in Quantitative Economics as Python and Julia Notebooks" https://news.ycombinator.com/item?id=19083479 links to these excellent lectures and a number of tools for working with actual data from FRED, ECB, Eurostat, ILO, IMF, OECD, UNSD, UNESCO, World Bank, Quandl.
One thing that many finance majors, courses, and resources often fail to identify is the role that startup and small businesses play in economic growth and actual value creation: jobs, GDP, return on direct capital investment. Most do not succeed, but it is possible to do better than index funds and have far more impact in terms of sustainable investment than as an owner of a nearly-sure-bet index fund that owns some shares and takes a hands-off approach to business management, research, product development, and operations.
Is it possible to possess a comprehensive understanding of finance and economics but still not have personal finance down? Personal finance: r/personalfinance/wiki, "Consumer science (a.k.a. home economics) as a college major" https://news.ycombinator.com/item?id=17894632
Ask HN: Relationship between set theory and category theory
I have an idea about the relationship between set theory and category theory and I would like some feedback. I would like others to see it too, and I don't know how to do it. I think it's at least interesting to look at as a slightly crazy collage, but I was a bit more excited than normal when the idea hit, so I just had to dump it all at once in this image: https://twitter.com/FamilialRhino/status/1101777965724168193 (You will have to zoom the picture in order to be able to read the scribbles.)
It has to do with resonance in the energy flowing in emergent networks. Can't quite put my finger on it, so I'll be here to answer any questions.
Thanks for reading.
"Categorical set theory" > "References" https://en.wikipedia.org/wiki/Categorical_set_theory#Referen...
From "Homotopy category" > "Concrete categories" https://en.wikipedia.org/wiki/Homotopy_category#Concrete_cat... :
> While the objects of a homotopy category are sets (with additional structure), the morphisms are not actual functions between them, but rather a classes of functions (in the naive homotopy category) or "zigzags" of functions (in the homotopy category). Indeed, Freyd showed that neither the naive homotopy category of pointed spaces nor the homotopy category of pointed spaces is a concrete category. That is, there is no faithful functor from these categories to the category of sets.
My "understanding" of category theory is extremely shallow, but that's exactly why I think my proposal makes sense. It is the kind of thing that everybody ignores for decades precisely because it's transparently obvious, like a fish that doesn't understand water.
Here is the statement:
The meaning of no category is every category.
reference: https://terrytao.wordpress.com/2008/02/05/the-blue-eyed-isla...
This was already understood by everybody in the field, no doubt. It's just that somebody has to actually say it to someone else in order for the symmetry to break. The link above has the exact description of this, from Terence Tao.
The most popular docker images each contain at least 30 vulnerabilities
Although vulnerability scanners can be a useful tool, I find it very troublesome that you can utter the sentence "this package contains XX vulnerabilities, and that package contains YY vulnerabilities" and then stop talking. You've provided barely any useful information!
The quantity of vulnerabilities in an image is not really all that useful information. A large amount of vulnerabilities in a Docker image does not necessarily imply that there's anything insecure going on. Many people don't realize that a vulnerability is usually defined as "has a CVE security advisory", and that CVEs get assigned based on a worst-case evaluation of the bug. As a result, having a CVE in your container barely tells you anything about your actual vulnerability position. In fact, most of the time you will find that having a CVE in some random utility doesn't matter. Most CVEs in system packages don't apply to most of your containers' threat models.
Why not? Because an attacker is very unlikely to be able to use vulnerabilities in these system libraries or utilities. Those utilities are usually not in active use in the first place. Even if they are used, you are not usually in a position to exploit these vulnerabilities as an attacker.
Just as an example, a hypothetical outdated version of grep in one of these containers can hypothetically contain many CVEs. But if your Docker service doesn't use grep, then you would need to manually run grep to be vulnerable. And an attacker that is able to run grep in your Docker container has already owned you - it doesn't make a difference that your grep is vulnerable! This hypothetical vulnerable version of grep therefore makes no difference in the security of your container, despite containing many CVEs.
It's the quality of these vulnerabilities that matters. Can an attacker actually exploit the vulnerabilities to do bad things? The answer for almost all of these CVEs is "no". But that's not really the product that Snyk sells - Snyk sells a product to show you as many vulnerabilities as possible. Any vulnerability scanner company thinks it can provide most business value (and make the most money) by reporting as many vulnerabilities as it can. For sure it can help you to pinpoint those few vulnerabilities that are exploitable, but that's where your own analysis comes in.
I'm not saying there's not a lot to improve in terms of container security. There's a whole bunch to improve there. But focusing on quantities like "amount of CVEs in an image" is not the solution - it's marketing.
This entire depending on a base container for setup and then essentially throwing away everything you get from the package manager is part of the issue. There is no real package management for Docker. Hell, there isn't even an official way to determine if your image needs an upgrade (I wrote a ruby script that gets the latest tags from a docker repo, extracts the ones with numbers and sorts them in order and compares them to what I have running).
Relying on an Alpine/Debian/Ubuntu base helps to get dependencies installed quickly. Docker could have just created their own base distro and some mechanism to track package updates across images, but they did not.
There are guides for making bare containers, they contain nothing .. no ip, grep, bash .. only the bare minimum libraries and requirements to run your service. They are minimal, but incredibly difficult to debug (sysdig still sucks unless you shell out money for enterprise).
I feel like containers are alright, but Docker is a partial dumpster fire. cgroup isolation is good, the crazy way we deal with packages in container systems is not so good.
Sure if you're just checking for base-distro packages for security vulnerabilities, you're going to find security issues that don't apply (e.g. an exploit in libpng even though your container runs nothing that even links to libpng), but it does excuse the whole issue with the way containers are constructed.
I think this space is really open too, for people to find better systems that are also portable: image formats that are easy to build, easy to maintain dependencies for, and easy to run as FreeBSD jails OR Linux cgroup managed containers (Docker for FreeBSD translated images to jails, but it's been unmaintained for years).
I agree the tooling is a tire fire :(
> e.g. an exploit in libpng even though your container runs nothing that even links to libpng
it's a problem because some services or api's could be abused giving the attacker a path to these vulnerable resources and then use that vulnerability regardless if it is used currently by your service.
I like my images to only contain what's absolutely needed to do that job only. It's not so difficult to do, provided people would be willing to architect systems from the ground up, instead of pulling in a complete debian or fedora installation and then removing things (that should be outlawed imho lol). Not only do I get less attack surface but also smaller updates (which again then is incentive to do more often), less complexity, less logs, easier auditabile (now every log file or even log line might give valid clues), faster incident response, easier troubleshooting, sorry for going on and on ...
It's a cultural problem too: where people working in an environment where it's normal to have every command available on a production system (really?), and where there is no barrier to install anything new that is "required" without discussion & peer review (what are we pair programming for?) or where nobody even tracks the dead weight in production or whether configs are locked down?.
I sometimes think many companies lost control over this long ago. [citation needed] :(
I'm not familiar with Docker infrastructure but what is the alternative to "pulling in a complete debian or fedora installation and then removing things"? Compiling your own kernel and doing the whole "Linux From Scratch" thing? Isn't that incredibly time-intensive to do for every single container?
Just have an image with very a minimal user land. Compiling your own kernel is irrelevant because you need the host kernel to run the container, and container images don't contain a kernel.
The busybox image is a good starting point. Take that, then copy your executables and libraries. If you are willing to go further, you can rather easily compile your own busybox with most utilities stripped out. It's not time intensive because you need to do it just once, and it takes just an afternoon to figure out how.
I don't think this is a tooling problem at all.
"The tooling makes it too easy to do it wrong." Compared to shell scripts with package manager invocations? Nobody configures a system with just packages: there are always scripts to call, chroots to create, users and groups to create, passwords to set, firewall policies to update, etc.
There are a bunch of ways to create LXC containers: shell scripts, Docker, ansible. Shell scripts preceded Docker: you can write a function to stop, create an intermediate tarball, and then proceed (so that you don't have to run e.g. debootstrap without a mirror every time you manually test your system build script; so that you can cache build steps that completed successfully).
With Docker images, the correct thing to do is to extend FROM the image you want to use, build the whole thing yourself, and then tag and store your image in a container repository. Neither should you rely upon months-old liveCD images.
"You should just build containers on busybox." So, no package management? A whole ensemble of custom builds to manually maintain (with no AppArmor or SELinux labels)? Maintainers may prefer for distros to field bug reports for their own common build configurations and known-good package sets. Please don't run as root in a container ("because it's only a container that'll get restarted someday"). Busybox is not a sufficient OS distribution.
It's not the tools, it's how people are choosing to use them. They can, could, and should try and use idempotent package management tasks within their container build scripts; but they don't and that's not Bash/Ash/POSIX's fault either.
> With Docker images, the correct thing to do is to extend FROM the image you want to use, build the whole thing yourself, and then tag and store your image in a container repository. Neither should you rely upon months-old liveCD images.
This should rebuild all. There should be an e.g. `apt-get upgrade -y && rm -rf /var/lib/apt/lists` in there somewhere (because base images are usually not totally current (and neither are install ISOs)).
`docker build --no-cache --pull`
You should check that each Dockerfile extends FROM `tag:latest` or the latest version of the tag that you support. Its' not magical, you do have to work it.
Also, IMHO, Docker SHOULD NOT create another Linux distribution.
Tinycoin: A small, horrible cryptocurrency in Python for educational purposes
The 'dumbcoin' jupyter notebook is also a good reference: "Dumbcoin - An educational python implementation of a bitcoin-like blockchain" https://nbviewer.jupyter.org/github/julienr/ipynb_playground...
When does the concept of equilibrium work in economics?
"Modeling stock return distributions with a quantum harmonic oscillator" (2018) https://iopscience.iop.org/article/10.1209/0295-5075/120/380...
> We propose a quantum harmonic oscillator as a model for the market force which draws a stock return from short-run fluctuations to the long-run equilibrium. The stochastic equation governing our model is transformed into a Schrödinger equation, the solution of which features "quantized" eigenfunctions. Consequently, stock returns follow a mixed χ distribution, which describes Gaussian and non-Gaussian features. Analyzing the Financial Times Stock Exchange (FTSE) All Share Index, we demonstrate that our model outperforms traditional stochastic process models, e.g., the geometric Brownian motion and the Heston model, with smaller fitting errors and better goodness-of-fit statistics. In addition, making use of analogy, we provide an economic rationale of the physics concepts such as the eigenstate, eigenenergy, and angular frequency, which sheds light on the relationship between finance and econophysics literature.
"Quantum harmonic oscillator" https://en.wikipedia.org/wiki/Quantum_harmonic_oscillator
The QuantEcon lectures have a few different multiple agent models:
"Rational Expectations Equilibrium" https://lectures.quantecon.org/py/rational_expectations.html
"Markov Perfect Equilibrium" https://lectures.quantecon.org/py/markov_perf.html
"Robust Markov Perfect Equilibrium" https://lectures.quantecon.org/py/rob_markov_perf.html
"Competitive Equilibria of Chang Model" https://lectures.quantecon.org/py/chang_ramsey.html
... "Lectures in Quantitative Economics as Python and Julia Notebooks" https://news.ycombinator.com/item?id=19083479 (data sources (pandas-datareader, pandaSDMX), tools, latex2sympy)
"Econophysics" https://en.wikipedia.org/wiki/Econophysics
> Indeed, as shown by Bruna Ingrao and Giorgio Israel, general equilibrium theory in economics is based on the physical concept of mechanical equilibrium.
Simdjson – Parsing Gigabytes of JSON per Second
> Requirements: […] A processor with AVX2 (i.e., Intel processors starting with the Haswell microarchitecture released 2013, and processors from AMD starting with the Rizen)
Also noteworthy that on Intel at least, using AVX/AVX2 reduces the frequency of the CPU for a while. It can even go below base clock.
iirc, it's complicated. Some instructions don't reduce the frequency; some reduce it a little; some reduce it a lot.
I'm not sure AVX2 is as ubiquitous as the README says: "We assume AVX2 support which is available in all recent mainstream x86 processors produced by AMD and Intel."
I guess "mainstream" is somewhat subjective, but some recent Chromebooks have Celeron processors with no AVX2:
https://us-store.acer.com/chromebook-14-cb3-431-c5fm
https://ark.intel.com/products/91831/Intel-Celeron-Processor...
Because someone wanting 2.2GB/s JSON parsing is deploying to a chromebook...
It doesn't seem that laughable to me to want faster JSON parsing on a Chromebook, given how heavily JSON is used to communicate between webservers and client-side Javascript.
"Faster" meaning faster than Chromebooks do now; 2.2 GB/s may simply be unachievable hardware-wise with these cheap processors. They're kinda slow, so any speed increase would be welcome.
AVX2 also incurs some pretty large penalties for switching between SSE and AVX2. Depending on the amount of time taken in the library between calls, it could be problematic.
This looks mostly applicable to server scenarios where the runtime environment is highly controlled.
There is no real penalty for switching between SSE and AVX2, unless you do it wrong. What are you referring to specifically?
Are you talking about state transition penalties that can occur if you forget a vzeroupper? That's the only thing I'm aware of which kind of matches that.
A faster, more efficient cryptocurrency
Full disclosure: I work on the cryptocurrency in this article, Algorand.
There are a lot of questions and speculation here about this paper and Algorand. I would be happy to try an answer them to your satisfaction. Some context may be helpful first, though. This paper is an innovation about one aspect of our technology. Algorand has a very fast consensus mechanism and can add blocks as quickly as the network can deliver them. We become a victim of our success. The blockchain will grow very rapidly. A terabyte a month is possible. The storage issue associated with our performance can quickly become an issue. The Vault paper is focused on solving this and other storage scaling problems.
The Algorand pure proof-of-stake blockchain and associated cryptocurrency has many novel innovations aside from Vault. It possesses security and scalability properties beyond what any other blockchain technology allows while still being completely decentralized. Our website, algorand.com, and whitepaper are great places to start to learn more.
If you learn best from videos then I suggest you watch Turing award winner and cryptographic pioneer, Silvio Micali, talk about Algorand: https://youtu.be/NykZ-ZSKkxM. He is a captivating speaker and the founder of Algorand.
Are there reasons that e.g. Bitcoin and Ethereum and Stellar could not implement some of these more performant approaches that Algorand [1] and Vault [2] have developed, published, and implemented? Which would require a hard fork?
[2] https://dspace.mit.edu/handle/1721.1/117821
My understanding is that PoS approaches follow normal byzantine agreement theory which states that adversaries cannot control more than 1/3rd of the accounts (or money in the case of algorand). You can also delay new blocks more easily.
Ethereum is scared or that so they are implementing some hybrid form.
Bitcoin is doomed from my perspective, because of the focus on proof of work and the confirmation times. When you realize that algorand is super fast, there is no "confirmation time", and there is no waste in energy to mine, then it is hard to back up any cryptocurrency focusing on proof of work.
And what of decentralized premined chains (with no PoW, no PoS, and far less energy use) that release coins with escrow smart contracts over time such as Ripple and Stellar (and close a new ledger every few seconds)?
> Algorand has a very fast consensus mechanism and can add blocks as quickly as the network can deliver them. We become a victim of our success. The blockchain will grow very rapidly. A terabyte a month is possible. The storage issue associated with our performance can quickly become an issue. The Vault paper is focused on solving this and other storage scaling problems.
What prevents a person from using a chain like IPFS?
Ethereum Casper PoS has been under review for quite some time.
Why isn't all Bitcoin on Lightning Network?
Bitcoin could make bootstrapping faster by choosing a considered-good blockhash and balances, but AFAIU, re-verifying transactions like Bitcoin and derivatives do prevents hash collision attacks that are currently considered infeasible for SHA-256 (especially given a low block size).
There was an analysis somewhere where they calculated the cloud server instance costs of mounting a ~51% attack (which applies to PoW chains) for various blockchains.
Bitcoin is not profitable to mine in places without heavily subsidized dirty/clean energy anymore: energy and Bitcoin commodity costs and prices have intersected. They'll need any of: inexpensive clean energy, more efficient chips, higher speculative value.
Energy arbitrage (grid-scale energy storage) may be more profitable now. We need energy storage in order to reach 100% renewable energy (regardless of floundering policy support).
Ripple is not decentralized. I don't know enough about Stellar to answer.
Bitcoin is software and can easily implement these features but the community is divided and can't reach consensus on anything. Lightning Network as layer two solution is pretty good from what I know.
Ethereum improvements are coming along very slowly and that's good. They're the only blockchain with active engagement by thousands of multiple parties.
Aragaon and Vault's papers might sound good, but who knows how they'll turn out in production.
People argue this all day. There's a lot of FUD.
Ripple only runs ~7% of validator nodes; which is far less centralized control than major Bitcoin mining pools and businesses (who do the deciding in regards to the many Bitcoin hard forks); that's one form of decentralization.
Ripple clients can use their own UNL or use the Ripple-approved UNL.
Ripple is traded on a number of exchanges (though fewer than Bitcoin for certain); that's another form of decentralization.
As an open standard, ILP will further reduce vendor lock in (and increase interoperability between) networks that choose to implement it.
There are forks of Ripple (e.g. Stellar) just like there are forks of Bitcoin and Ethereum.
From https://ripple.com/insights/the-inherently-decentralized-nat... :
> In contrast, the XRP Ledger requires 80 percent of validators on the entire network, over a two-week period, to continuously support a change before it is applied. Of the approximately 150 validators today, Ripple runs only 10. Unlike Bitcoin and Ethereum — where one miner could have 51 percent of the hashing power — each Ripple validator only has one vote in support of an exchange or ordering a transaction.
How does your definition of 'decentralized' differ?
[deleted]
Git-signatures – Multiple PGP signatures for your commits
Is there anything out there that doesn't need GPG? Having a working GPG install is a huge lift for developers.
I take this to mean: apart from the barnacles on GPG, could there be a system which does what GPG does for software development (signing), without the non-functioning web-of-trust of GPG, or the hierarchical system of x509 signing? Something that deals with lost keys, compromised keys/accounts, loss of DNS control, MitMing, MitBing, etc?
I think it is probably in the class of problems where there are no great foolproof solutions. However, I can imagine that techniques like certificate transparency (all signed x509 certificates pushed to a shared log) would be quite useful. Even blockchain techniques. Maybe send someone to check on me, I'm feeling unwell having written that.
> I think it is probably in the class of problems where there are no great foolproof solutions. However, I can imagine that techniques like certificate transparency (all signed x509 certificates pushed to a shared log) would be quite useful.
Securing DNS: "https://news.ycombinator.com/item?id=19181362"
> Certs on the Blockchain: "Can we merge Certificate Transparency with blockchain?" https://news.ycombinator.com/item?id=18961724
> Namecoin (decentralized blockchain DNS): https://en.wikipedia.org/wiki/Namecoin
(Your first link is broken.)
My main problem with blockchain is the excessive energy consumption of PoW. I know there are PoS efforts, but they seem problematical.
I like the recent CertLedger paper: https://eprint.iacr.org/2018/1071.pdf
My mistake. How ironic. Everything depends upon the red wheelbarrow. Here's that link without the trailing ": https://news.ycombinator.com/item?id=19181362
> My main problem with blockchain is the excessive energy consumption of PoW. I know there are PoS efforts, but they seem problematical.
One report said that 78% of Bitcoin energy usage is from renewable sources (many of which would otherwise be curtailed and otherwise unfunded due to flat-to-falling demand for electricity). But PoW really is expensive and hopefully the market will choose less energy-inefficient solutions from the existing and future blockchain solutions while keeping equal or better security assurances.
>> Proof of Work (Bitcoin, ...), Proof of Stake (Ethereum Casper), Proof of Space, Proof of Research (GridCoin, CureCoin,)
The spec should be: DDOS resiliant (without a SPOF), no one entity with control over API and/or database credentials and database backups and the clock, and immutable.
Immutability really cannot be ensured with hashed records that incorporate the previous record's hash as a salt in a blocking centralized database because someone ultimately has root and the clock and all the backups and code vulnerable to e.g. [No]SQL injection; though distributed 'replication' and detection of record modification could be implemented. git push -f may be detected if it's on an already-replicated branch; but git depends upon local timestamps. google/trillian does Merkle trees in a centralized database (for Certificate Transparency).
In quickly reading the git-signatures shell script sources, I wasn't certain whether the git-notes branch with the .gitsigners that are fetched from all n keyservers (with DNS) is also signed?
I also like the "Table 1: Security comparison of Log Based Approaches to Certificate Management" in the CertLedger paper. Others are far more qualified to compare implementations.
You read my mind. I'd love if it could be rooted in a Yubikey.
Decoupling the "signing" and "verifying" parts seem like a good idea. As random Person signs something, how someone else figures out how to go trust that signature is a separate problem.
> I'd love if it could be rooted in a Yubikey.
FIDO2 and Yubico helped develop the new W3C WebAuthn standard: https://en.wikipedia.org/wiki/WebAuthn
But WebAuthn does not solve for WoT or PKI or certificate pinning.
> Decoupling the "signing" and "verifying" parts seem like a good idea. As random Person signs something, how someone else figures out how to go trust that signature is a separate problem.
Someone can probably help with terminology here. There's identification (proving that a person has the key AND that it's their key (biometrics, challenge-response)), signing (using a key to create a cryptographic signature – for the actual data or a reasonably secure cryptographic hash of said data – that could only could have been created with the given key), signature verification (checking that the signature was created by the claimed key for the given data), and then there's trusting that the given key is authorized for a specific purpose (Web of Trust (key-signing parties), PKI, ACME, exchange of symmetric keys over a different channel such as QKD) by e.g. signing a structured document that links cryptographic keys with keys for specific authorized functions and trusting the key(s) used to sign said authorizing document.
Private (e.g. Zero Knowledge) blockchains can be used for key exchange and key rotation. Public blockchains can be used for sharing (high-entropy) key components; also with an optional exchange of money to increase the cost of key compromise attempts.
There's also WKD: "Web Key Directory"; which hosts GPG keys over HTTPS from a .well-known URL for a given user@domain identifier: https://wiki.gnupg.org/WKD
Compared to existing PGP/GPG keyservers, WKD does rely upon HTTPS.
TUF is based on Thandy. TUF: "The Update Framework" does not presume channel security (is designed to withstand channel compromise) https://en.wikipedia.org/wiki/The_Update_Framework_(TUF)
The TUF spec doesn't mention PGP/GPG: https://github.com/theupdateframework/specification/blob/mas...
There's a derivative of TUF for automotive applications called Uptane: https://uptane.github.io
The Bitcoin article on multisignature; 1-of-2, 2-of-2, 2-of-3, 3-of-5, etc.: https://en.bitcoin.it/wiki/Multisignature
Running an LED in reverse could cool future computers
"Near-field photonic cooling through control of the chemical potential of photons" (2019) https://www.nature.com/articles/s41586-019-0918-8
Compounding Knowledge
Buffet’s approach to life is interesting for the same reason an Olympic gymnast is interesting. He has specialized to an extreme and is taking advantage of the rewards of that specialization and natural talent in a unique way.
It’s easy for me to feel shame that I don’t read 8 hours per day, as Warren and Charlie do. Buffett is a phenomenal investor but by all accounts, rather odd. He eats like crap, doesn’t exercise, had a profoundly weird relationship with his wife, and seems addicted to his work at the expense of everything else.
My point is this: his practice of reading income statements and business reports 8 hours per day for 60 years doesn’t make him the kind of person I wish to emulate.
Don’t get me wrong, I respect his levelheadedness toward money, lack of polish wrt PR, and generous philanthropic efforts. There’s a lot of good. But it’s easy to idolize the guy.
Controversial take: Buffett is actually not a great investor in the way people think. Via the float in his insurance companies, he receives a 0% infinite maturity loan to plow into the market. That financial leverage gives him the ability to beat the market year after year - not his own stockpicking prowess.
If you were to start with $1B, then get an extra $2B that you never had to pay back, you too would do quite well holding Coca-Cola and other safe companies with moats. Your returns would be 3x everyone else's.
Buffett himself has tried to explain the impact of float on Berkshire Hathaway but it never seems to sink in with people.
Ill take it even farther: Buffet isn't statistically different than average. He just found a strategy that happened to work, was stubborn enough to stick to it through bad times, and used copious amounts of leverage to juice returns.
A really interesting paper called "Buffet's Alpha" talks about this, and was able to replicate his performance by following a few simple rules. They found that he produced very little actual alpha. To his credit, he seems to have been observant enough to stumble into factors(value, quality, and low beta) before anyone else knew they existed, which is his real strength and contribution.
A summary of that paper from the authors is here: https://www.aqr.com/Insights/Research/Journal-Article/Buffet....
They work at AQR, a firm which is notable for being one of the only successful hedge funds to actually publish meaningful research.
The paper's conclusion is noteworthy:
> The efficient market counterargument is that Buffett was simply lucky. Our findings suggest that Buffett’s success is neither luck nor magic but is a reward for a successful implementation of value and quality exposures that have historically produced high returns. Second, we illustrated how Buffett’s record can be viewed as an expression of the practical implementability of academic factor returns after transaction costs and financing costs. We simulated how investors can try to take advan- tage of similar investment principles. Buffett’s success shows that the high returns of these academic factors are not simply “paper” returns; these returns can be realized in the real world after transaction costs and funding costs, at least by Warren Buffett. Furthermore, Buffett’s exposure to the BAB factor and his unique access to leverage are consistent with the idea that the BAB factor represents reward to the use of leverage.
BTW, AQR funded the initial development of pandas; which now powers tools like alphalens (predictive factor analysis) and pyfolio.
There's your 'compounding knowledge'.
(Days later)
"7 Best Community-Built Value Investing Algorithms Using Fundamentals" https://blog.quantopian.com/fundamentals-contest-winners/
(The Zipline backtesting library also builds upon Pandas)
How can we factor ESG/sustainability reporting into these fundamentals-driven algorithms in order to save the world?
I wonder to what extent Warren Buffet is Warren Buffet because of how he thinks and acts (like all of these non-fiction authors selling books by using his name would have us believe), and to what extent he is the product of media selection bias. -- If you take a large enough group of people who take risky stakes that are large enough (like the world of financial asset management), then one of them is bound to be as successful as Warren Buffet, even if they all behave randomly.
Funnily enough, Buffett actually calculated this in his essay "The Super Investors of Graham and Doddsville"
Basically, advocates of the Free Market Hypotheses believed that it was impossible for anyone to deliberately, repeatedly generate alpha. The market was rational, and success was probabilistically distributed. With a large enough population, you will get Buffett level returns - therefor Buffett is a fluke.
However, Buffett calculated that there weren't enough investors for his success to be a factor of random distribution, plus there were 2 dozen others who followed a similar strategy who also consistently generate alpha
"The Superinvestors of Graham and Doddsville" (1984) https://scholar.google.com/scholar?cluster=17265410477248371...
From https://en.wikipedia.org/wiki/The_Superinvestors_of_Graham-a... :
> The speech and article challenged the idea that equity markets are efficient through a study of nine successful investment funds generating long-term returns above the market index.
This book probably doesn't mention that he's given away over 71% to charity since Y2K. Or that it's really cold and windy and snowy in Omaha; which makes for lots of reading time.
"Warren Buffett and the Interpretation of Financial Statements: The Search for the Company with a Durable Competitive Advantage" (2008) [1], "Buffetology" (1999) [2], and "The Intelligent Investor" (1949, 2009) [3] are more investment-strategy-focused texts.
[1] https://smile.amazon.com/Warren-Buffett-Interpretation-Finan...
[2] https://smile.amazon.com/Buffettology-Previously-Unexplained...
[3] https://smile.amazon.com/Intelligent-Investor-Definitive-Inv...
Value Investing: https://en.wikipedia.org/wiki/Value_investing https://www.investopedia.com/terms/v/valueinvesting.asp
Why CISA Issued Our First Emergency Directive
There are a number of efforts to secure DNS (and SSL/TLS which generally depends upon DNS; and upon which DNS-over-HTTPS depends) and the identity proof systems which are used for record-change authentication and authorization.
Domain registrars can and SHOULD implement multi-factor authentication. https://en.wikipedia.org/wiki/Multi-factor_authentication
Are there domain registrars that support FIDO/U2F or the new W3C WebAuthn spec? https://en.wikipedia.org/wiki/WebAuthn
Credentials and blockchains (and biometrics): https://gist.github.com/westurner/4345987bb29fca700f52163c33...
DNSSEC: https://en.wikipedia.org/wiki/Domain_Name_System_Security_Ex...
ACME / LetsEncrypt certs expire after 3 months (*) and require various proofs of domain ownership: https://en.wikipedia.org/wiki/Automated_Certificate_Manageme...
Certificate Transparency: https://en.wikipedia.org/wiki/Certificate_Transparency
Certs on the Blockchain: "Can we merge Certificate Transparency with blockchain?" https://news.ycombinator.com/item?id=18961724
Namecoin (decentralized blockchain DNS): https://en.wikipedia.org/wiki/Namecoin
DNSCrypt: https://en.wikipedia.org/wiki/DNSCrypt
DNS over HTTPS: https://en.wikipedia.org/wiki/DNS_over_HTTPS
DNS over TLS: https://en.wikipedia.org/wiki/DNS_over_TLS
Chrome will Soon Let You Share Links to a Specific Word or Sentence on a Page
I would like to urge the browser developers/makers to adopt existing proposals which came through open consensus which do precisely cover the same use cases (and more!)
W3C Reference Note on Selectors and States: https://www.w3.org/TR/selectors-states/
It is part of the suite of specs that came through the W3C Web Annotation Working Group: https://www.w3.org/annotation/
More examples in W3C Note Embedding Web Annotations in HTML: https://www.w3.org/TR/annotation-html/
Different kinds of Web resources can combine multiple selectors and states. Here is a simple one using `TextQuoteSelector` handled by the https://dokie.li/ clientside application:
http://csarven.ca/dokieli-rww#selector(type=TextQuoteSelecto...
A screenshot/how-to: https://twitter.com/csarven/status/981924087843950595
"Integration with W3C Web Annotations" https://github.com/bokand/ScrollToTextFragment/issues/4
> It would be great to be able to comment on the linked resource text fragment. W3C Web Annotations [implementations] don't recognize the targetText parameter, so AFAIU comments are then added to the document#fragment and not the specified text fragment. [...]
> Is there a simplified mapping of W3C Web Annotations to URI fragment parameters?
Guidelines for keeping a laboratory notebook
My paper notebooks were always horrific. Writing has always been painful and awkward for me. But I could type like nobody's business thanks to programming. I survived college in the early 80s by being one of the first students to get a word processor.
By the time I was keeping a notebook, my work was generating mountains of computer readable data, source code, and so forth. We managed by agreeing on a format for data files, where the filename referenced a notebook page, and it worked OK.
Today, it's unavoidable that people are going to keep their notes electronically, and there are no perfect solutions for doing this. Wet chemists still like paper notebooks, since it's hard to get a computer close to the bench, and to type while wearing rubber gloves. Academic workers are expected to supply their own computers, and are nervous about getting them damaged or contaminated. Plus, drawing pictures and writing equations on a computer are both awkward.
Computation related fields lend themselves well to purely electronic notebooks, no surprise. Today, a lot of my work fits perfectly in a Jupyter notebook.
Commercial notebook software exists, but it tends to be sold largely for enterprise use, i.e., the solution it solves is how to control lab workers and secure their results, not how to enable independent, creative work.
> Computation related fields lend themselves well to purely electronic notebooks, no surprise. Today, a lot of my work fits perfectly in a Jupyter notebook.
Some notes and ideas regarding Jupyter notebooks as lab notebooks from "Keeping a Lab Notebook [pdf]": https://news.ycombinator.com/item?id=15710815
Superalgos and the Trading Singularity
Though others didn't, you might find this interesting: "Ask HN: Why would anyone share trading algorithms and compare by performance?" https://news.ycombinator.com/item?id=15802785 ( https://westurner.github.io/hnlog/#story-15802785 )
I think there is value in a back-testing module, however, sharing an algo doesn't make sense to me, until unless someone wants to buy mine for an absurd amount.
I think part of the value of sharing knowledge and algorithmic implementations comes from getting feedback from other experts; like peer review and open science and teaching.
Case in point: the first algorithm on this list [1] of community contributed algorithms that were migrated to their new platform is "minimum variance w/ constraint" [2]. Said algorithm showed returns of over 200% as compared with 77% returns from the SPY S&P 500 ETF over the same period, ceteris paribus. In the 69 replies, there are modifications by community members and the original author that exceed 300%.
Working together on open algorithms has positive returns that may exceed advantages of closed algorithmic development without peer review.
[1] https://www.quantopian.com/posts/community-algorithms-migrat...
[2] https://www.quantopian.com/posts/56b6021b3f3b36b519000924
How well does it do in production though and what happens when multiple algos execute the same trades? Does it cause the rest of the algos to adapt and change results? It makes sense to back-test together and work on it, but if it's proven to work, someone will create something to monitor volume on those trades and work against it. I'd be curious to see the same algo do 300% in production, and if so, then my bias would be uncalled for.
> How well does it do in production though and what happens when multiple algos execute the same trades?
Price inflation.
> Does it cause the rest of the algos to adapt and change results?
Trading index ETFs? IDK
> It makes sense to back-test together and work on it, but if it's proven to work, someone will create something to monitor volume on those trades and work against it.
Why does it need to do lots of trades? Is it possible for anyone other than e.g. SEC to review trades by buyer or seller?
> I'd be curious to see the same algo do 300% in production, and if so, then my bias would be uncalled for.
pyfolio does tear sheets with Zipline algos: pyfolio/examples/zipline_algo_example.ipynb https://nbviewer.jupyter.org/github/quantopian/pyfolio/blob/...
alphalens does performance analysis of predictive factors: alphalens/examples/pyfolio_integration.ipynb https://nbviewer.jupyter.org/github/quantopian/alphalens/blo...
awesome-quant lists a bunch of other tools for algos and superalgos: https://github.com/wilsonfreitas/awesome-quant
What's a good platform for paper trading (with e.g. zipline or moonshot algorithms)?
I disagree with price inflation just because everything is hedged, but it may be true.
The too many trades is if there are 300 algos, and I look in the order book and see different orders from different exchanges at the same price point, then I would be adapting to see what's happening, not myself, but there are people who watch order flows.
I don't paper trade, either it works in production with real money or not. Have to get a feel for spreads, commissions, and so on.
Also, in my case, I am hesitant to even use paid services as someone can be watching it, so most my tools are made by me. Good luck with your trading though, if it works out, let me know, I'd pay to use it along side my other trades.
Crunching 200 years of stock, bond, currency and commodity data
Coming from a CS/Econ/Finance background....
Efficiency as “the market fully processes all information” is decidedly untrue.
Efficiency as “Its very hard to arb markets without risk better than a passive strategy” is very true.
Most professional money managers don’t beat the market once you factor in fees. Many private equity funds produce outsize returns, but it’s based on higher risk and taking advantage of tax laws.
Over time, positive performance for mutual funds don’t persist. (If you’re in the top decile performance one year, you’re no more likely to be there next year)
Despite all of this, there are ways people make money. But it’s a small subset of professionals. It’s high frequency traders who find ways to cut the line on mounds of pennies. It inside information from hedge funds. Or earlier access to non public information. But it’s generally not in areas that normal people like you and me can access.
It inside information from hedge funds. Or earlier access to non public information. But it’s generally not in areas that normal people like you and me can access.
Asymmetric information is pretty far from what used to be said about the perfect market and rational actors. It's "there's a sucker born every minute" and "if it seems too good to be true it probably is" economics.
I might be misunderstanding what you're saying here, but are you sure you're right? Fama originally predicated the model of the efficient market (the efficient market hypothesis) on the idea of informational efficiency. Information asymmetry is a fundamental measure involved in the idealized model of an efficient market.
What you're mentioning about rational actors is actually a different topic altogether in economics.
Or have I misunderstood what you're getting at?
I was interested, so I did some research here.
Rational Choice Theory https://en.wikipedia.org/wiki/Rational_choice_theory
Rational Behavior https://www.investopedia.com/terms/r/rational-behavior.asp
> Most mainstream academic economics theories are based on rational choice theory.
> While most conventional economic theories assume rational behavior on the part of consumers and investors, behavioral finance is a field of study that substitutes the idea of “normal” people for perfectly rational ones. It allows for issues of psychology and emotion to enter the equation, understanding that these factors alter the actions of investors, and can lead to decisions that may not appear to be entirely rational or logical in nature. This can include making decisions based primarily on emotion, such as investing in a company for which the investor has positive feelings, even if financial models suggest the investment is not wise.
Behavioral finance https://www.investopedia.com/terms/b/behavioralfinance.asp
Bounded rationality > Relationship to behavioral economics https://en.wikipedia.org/wiki/Bounded_rationality
Perfectly rational decisions can be and are made without perfect information; bounded by the information available at the time. If we all had perfect information, there would be no entropy and no advantage; just lag and delay between credible reports and order entry.
Information asymmetry https://en.wikipedia.org/wiki/Information_asymmetry
Heed these words wisely: What foolish games! Always breaking my heart.
https://deepmind.com/blog/game-theory-insights-asymmetric-mu...
> Asymmetric games also naturally model certain real-world scenarios such as automated auctions where buyers and sellers operate with different motivations. Our results give us new insights into these situations and reveal a surprisingly simple way to analyse them. While our interest is in how this theory applies to the interaction of multiple AI systems, we believe the results could also be of use in economics, evolutionary biology and empirical game theory among others.
https://en.wikipedia.org/wiki/Pareto_efficiency
> A Pareto improvement is a change to a different allocation that makes at least one individual or preference criterion better off without making any other individual or preference criterion worse off, given a certain initial allocation of goods among a set of individuals. An allocation is defined as "Pareto efficient" or "Pareto optimal" when no further Pareto improvements can be made, in which case we are assumed to have reached Pareto optimality.
Which, I think, brings me to equitable availability of maximum superalgo efficiency and limits of real value creation in capital and commodities markets; which'll have to be a topic for a different day.
Show HN: React-Schemaorg: Strongly-Typed Schema.org JSON-LD for React
I have a slightly longer post describing this work and the reasoning behind it on dev.to[1].
[1]: https://dev.to/eyassh/react-schemaorg-strongly-typed-schemaorg-json-ld-for-react-4lhdhttps://dev.to/eyassh/react-schemaorg-strongly-typed-schemao...
Is there a good way to generate JSONschema and thus forms from schema.org RDFS classes and (nested, repeatable) properties?
By JSONschema do you mean [this standard](https://json-schema.org/)? I don't know of a tool that does that yet, the JSON Schema is general enough with allOf/anyOf that it should express schema.org schema as well.
Depends on the purpose here. With this, my goal was to speed up the Write-Update-Debug development loop. Depends on the use case, simply using Google's Structured Data Testing Tool [1] might be a better way to verify schema than JSON-schema?
[1]: https://search.google.com/structured-data/testing-toolThere are a number of tools for generating forms and requisite client and serverside data validations from JSONschema; but I'm not aware of any for RDFS (and thus the schema.org schema [1]). A different use case, for certain.
https://schema.org/docs/developers.html#defs
Definitely a cool area for exploration. I'm not aware of JSON Schema generators from RDFS either.
It should be possible to model the basics (nested structure, repeated structure, defining the "domain" and "range" of a value).
Schema.org definitions however have no conception of "required" values[], however, so some of the cool form validation we see in some of these tools might not apply here.
[*] _Consumers_ of Schema.org JSON-LD or microdata, however, might define their own data requirements. E.g. Google has some concept of required fields, which you can see when using the Structured Data Testing Tool.Consumer Protection Bureau Aims to Roll Back Rules for Payday Lending
From the article:
> The way payday loans work is that payday lenders typically offer small loans to borrowers who promise to pay the loans back by their next paycheck. Interest on the loans can have an annual percentage rate of 390 percent or more, according to a 2013 report by the CFPB. Another bureau report from the following year found that most payday loans — as many as 80 percent — are rolled over into another loan within two weeks. Borrowers often take out eight or more loans a year.
390%
From https://www.npr.org/2019/02/06/691944789/consumer-protection... :
> TARP recovered funds totalling $441.7 billion from $426.4 billion invested, earning a $15.3 billion profit or an annualized rate of return of 0.6% and perhaps a loss when adjusted for inflation.[2][3]
0.6%
Is your point that the US government should get into the payday lending business?
Lectures in Quantitative Economics as Python and Julia Notebooks
It's amazing how we are watching use cases for notebooks and spreadsheets converging. I wonder what the killer feature will be to bring a bigger chunk of the Excel world into a programmatic mindset... Or alternatively, whether we will see notebook UIs embedded in Excel in the future in place of e.g. VBA.
That’s not a bad idea. Spreadsheets are pure functional languages built that use literal spaces instead of namespaces.
Notebooks are cells of logic. You could conceivably change the idea of notebook cells to be an instance of a function that points to raw data and returns raw data.
Perhaps this just Alteryx though
This is brilliant.
I'm picturing the ability to write a Python function with the parameters being just like the parameters in an Excel function. You can drag the cell and have it duplicated throughout a row, updating the parameters to correspond to the rows next to it.
It would exponentially expand the power of excel. I wouldn't be limited to horribly unmaintainable little Excel functions.
VBA can't be used to do that, can it? As far as I understand (and I haven't investigated VBA too much) VBA works on entire spreadsheets.
Essentially, replace the excel formula `=B3-B4` with a Python function `subtract(b3, b4)` where Subtract is defined somewhere more conveniently (in a worksheet wide function definition list?).
This would require a reactive recomputing of cells to be anything like a spreadsheet. > Essentially, replace the excel formula `=B3-B4` with a Python function `subtract(b3, b4)`
as of now jupyter/ipython would not recompute `subtract(b3, b4)` if you change b3 or b4, this has positive and negative (reliance on hidden state and order of execution) effects.
I too would really like something like this, but I think it is pretty far away from where jupiter is now.
You can build something like this with Jupyter today.
> Traitlets is a framework that lets Python classes have attributes with type checking, dynamically calculated default values, and ‘on change’ callbacks. https://traitlets.readthedocs.io/en/stable/
> Traitlet events. Widget properties are IPython traitlets and traitlets are eventful. To handle changes, the observe method of the widget can be used to register a callback https://ipywidgets.readthedocs.io/en/stable/examples/Widget%...
You can definitely build interactive notebooks with Jupyter Notebook and JupyterLab (and ipywidgets or Altair or HoloViews and Bokeh or Plotly for interactive data visualization).
> Qgrid is a Jupyter notebook widget which uses SlickGrid to render pandas DataFrames within a Jupyter notebook. This allows you to explore your DataFrames with intuitive scrolling, sorting, and filtering controls, as well as edit your DataFrames by double clicking cells. https://github.com/quantopian/qgrid
Qgrid's API includes event handler registration: https://qgrid.readthedocs.io/en/latest/
> neuron is a robust application that seamlessly combines the power of Visual Studio Code with the interactivity of Jupyter Notebook. https://marketplace.visualstudio.com/items?itemName=neuron.n...
"Excel team considering Python as scripting language: asking for feedback" (2017) https://news.ycombinator.com/item?id=15927132
OpenOffice Calc ships with Python 2.7 support: https://wiki.openoffice.org/wiki/Python
Procedural scripts written in a general purpose language with named variables (with no UI input except for chart design and persisted parameter changes) are reproducible.
What's a good way to review all of the formulas and VBA and/or Python and data ETL in a spreadsheet?
Is there a way to record a reproducible data transformation script from a sequence of GUI interactions in e.g. OpenRefine or similar?
OpenRefine/OpenRefine/wiki/Jupyter
"Within the Python context, a Python OpenRefine client allows a user to script interactions within a Jupyter notebook against an OpenRefine application instance, essentially as a headless service (although workflows are possible where both notebook-scripted and live interactions take place. https://github.com/OpenRefine/OpenRefine/wiki/Jupyter
Are there data wrangling workflows that are supported by OpenRefine but not Pandas, Dask, or Vaex?
This interesting need to have a closer look, possibly refine can be more efficient? But haven't used it enough to know, just payed around with it a bit. Didn't realise you could combine it with jupyter.
There are undergraduate and graduate courses in each language:
Python version: https://lectures.quantecon.org/py/
Julia version: https://lectures.quantecon.org/jl/
Does anyone else find it strange that there is no real-world data in these notebooks? It's all simulations or abstract problems.
This gives me the sense, personally, that economists aren't interested in making accurate predictions about the world. Other fields would, I think, test their theories against observations.
pandas-datareader can pull data from e.g. FRED, Eurostat, Quandl, World Bank: https://pandas-datareader.readthedocs.io/en/latest/remote_da...
pandaSDMX can pull SDMX data from e.g. ECB, Eurostat, ILO, IMF, OECD, UNSD, UNESCO, World Bank; with requests-cache for caching data requests: https://pandasdmx.readthedocs.io/en/latest/#supported-data-p...
The scikit-learn estimator interface includes a .score() method. "3.3. Model evaluation: quantifying the quality of predictions" https://scikit-learn.org/stable/modules/model_evaluation.htm...
statsmodels also has various functions for statistically testing models: https://www.statsmodels.org/stable/
"latex2sympy parses LaTeX math expressions and converts it into the equivalent SymPy form" and is now merged into SymPy master and callable with sympy.parsing.latex.parse_latex(). It requires antlr-python-runtime to be installed. https://github.com/augustt198/latex2sympy https://github.com/sympy/sympy/pull/13706
IDK what Julia has for economic data retrieval and model scoring / cost functions?
If Software Is Funded from a Public Source, Its Code Should Be Open Source
From the US Digital Services Playbook [1]:
> PLAY 13
> Default to open
> When we collaborate in the open and publish our data publicly, we can improve Government together. By building services more openly and publishing open data, we simplify the public’s access to government services and information, allow the public to contribute easily, and enable reuse by entrepreneurs, nonprofits, other agencies, and the public.
> Checklist
> - Offer users a mechanism to report bugs and issues, and be responsive to these reports
> [...]
> - Ensure that we maintain contractual rights to all custom software developed by third parties in a manner that is publishable and reusable at no cost
> [...]
> - When appropriate, publish source code of projects or components online
> [...]
> Key Questions
> [...]
> - If the codebase has not been released under an open source license, explain why.
> - What components are made available to the public as open source?
> [...]
Apache Arrow 0.12.0
> Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. It also provides computational libraries and zero-copy streaming messaging and interprocess communication. Languages currently supported include C, C++, C#, Go, Java, JavaScript, MATLAB, Python, R, Ruby, and Rust.
Statement on Status of the Consolidated Audit Trail (2018)
U.S. Federal District Court Declared Bitcoin as Legal Money
"Application of FinCEN's Regulations to Persons Administering, Exchanging, or Using Virtual Currencies" (2013) https://www.fincen.gov/resources/statutes-regulations/guidan...
"Legality of bitcoin by country or territory" https://en.wikipedia.org/wiki/Legality_of_bitcoin_by_country...
"Know your customer" https://en.wikipedia.org/wiki/Know_your_customer
"Anti-money-laundering measures by region" https://en.wikipedia.org/wiki/Money_laundering#Anti-money-la...
"Anti-money-laundering measures by region > United States" https://en.wikipedia.org/wiki/Money_laundering#United_States
Post Quantum Crypto Standardization Process – Second Round Candidates Announced
> As the latest step in its program to develop effective defenses, the National Institute of Standards and Technology (NIST) has winnowed the group of potential encryption tools—known as cryptographic algorithms—down to a bracket of 26. These algorithms are the ones NIST mathematicians and computer scientists consider to be the strongest candidates submitted to its Post-Quantum Cryptography Standardization project, whose goal is to create a set of standards for protecting electronic information from attack by the computers of both tomorrow and today.
> “These 26 algorithms are the ones we are considering for potential standardization, and for the next 12 months we are requesting that the cryptography community focus on analyzing their performance,”
Links to the 17 public-key encryption and key-establishment algorithms and 9 digital signature algorithms are here: "Round 2 Submissions" https://csrc.nist.gov/Projects/Post-Quantum-Cryptography/Rou...
"Quantum Algorithm Zoo" has moved to https://quantumalgorithmzoo.org .
Ask HN: How do you evaluate security of OSS before importing?
What tools can I use to evaluate the security posture of an OSS project before I approve its usage with high confidence?
Oddly, whether a project has at least one CVE reported could be interpreted in favor of the project. https://www.cvedetails.com
Do they have a security disclosure policy? A dedicated security mailing list?
Do they pay bounties or participate in e.g Pwn2own?
Do they cryptographically sign releases?
Do they cryptographically sign VCS tags (~releases)? commits? `git tag -s` / `git commit/merge -S` https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work
Downstream packagers do sometimes/often apply additional patches and then sign their release with the repo (and thus system global) GPG key.
Whether they require "Signed-off-by" may indicate that the project has mature controls and possibly a formal code review process requirement. (Look for "Signed-off-by:" in the release branch (`git commit/merge -s/--signoff`)
How have they integrated security review into their [iterative] release workflow?
Is the software formally verified? Are parts of the software implementation or spec formally verified?
Does the system trust the channel? The host? Is it a 'trustless' system?
What are the single points of failure?
How is logging configured? To syslog?
Do they run the app as root in a Docker container? Does it require privileged containers?
If it has to run as root, does it drop privileges at startup?
Does the package have an SELinux or AppArmor policy? (Or does it say e.g. "just set SELinux to permissive mode)
Is there someone you can pay to support the software in an enterprise environment? Open or closed, such contacts basically never accept liability; but if there is an SLA, do you get a pro-rated bill?
As far as indicators of actual software quality:
How much test coverage is there? Line coverage or statement coverage?
Do they run static analysis tools for all pull requests and releases? Dynamic analysis? Fuzzing?
Of course, closed or open source projects may do none or all of these and still be totally secure, insecure, or unsecure.
This is a pretty extensive list. Thanks for sharing!
Ask HN: How can I use my programming skills to support nonprofit organizations?
Lately I've been thinking about doing programming for nonprofits, both because I want to help out with what I'm good at but also to hone my skills and potentially get some open source credit.
So far I've had a hard time finding nonprofit projects where I can just pick up something and start programming. I know about freecodecamp.org, but they force you to go through their courses, and as I already have multiple years of e
> - Reduce emissions with trusted analytics from a trusted platform. Analyzing carbon emissions from energy usage and company travel can be daunting and time-consuming. But with all your data flowing directly onto one platform, you can efficiently quantify your carbon footprint. Formulate a climate action plan for your company from a single source of truth, built on our trusted and secure data platform.
> - Take action with data-driven insights. Prove to customers, employees, and potential investors your commitment to carbon-conscious and sustainable practices. Offer regulatory agencies a clear snapshot of your energy usage patterns. Extrapolate energy consumption and track carbon emissions with cutting-edge analytics — and take action.
> - Tackle carbon accounting audits in weeks instead of months. Carbon analysis can be an overwhelming time commitment, even a barrier to action for companies that want to get it right. Use preloaded datasets from the U.S. EPA, IPCC, and others to accurately assess your carbon accounting. Streamline your data gathering and climate action plan with embedded guides and user flows.
> - Empower decision makers with executive-ready dashboard data. Evaluate corporate environmental impact with rich data visualization and dashboards. Track energy patterns and emission trends, then make the business case to executives. Once an organization understands its carbon footprint, decision makers can begin to drive sustainability solutions.