References

The advent of multicore processors as the standard computing platform will force major changes in software design.

A defining characteristic of hypermedia systems is the presence of hypermedia controls. In this paper we examine hypermedia controls as found "in the wild", in particular in the World Wide Web. These hypermedia controls are analyzed to derive a functional hypermedia mechanic that can be used to characterize them. This functional mechanic is used to create first an informal and then formal definition of the term "hypermedia control". Using this formal definition we then derive a generalization of the concept, referring contextually to the World Wide Web. We then examine two hypermedia technologies that implement this concept of generalized hypermedia controls: htmx, which does so in the context of the WWW and Hyperview, which does so in a mobile context.

Keywords: htmx,Hypermedia,Hypermedia Controls

Summary There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

Facebook takes performance monitoring seriously. Performance issues can impact over one billion users so we track thousands of servers, hundreds of PB of daily network traffic, hundreds of daily code changes, and many other metrics. We require latencies of under a minute from events occuring (a client request on a phone, a bug report filed, a code change checked in) to graphs showing those events on developers' monitors.Scuba is the data management system Facebook uses for most real-time analysis. Scuba is a fast, scalable, distributed, in-memory database built at Facebook. It currently ingests millions of rows (events) per second and expires data at the same rate. Scuba stores data completely in memory on hundreds of servers each with 144 GB RAM. To process each query, Scuba aggregates data from all servers. Scuba processes almost a million queries per day. Scuba is used extensively for interactive, ad hoc, analysis queries that run in under a second over live data. In addition, Scuba is the workhorse behind Facebook's code regression analysis, bug report monitoring, ads revenue monitoring, and performance debugging.

Large language models (LLMs), such as GPT-4, are revolutionizing software's ability to understand, process, and synthesize language. The authors of this paper believe that this advance in technology is significant enough to prompt introspection in the data management community, similar to previous technological disruptions such as the advents of the world wide web, cloud computing, and statistical machine learning. We argue that the disruptive influence that LLMs will have on data management will come from two angles. (1) A number of hard database problems, namely, entity resolution, schema matching, data discovery, and query synthesis, hit a ceiling of automation because the system does not fully understand the semantics of the underlying data. Based on large training corpora of natural language, structured data, and code, LLMs have an unprecedented ability to ground database tuples, schemas, and queries in real-world concepts. We will provide examples of how LLMs may completely change our approaches to these problems. (2) LLMs blur the line between predictive models and information retrieval systems with their ability to answer questions. We will present examples showing how large databases and information retrieval systems have complementary functionality.

https://x.com/norpadon/status/1828905291242467678. [ bib | arXiv | http ]

Keywords: LLM

Keywords: RAG

Keywords: dblp

Keywords: exoplanets,N-Body,programming languages,Rust,simulations

In many environments, multi-threaded code is written in a language that was originally designed without thread support (e.g. C), to which a library of threading primitives was subsequently added. There appears to be a general understanding that this is not the right approach. We provide specific arguments that a pure library approach, in which the compiler is designed independently of threading issues, cannot guarantee correctness of the resulting code.We first review why the approach almost works, and then examine some of the surprising behavior it may entail. We further illustrate that there are very simple cases in which a pure library-based approach seems incapable of expressing an efficient parallel algorithm.Our discussion takes place in the context of C with Pthreads, since it is commonly used, reasonably well specified, and does not attempt to ensure type-safety, which would entail even stronger constraints. The issues we raise are not specific to that context.

Keywords: c,implementation,language,operatingsystem,programming,semantics,thread

First Page of the Article

Keywords: software_engineering

Keywords: *file-import-13-06-09

Keywords: *file-import-13-06-09

While LLMs can often infer correct mapping functions inductively, they struggle to apply these functions deductively, especially for unfamiliar tasks.

👨‍🔧 Definition: Deductive reasoning is moving from general principles to specific conclusions, like applying given rules to solve problems, while inductive reasoning involves inferring general patterns or rules from specific observations. Deductive reasoning starts with a hypothesis and derives specific outcomes, whereas inductive reasoning formulates broad generalizations from individual instances.

📌 The paper introduces a novel framework called SolverLearner to isolate and evaluate pure inductive reasoning abilities of LLMs.

📌 SolverLearner uses a two-stage approach: 1) Function Proposal - LLMs learn an input-output mapping function from few-shot examples. 2) Function Execution - The learned function is applied through external code interpreters to solve test queries, removing LLM-based deductive reasoning.

📌 The study evaluates LLMs on four tasks: arithmetic in different bases, basic syntactic reasoning with altered word orders, spatial reasoning with modified coordinate systems, and cipher decryption.

📌 For each task, performance is compared across deductive settings (zero-shot and few-shot with explicit mapping functions) and inductive settings (few-shot without mapping functions and SolverLearner).

📌 Results show LLMs struggle with deductive reasoning, especially for "counterfactual" tasks rarely seen in pretraining. However, they demonstrate strong inductive abilities through SolverLearner, often achieving perfect performance.

📌 The effectiveness of inductive reasoning varies between models - GPT-4 consistently outperforms GPT-3.5 in learning correct input-output mappings.

via - https://x.com/rohanpaul_ai/status/1828778688151683227. [ bib | arXiv | http ]

Power-law distributions occur in many situations of scientific interest and have significant consequences for our understanding of natural and man-made phenomena. Unfortunately, the detection and characterization of power laws is complicated by the large fluctuations that occur in the tail of the distribution -- the part of the distribution representing large but rare events -- and by the difficulty of identifying the range over which power-law behavior holds. Commonly used methods for analyzing power-law data, such as least-squares fitting, can produce substantially inaccurate estimates of parameters for power-law distributions, and even in cases where such methods return accurate answers they are still unsatisfactory because they give no indication of whether the data obey a power law at all. Here we present a principled statistical framework for discerning and quantifying power-law behavior in empirical data. Our approach combines maximum-likelihood fitting methods with goodness-of-fit tests based on the Kolmogorov-Smirnov statistic and likelihood ratios. We evaluate the effectiveness of the approach with tests on synthetic data and give critical comparisons to previous approaches. We also apply the proposed methods to twenty-four real-world data sets from a range of different disciplines, each of which has been conjectured to follow a power-law distribution. In some cases we find these conjectures to be consistent with the data while in others the power law is ruled out.

Keywords: machine-learning

An abstract is not available.

Keywords: history,logic-programming,programming-languages,survey

The performance of trend following strategies can be ascribed to the difference between long-term and short-term realized variance. We revisit this general result and show that it holds for various definitions of trend strategies. This explains the positive convexity of the aggregate performance of Commodity Trading Advisors (CTAs) which -- when adequately measured -- turns out to be much stronger than anticipated. We also highlight interesting connections with so-called Risk Parity portfolios. Finally, we propose a new portfolio of strangle options that provides a pure exposure to the long-term variance of the underlying, offering yet another viewpoint on the link between trend and volatility.

Keywords: Convexity,CTA,Option Hedging,Protection,Risk Parity,Tail Risk,Trend Following,Variance Swap,Volatility

The alignment of generative models with human feedback has significantly improved the performance of natural language generation tasks. For large language models (LLMs), alignment methods like reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) have consistently worked better than just supervised fine-tuning (SFT) alone based on current

As CPU cores become both faster and more numerous, the limiting factor for most programs is now, and will be for some time, memory access. Hardware designers have come up with ever more sophisticated memory handling and acceleration techniques–such as CPU caches–but these cannot work optimally without some help from the programmer. Unfortunately, neither the structure nor the cost of using the memory subsystem of a computer or the caches on CPUs is well understood by most programmers. This paper explains the structure of memory subsystems in use on modern commodity hardware, illustrating why CPU caches were developed, how they work, and what programs should do to achieve optimal performance by utilizing them.

Keywords: hardware,linux,memory,unix

Keywords: heartbleed,internet-wide scanning,openssl,security

Source code of azlen.me, that implmenets Andy Matuschak's Evergreen Notes style website.

An exploration of the evolution and meaning of the Forth programming language and its context in history.

Keywords: forth,programming

Keywords: Extensible systems,fast servers,network services,OS structure

Keywords: GPT,LLM,RAG

An Inconvenient Observation Knowledge workers are bad at working. I say this because unlike every other skilled labor class in the history of skilled labor, ... Read more

A simple toolchain that Koding Kitty uses for building its web.

Keywords: toread

Reward models (RMs) are at the crux of successful RLHF to align pretrained models to human preferences, yet there has been relatively little study that focuses on evaluation of those reward models. Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models and which values are embedded in them. To date, very few descriptors of capabilities, training methods, or open-source reward models exist. In this paper, we present RewardBench, a benchmark dataset and code-base for evaluation, to enhance scientific understanding of reward models. The RewardBench dataset is a collection of prompt-win-lose trios spanning chat, reasoning, and safety, to benchmark how reward models perform on challenging, structured and out-of-distribution queries. We created specific comparison datasets for RMs that have subtle, but verifiable reasons (e.g. bugs, incorrect facts) why one answer should be preferred to another. On the RewardBench leaderboard, we evaluate reward models trained with a variety of methods, such as the direct MLE training of classifiers and the implicit reward modeling of Direct Preference Optimization (DPO), and on a spectrum of datasets. We present many findings on propensity for refusals, reasoning limitations, and instruction following shortcomings of various reward models towards a better understanding of the RLHF process. [ bib | arXiv ]

Keywords: LLM,Reward Models

Keywords: LLM,Text Embeddings

Keywords: llm

https://github.com/ggerganov/llama.cpp/issues/2555

via - https://news.ycombinator.com/item?id=39890570. [ bib | http ]

I wrote 84 new matmul kernels to improve llamafile CPU performance.

Keywords: functional programming,hypervisor,microkernel

Keywords: functional programming,hypervisor,microkernel

Statistical training helps individuals analyze and interpret data. However, the emphasis placed on null hypothesis significance testing in academic training and reporting may lead researchers to interpret evidence dichotomously rather than continuously. Consequently, researchers may either disregard evidence that fails to attain statistical significance or undervalue it relative to evidence that attains statistical significance. Surveys of researchers across a wide variety of fields (including medicine, epidemiology, cognitive science, psychology, business, and economics) show that a substantial majority does indeed do so. This phenomenon is manifest both in researchers’ interpretations of descriptions of evidence and in their likelihood judgments. Dichotomization of evidence is reduced though still present when researchers are asked to make decisions based on the evidence, particularly when the decision outcome is personally consequential. Recommendations are offered. This paper was accepted by Yuval Rottenstreich, judgment and decision making.

Keywords: *file-import-13-06-09

Keywords: benchmarks,mapreduce,parallel database

Keywords: stat_theory

Keywords: cloud computing,clustering,databases,internet of things,partitioning,time-series data

Keywords: LLM,RAG

Keywords: LLM

Productivity is one of my pet topics, because it's always dogged me a bit, especially early in my career.  I'd pull long days and nights and...

Keywords: *file-import-13-06-09

Keywords: generative-ai,LLM

single file distributable web server

We present AutoGen, an open-source framework that allows developers to build LLM applications via multiple agents that can converse with each other to accomplish tasks. AutoGen agents are customizable, conversable, and can operate in various modes that employ combinations of LLMs, human inputs, and tools. Using AutoGen, developers can also flexibly define agent interaction behaviors. Both natural language and computer code can be used to program flexible conversation patterns for different applications. AutoGen serves as a generic infrastructure to build diverse applications of various complexities and LLM capacities. We provide many examples to build effective applications for domains ranging from mathematics, coding, question answering, operations research, online decision-making, entertainment, etc.

We accomplish these results by designing simple LM-centric commands and specially-built input and output formats to make it easier for the LM to browse the repository, view, edit and execute code files. We call this Agent-Computer Interface (ACI) and build the SWE-agent repository to make it easy to iterate on ACI design for repository-level coding agents. [ bib | http ]

Keywords: LLM

Abstract

We posit that to achieve superhuman agents, future models require super- human feedback in order to provide an adequate training signal. Current approaches commonly train reward models from human preferences, which may then be bottlenecked by human performance level, and secondly these separate frozen reward models cannot then learn to improve during LLM training. In this work, we study Self-Rewarding Language Models, where the language model itself is used via LLM-as-a-Judge prompting to provide its own rewards during training. We show that during Iterative DPO training that not only does instruction following ability improve, but also the ability to provide high-quality rewards to itself. Fine-tuning Llama 2 70B on three iterations of our approach yields a model that outperforms many existing systems on the AlpacaEval 2.0 leaderboard, including Claude 2, Gemini Pro, and GPT-4 0613. While there is much left still to explore, this work opens the door to the possibility of models that can continually improve in both axes. [ bib | arXiv | http ]

Keywords: LLM,Reward Models