A lot of us feel a little lost as we start to use Clojure and try to make sense of how we should model data. If we came from a schema or type-based system many of us are tempted to apply those skills directly. We have a notion (reinforced by talks we watch) that these namespaced keywords are important, but we often struggle to apply those concepts pragmatically.

In this talk, I’ll discuss how the ideas of context-free federated names (a.k.a. namespace-qualified keywords) have become more concrete and pragmatic to me during my use of them at NuBank. I’ll talk about my realizations on how they can be used in practical ways that lead to a high level of clarity in our data models and communication systems, and open up better flexibility and reach.

Finally, I’ll relate these concepts to recent advances in the communication of this kind of data at the systems level as a graph, and show how industry standards like GraphQL continue to suffer from problems of composition and generality, while Clojure’s approach leads to an open information system with maximal utility: the Maximal Graph.

Pull is a declarative way to make hierarchical (and possibly nested) selections of information about entities. Pull applies a pattern to a collection of entities, building a map for each entity. Pull is available

• via the standalone pull API Peer | Client
• via the standalone pull-many API in Peer
• as a find specification in query

Graph is a simple, declarative abstraction to express compositional structure.

Declarative means that we should explicitly list a system’s components and dependencies in a way that is accessible to our tooling. This solves the issues of the previous section, enabling abstractions over a system’s components as well as reasoning about the composition as a whole. Of course, this idea is not new; for example, it is the basis of graph computation frameworks like Pregel, Dryad, and Storm, and existing libraries for system composition such as react.

Our primary objective in Graph is to distill this idea to its simplest, most idiomatic expression in Clojure, our language of choice. Concretely, a Graph is just a Clojure map of functions that can depend on the outputs of other functions. Because Graphs are just ordinary data, we can manipulate them for free using our favorite existing tools, making Graphs trivially easy to create, modify, run, reason about, test, and build upon. Put simply, Graph is an [FCA][swe] for composition.

It is better to have 100 functions operate on one data structure than 10
functions on 10 data structures.
  -- Alan Perlis

As a first attempt in this direction, we could rewrite our stats example as a Clojure map, turning let variables into keywords and wrapping each of the corresponding value expressions in anonymous functions.

This gets us 90% of the way there. The individual components of the computation are now explicit, but the dependency information is still missing. For instance, there’s no way for our tools to know that the m in the arguments to the :v function refers to the mean computed in the second step of the graph – after compilation, it’s just the first argument to an anonymous function.

• GitHub - plumatic/plumbing: Prismatic's Clojure(Script) utility belt · GitHub
• Plumbing 0.4.0 API documentation

A Clojure(Script) library for declarative data description and validation.

One of the difficulties with bringing Clojure into a team is the overhead of understanding the kind of data (e.g., list of strings, nested map from long to string to double) that a function expects and returns. While a full-blown type system is one solution to this problem, we present a lighter weight solution: schemas. (For more details on why we built Schema, check out this post.

• Referenced post: Schema for Clojure(Script) Data Shape Declaration and Validation

Schema is a rich language for describing data shapes, with a variety of features:

• Data validation, with descriptive error messages of failures (targeted at programmers)
• Annotation of function arguments and return values, with optional runtime validation
• Schema-driven data coercion, which can automatically, succinctly, and safely convert complex data types (see the Coercion section below)
• Other
  • Schema is also built into our plumbing and fnhouse libraries, which illustrate how we build services and APIs easily and safely with Schema
  • Schema also supports experimental clojure.test.check data generation from Schemas, as well as completion of partial datums, features we've found very useful when writing tests as part of the schema-generatorslibrary

edn is an extensible data notation. A superset of edn is used by Clojure to represent programs, and it is used by Datomic and other applications as a data transfer format. This spec describes edn in isolation from those and other specific use cases, to help facilitate implementation of readers and writers in other languages, and for other uses.

InfluxDB 2.7 is the platform purpose-built to collect, store, process and visualize time series data. Time series data is a sequence of data points indexed in time order. Data points typically consist of successive measurements made from the same source and are used to track changes over time. Examples of time series data include:

• Industrial sensor data
• Server performance metrics
• Heartbeats per minute
• Electrical activity in the brain
• Rainfall measurements
• Stock prices

This multi-part tutorial walks you through writing time series data to InfluxDB 2.7, querying that data, processing and alerting on the data, and then visualizing the data.

The Simple & Reliable Logs Solution That Scales
High Performance | Open Source | Logging Solution

VictoriaMetrics is a fast, cost-saving, and scalable solution for monitoring and managing time series data. It delivers high performance and reliability, making it an ideal choice for businesses of all sizes.

• VictoriaMetrics/VictoriaMetrics: VictoriaMetrics: fast, cost-effective monitoring solution and time series database

![VictoriaLogs](http://The Simple & Reliable Logs Solution That Scales
High Performance | Open Source | Logging Solution)

cowsql (/ˈkaʊ,siːkwəl/ listen) is a C library that implements an embeddable and replicated SQL database engine with high availability and automatic failover.

cowsql extends SQLite with a network protocol that can connect together various instances of your application and have them act as a highly-available cluster, with no dependency on external databases.

The name "cowsql" loosely refers to the "pets vs. cattle" concept, since it's generaly fine to delete or rebuild a particular node of an application that uses cowsql for data storage.

Automate code & data workflows with interactive notebooks
Get rid of scripts, manual steps, and outdated docs. Use Elixir and Livebook to share knowledge, deploy apps, visualize data, run machine learning models, debug systems, and more!

NooBaa is a highly customizable and dynamic data gateway for objects, providing data services such as caching, tiering, mirroring, dedup, encryption, compression, over any storage resource including S3, GCS, Azure Blob, Filesystems, etc.

The goal is to simplify data flows for admins by connecting to any of the storage silos from private or public clouds, and providing a single scalable data services, using the same S3 API and management tools. NooBaa allows full control over data placement with dynamic policies per bucket or account.

• noobaa.io
  

Abstract

This data sheet describes the ZED-F9P high precision module with multi-band GNSS receiver. The module provides multi-band RTK with fast convergence times, reliable performance and easy integration of RTK for fast time-to-market. It has a high update rate for highly dynamic applications and centimeter-level accuracy in a small and energy-efficient module.

• ZED-F9P module, u-blox F9 high precision GNSS module

GpsPrune is an application for viewing, editing and converting coordinate data from GPS systems. Basically it's a tool to let you play with your GPS data after you get home from your trip.
GpsPrune screenshot

Screenshot from a Linux system showing
the map view and altitude profile

It can load data from arbitrary text-based formats (for example, any tab-separated or comma-separated file) or Xml, or directly from a GPS receiver. It can display the data (as map view using openstreetmap images and as altitude

The Meteostat JSON API provides simple access to a large archive of historical weather and climate data. The records are queried by weather station or geo location and can be filtered by specifying a date range and other optional parameters. The API is available via this URL:

https://api.meteostat.net

Access to the API requires users to send their API key along with every request. All API endpoints return a JSON object that contains two properties: meta and data. The meta object provides general information about the data output and debugging information. For instance, the source string which holds the names of the organizations which provided the raw data. The data property is either an object or an array that contains the actual data output.

All API endpoints are accessible using HTTP Get requests. For debugging, please utilize the HTTP status code of the response.

!/bin/bash

#
# (c) Frank Matthieß 2020
#
#

NAME=$(basename $0)
info() { echo "$*">&2; }
error() { info "ERROR: $*"; }

APIURL=https://api.opensensemap.org/boxes
TMPFILE=$(mktemp)
STIDFILE=$(mktemp)
trap 'rm -f ${TMPFILE} ${STIDFILE} ' INT EXIT

# At the map, click on an station icon and you get, for example, this url:
#  https://opensensemap.org/explore/5b4d11485dc1ec001b5452c7
#
# Add the id tring at the end of the url with a proper name to the sensor id list

# Sensor id list
cat<<EOSTIDS>$STIDFILE
5c22ef44919bf8001a10acc7    Berlin Charlottenburg
5a9c2087bc2d4100191d6ff9    Stuttgart Unteraichen (A8)
5b4d11485dc1ec001b5452c7    München Fakultät Informatik
EOSTIDS

#
#
# Adjust the field length of the output
llen=0
while read stid location; do
    l=$(wc -c <<< "$location")
    if [ $llen -lt $l ]; then
        llen=$l
    fi
done < $STIDFILE

#
# .. and get the need values via opensensemap api
#
while read stid location; do
    curl -sL ${APIURL}/${stid} > ${TMPFILE}
    reallocation=$(jq .name < ${TMPFILE})
    temperature=$(jq '.sensors[]  | select(.title|test("Temperatur")) | .title, .lastMeasurement.value, .unit' < ${TMPFILE} | xargs )
    humidity=$(jq '.sensors[]  | select(.title|test("rel. Luftfeuchte")) | .title, .lastMeasurement.value, .unit' < ${TMPFILE} | xargs )
    rawcoordinates=$(jq '.currentLocation.coordinates[]' < ${TMPFILE} | xargs )
    coordinates=$(while read lon lat heigh; do echo "https://www.openstreetmap.org/?mlat=${lat}&mlon=${lon}#map=12/${lat}/$lon"; done <<< "${rawcoordinates}")

    # 
    printf "%-${llen}s:\n\tStationsname : %s\n\t%s\n\t%s\n\tOpenstreetmap: %s\n"    \
        "${location}"               \
            "${reallocation//\"/}"      \
            "${temperature}"        \
            "${humidity}"           \
            "${coordinates}"

done < $STIDFILE

# vim:set ts=8 sts=8 sw=8 ft=sh ai tw=80:

Cerberus provides powerful yet simple and lightweight data validation functionality out of the box and is designed to be easily extensible, allowing for custom validation. It has no dependencies and is thoroughly tested from Python 2.7 up to 3.6, PyPy and PyPy3.

Despite overall Elasticsearch stability, it is still possible for a cluster to get into a "red" state. One of the reasons for that to happen is if an index becomes corrupt. This can be caused by an abrupt loss of power, hardware failure or—more commonly—running out of disk space. In this post we'll discuss how to bring the cluster to a healthy state with minimal or no data loss in such situation.

Simple yet Powerful Turnkey Solution to Build Clouds and Manage Data Center Virtualization