k.LAB: a semantic web platform for science

Additional server components fill specific needs on the k.LAB network and are less commonly needed at partner sites. Among them the following are noteworthy:

Other, less critical server components are in development and are not discussed here. Among these, a statistics server collects anonymized information from successful and unsuccessful resolutions and processes them using machine learning techniques to improve the resolution algorithm.

The set of active, connected nodes and engines at any given time forms what can be seen collectively as a distributed container, where scientific knowledge is found in three layers handling information at increasing levels of abstraction: the resources, semantic and reactivity layers. The first can be seen as a data curation platform based on modern linked data concepts, using a generalized and flexible data model. Semantic and reactive content for the platform is developed in the respective layers using two specialized languages, k.IM for semantic resources and k.Actors for reactive behaviors and applications. The k.Modeler IDE provides drag-and-drop interfaces to build resources and a specialized environment for k.LAB projects, to ease editing and debugging of the k.IM and k.Actors languages.

The separation of concerns and APIs in the three layers maximizes their value: for example, the resources layer can be seen through different semantics, therefore serving different purposes in different networks by reinterpreting it through the logical "lens" of a differently configured semantic layer.

The k.LAB system can be accessed through (1) client software (usually an application running in a browser within the k.LAB Explorer web platform, or the k.LAB integrated development environment (IDE), k.Modeler) or (2) through its API by software applications. Providers of content may use the IDE or, in the near future, a content provider web interface available on all k.LAB Nodes, including of course any nodes deployed at the provider’s end. All users must be authenticated through a valid, secure certificate, which also establishes the semantic worldview of reference and any user permissions for the certificate holder. Content in all layers may be made available in public form or be linked to specific users or groups by its owners; access permissions are a mandatory part of metadata for all informational assets.

In addition to uploading content to existing nodes, institutionals contributors can use k.LAB Node software to deploy sites that contribute to the k.LAB network while retaining full control of all distribution details. Nodes are deployed as containers that can be easily set up and authorized by certified partners. k.LAB’s distributed paradigm supports an approach where (1) information remains under the ownership of its authoritative sources while (2) maximizing its availability and interoperability, (3) and compatibly with both public and commercial services, thanks to careful attribution of ownership and state-of-the-art encryption, access control and security.

Diverse, extendible sourcing of information for resources is enabled through the use of adapters, software plug-ins that adapt a specific data or service format to the API. The adapter identifier and parameters are specified in the metadata associated to the URN and used to select the methods for contextualization, import, export and indexing. Adapters are made available as k.LAB components, installable in k.LAB Engines and k.LAB Nodes, and can be extended by developers using the Java API to support formats and services not yet available. External APIs (e.g. datacubes) can be supported by deploying a bridge adapter to one or more k.LAB Nodes.

To date, adapters for many file formats (CSV, spatial raster and vector data, NetCDF), protocols (WCS, WFS, OpenDAP, SDMX) and specialized services (OpenStreetMap, weather station data bridging to multiple databases and sources) are available and others (such as RDF/SPARQL) are in development. Other adapters enable specialized services, like scale-dependent selection of hierarchically organized datasets such as administrative regions or river basins. URN parameters can be added to the base URN to trigger specialized processing at the node’s end, such as resolution-dependent simplification of polygons, selection of interpolation methods, or any other adapter-dependent option that will best suit the desired semantics.

Resources start their life as local within a user project, imported from files or through a resource editor integrated with the k.Modeler client software. Such local resources go through a validation process, meant to ensure that the k.LAB runtime can perform any operations on the resource that may be required during contextualization: for example, spatial data must have proper projections and valid polygons throughout. When a local resource is accepted, it can be used inside the project that contains it or in any other project that shares the same local workspace, but it is not visible to other k.LAB users. Local resources may be sufficient for the needs of a specific, short-term project, but the natural lifecycle of a resource continues with publication, which makes it available across the k.LAB network.

Publication of a resource is conditional on further validation; resources with incomplete metadata, licensing or ownership information are not accepted by the software. Successful publication uploads the resource to the staging area of a chosen k.LAB node, where it can be made available for general use and further edited in-place by its owner. Every edit of a published resource creates a new version of the resource and full history is kept. Published resources are independent of projects and obtain a unique, persistent URN; the hosting k.LAB Node may optimize their data content for faster serving and automatically mirror the resource to other nodes for increased availability. The resource publisher may choose to make access private (the default), available only to selected users or groups of users, or publicly accessible. URNs can be resolved by any node or engine through a distributed resolution service, and used in k.IM models without the need for any registration or download, as long as the user is connected to the k.LAB network. A model that references a URN that the running user has no access to will be automatically deactivated and not participate in resolution.

At the time of this writing, the staging "tier" of the resource layer is the only one enabled in k.LAB Node software. It is envisioned that an iterative resource review process, operated with participants drawn from the scientific community, will be used to promote resources to higher-ranking tiers, the level of which may affect the resolution algorithm (which will also incorporate user feedback and machine-learned usage statistics as resources are used in models). This process may eventually involve the attribution of a DOI to resources, resolved both through standard DOI proxy servers and directly by k.LAB, in which case the DOI may replace the URN in semantic models.

In simple cases, the query "observe observable in context" is answered by locating a model annotating a data source as an observation of the specified observable. For example, setting the context to a geographical region (e.g. a country’s extent with a spatial grid model at 100m resolution and temporal context, e.g. the year 2010) and querying an observable such as geography:Elevation in m may retrieve, among others, the following model:

which annotates a network-available resource specified by the URN im.data:geography:morphology:dem90 as an observation of the geography:Elevation concept. The URN gives access to metadata including the original spatial and temporal coverage and resolution, through which the model, whose semantics is identical to the query’s, can be ranked for match to the context. If the model is deemed to be the best match, the k.LAB Engine will translate it into a set of processing steps (in this case simply a resource retrieval operation plus any necessary mediation) and pass the resulting dataflow to the runtime to compute and produce the resulting observation, in this case a raster map of elevation, with 100m resolution, reflecting the boundaries and time of the context. The dataflow will include any necessary reprojection, resampling, or unit transformation to match the query and the context. Other models may compete for the choice, made on the basis of criteria such as resolution and extent match, specificity, semantic match, and including criteria such as peer review results or usage feedback for the original data. If the chosen model only partially covers the context, additional models may contribute to its complete characterization, as long as they specify a compatible observable and their ranks are close enough.

Besides such simple and direct matches, machine reasoning backed by an observation-centered ontological framework can enable more sophisticated observation tasks that do not correspond to readily available annotations and are normally only possible through specialized, time-consuming work. In a straightforward example, attributes such as im:Normalized may be prepended to another observable to reinterpret the result, where the attribute would be resolved to an independent model (model im:Normalized using <normalization function>), possibly restricted to certain classes of observables (e.g. model im:Normalized of im:Quantity …​ to restrict its application to numerically quantifiable observables) and used to modify a straight observation of geography:Elevation if the normalized observable cannot be resolved directly. More interestingly, resolution strategies may cross inherency barriers to infer the best observation strategy when a direct match is not available. For example, a hypothetical query for (ecology:AboveGround ecology:Biomass) of biology:Eucalyptus biology:Tree ^[2] operated in the same country context would refer, by virtue of the inherency operator of, to a quality (above-ground biomass) inherent but to a particular subset (Eucalyptus) of the observations of a secondary subject (Tree) located in the primary context of the query (a geographical region). It would be resolved by the following strategy:

Similar reasoning strategies can be applied to diverse situations, using semantic inference driven by the phenomenological understanding of the entities involved and the observation process applied to them. For example, a query for presence of biology:Tree that cannot be directly resolved could be satisfied by a model of (ecology:AboveGround ecology:Biomass) of biology:Tree because biomass (a im:Mass in a higher-level ontology) is an extensive property, therefore its non-zero value implies the existence of its inherent subject. The presence can be computed as a true/false value attributed to the context wherever the biomass of any tree is nonzero. In another commonly encountered use case, qualities that can only be correctly computed in specific contexts (for example hydrological qualities, such as "upstream area", which only produce correct results when computed in a correctly delineated river basin) can be automatically computed in arbitrary contexts. To do so, k.LAB first looks up a model to delineate all the relevant contexts (river basins) intersecting the areas, then applies the necessary models to compute the qualities inherently to those, then re-distributes the values over the desired context. Such behavior can be automated using a concept definition such as

which incorporates the "natural" context RiverBasin in the semantics of a new quality; alternatively, and more correctly if the RiverBasin context is required only because of modeling constraints and not directly implied by the concept, the observable can be left unconstrained and models can be defined as

In either case, the within operator mandates a RiverBasin context for the quality, which will trigger the distributed resolution process described previously whenever the observable is queried in any context where river basins can be observed. The same considerations hold for more complex observables such as processes, which have the ability to affect the value of qualities through time and to generate events or other objects; these, in turn, can become the context for other qualities or processes. The ability to automatically negotiate mediations based on inherency and phenomenological reasoning dramatically improves the capability of connecting diverse models without error, offering integration possibilities orders of magnitude beyond those allowed by the mere semantic matching of observables to models. Conventional approaches to such tasks require substantial planning, technical expertise and time.

Much of k.LAB’s power comes from the fact that component models pertaining to the different aspects of a larger modeling problem may be provided and shared by independent experts, with no need for any coordination beyond adherence to the same worldview. Each new model can serve multiple potential purposes and does not just add to, but rather multiplies the value of preexisting knowledge on the platform when interacting with it, just like words in natural language. The power of the resulting paradigm shift becomes obvious when the problem area addressed by modeling spans multiple disciplines, expertise types and languages, emphasizing the importance of a collaboratively built and endorsed worldview.

Both annotation and inference, as described above, require a common set of ontologies that define the realm of knowledge that can be integrated and conform with the foundational principles of k.LAB’s observational model. We refer to this set of ontologies as the wordlview, a set of k.IM projects that are automatically synchronized to all users that adopt it. A worldview is linked to each user profile and associated certificate that connects each k.LAB Node to the k.LAB network; only engines and nodes that adopt the same worldview as the user’s are seen in that user’s k.LAB session.

Because a worldview is meant to describe observation of reality, not reality itself, it is naturally aware of scale; its semantics differentiates observables not only by phenomenological nature but also by the nature of the observation process applicable to them. For example, k.LAB distinguishes events from processes, a distinction that has no real epistemological rationale (and does not exist in ontologies such as BFO). Yet this distinction is fundamental from an observational perspective, as events are countable entities that must be instantiated, producing zero or more independent observations, prior to resolution (while only one instance of the same process may exist within the subject that provides a context for it). The range of scales of observation is key to the compatibility of worldviews: a single worldview can easily address the wide range of problems that are "visible" at the scale of observation of a human observer, encompassing for example economic, ecological and social phenomena. However, it would be difficult to maintain meaning if that same worldview was also used to annotate problems at extremely small or large observational scales (relevant to e.g. quantum physics or general relativity, respectively).

The development of a worldview is a large collaborative endeavor, whose success is essential to the full fruition of the k.LAB paradigm. To date, only one worldview (the im worldview, for Integrated Modeling) is being developed, initially by the k.LAB team and an extended group of collaborators. This worldview currently consists of Tier 1 namespaces, covering a set of disciplinary realms with only enough detail to enable k.LAB’s current applications. As applications of k.LAB grow, a process for the collaborative development, versioning and maintenance of the Tier 1 IM worldview will become increasingly important. Tier 2 namespaces will be defined to specialize and add semantic detail to the corresponding Tier 1 namespaces: for example, the Tier 1 hydrology namespace will be complemented by a project containing hydrology.xxx namespaces for each field of hydrology needed by specialized applications. Such Tier 2 projects will be tied to user groups that each user can opt into through their user profile on the k.LAB hub, so that those users can automatically access any projects and models that require Tier 2 concepts to be understood by the system. This modular approach will enable specific user groups to control the development of the needed terminology while remaining compatible with the core Tier 1 concepts, and allow a scaled and coordinated development of the knowledge base without overwhelming those users not needing highly domain-specific detail. The semantic server described in the introduction will recognize the user groups and provide suggestions for annotations matching the user’s chosen areas of expertise and level of detail.

An important part of modeling is the adaptation of a computation to known data, so that it can best reproduce a known output from a known set of inputs, in order to increase confidence in predicted results when the model is run with unknown inputs. The main use cases for this activity are machine learning, which iteratively trains a statistical model until it produces the best fit to known data, and model calibration or data assimilation, used to find the optimal parameterization of mechanistic models. No modeling platform would be complete without addressing these "learning" capabilities. In k.LAB, model learning exploits the separation of the resource and semantic layer and the ability to find both inputs and outputs by resolving semantics. Models introduced by the keyword learn instead of model will resolve their outputs in addition to their inputs, producing a computable resource with a specified URN, independent of semantics, using a specialized function connected to the k.LAB runtime (a contextualizer, specified after the using keyword). The resource produced contains the trained computation, ready for future reuse by providing inputs through a model. For example, a minimal Bayesian suitability model to inform a land cover change model could use the following specification:

The function call following the keyword using invokes a learning process from the Weka software, in this case a Bayesian learner, integrated in k.LAB. When run in a spatially distributed context, the above model will resolve both the output (land cover type) and all predictors in the context of execution, sample them to produce a training dataset, and pass the latter to Weka to build and train a Bayesian model. The model obtained is, in turn, used to produce the luc.suitability local resource (using the Weka adapter) in the same project where the model is found. An interpolated map with the model’s prediction, along with a report including all metrics of fit, is also produced to facilitate the evaluation of results. The trained Bayesian network can be modified and retrained as needed using Weka as integrated with k.LAB. When satisfactory results are obtained, the trained model can be used for prediction through the URN of the trained resource:

The above model uses the trained Bayesian classifier to produce probabilistic predictions of land cover type. With probabilistic resources such as this, an uncertainty map can also be obtained by adding the uncertainty concept correspondent to the main output (using the uncertainty of semantic operator) as a secondary output if desired:

Similar considerations apply to other learning algorithms such as those found in the rest of the Weka platform or other machine learning platforms like Google’s TensorFlow. The resource containing the trained model will link its inputs by name and data type, and can be published to a node for remote execution by any users just like any other resource. Similar considerations apply to the prediction of qualities within countable entities (subjects, events, relationships) that are part of the context, training a classifier using each instance and its attributes as a training sample instead of sampling a distributed dataset like in the example above.

The problem of calibration or data assimilation of numerical models can be handled in the same fashion, by linking appropriate algorithms to k.LAB. At the time of this writing, an interface to the open source OpenDA package is being investigated for future integration.

Within a k.LAB session, a user or application sets a context and observes as many concepts as desired. Observations that were already made in the context automatically resolve any subsequent query for compatible concepts. At any time, the user or application can set or unset one or more scenarios to affect the selection of models. A scenario in k.LAB is simply a namespace whose contained models become "visible" to the system only when it is explicitly activated: when a scenario is active, its models take priority over any others to resolve their observables, potentially using other models to complete observations in case the scenario is only defined to cover a part of the context. Using scenarios, the environment within a context may be interactively defined to reflect specific hypotheses. In interactive use (for example with k.LAB Explorer) it is possible to build observation sets that use different scenarios, incrementally defining a context that reflects any desired conditions.

A context always contains the complete history of observations made within it, including the metadata and provenance records for all resources and models used. As dataflows are resolved and contextualized, provenance records stored along with the knowledge will be extended with all logical steps followed to compute the corresponding observations, remaining available to form a complete record of how the information within the context has come into existence. All this information is available to the user in interactive, graphical form when using a k.LAB client, and becomes part of the set of downloadable artifacts accessible within a context. These include:

The set of outputs obtained and visualized during a k.LAB session ensures the transparency and communicability of the results to a degree not yet seen in a modeling platform. In some situations, even the paths not taken by the resolver can be documented, which may be relevant when multiple resources with close rankings are available in resolution. The possibility of producing digitally signed artifacts (including all outputs, a report, dataflow and full provenance graph, plus — if needed — verifying and documenting the provenance and the peer review status of all resources and models involved) opens the door to the production and the verification of endorsed artifacts when the system is used to produce information from official institutional applications, or in situations when use of the result can have critical consequences in decision-making.

The k.LAB Engines and Nodes can be extended at the software level to provide new adapters (interfacing to new types of data and services), contextualizers (interfacing to contributed algorithms to use within models), or other functionalities such as authorities. A mechanism to produce components that can be used as plug-ins uses well-defined and documented points of extension in the Java class structure, and is supported by Maven archetypes for convenient project setup, building and deployment. The design of the server components is highly modular, and each existing resource adapter, external package integration (such as the Weka machine learning software library) or functionality extension is written as a component that can be deployed to nodes and services. The contextualization runtime, which executes the resolved dataflows and can load them from a stored k.DL specification, can itself be swapped with an alternative execution runtime if wished, for example to support different runtime platforms (e.g. to run contextualizations on distributed file systems). The k.LAB default runtime is parallelized and multi-threaded, capable of handling concurrent sessions owned by different users and optimizing the use of RAM to enable large-scale simulations.

Integration of k.LAB with existing models can proceed in two directions. By using the k.LAB API from within an existing model, the inputs of the model can be satisfied using semantic resolution, streamlining and simplifying data access from a largely unmodified model. By contrast, deep integration of a model into the k.LAB framework normally requires significant redesign, but can make the model and its components available to k.LAB users and other models as part of the k.LAB ecosystem, greatly enhancing its original value.

Within the k.LAB runtime, the software "agents" capable of receiving a behavior are not only the observations built within sessions — the sessions themselves and the users that own them are as well. This opens the door to the application of behaviors for purposes beyond the modeling of agents within simulations. In particularly, when a behavior is applied to a user session, the session can be seen as an application whose actions are initiated by users through client software, the consequences of which can trigger observations or other events as required by the application logic. Coupled with the ability of k.Actors to interact with the runtime and use semantics for queries, this feature enables fast and intuitive building of user applications in k.Actors. The k.LAB Explorer web client is equipped to respond to specialized action verbs by creating user interface components (e.g. buttons, text fields, lists); users interacting with these components will "fire" events that are sent back to the k.Actors runtime for processing. The resulting interactive application is typically very quick to build. For example, the following code

creates a demonstrational application, not explained in detail here, that will show buttons to make observations and collects the results in an array so that the corresponding data can be downloaded in one action. The UI will appear in k.LAB Explorer. Using modular UI components also defined in k.Actors, interfaces such as ARIES for SEEA can be build by minimally trained programmers in a short time (the code for the ARIES for SEEA application at the time of this writing is only about 300 lines long), making sophisticated modeling services immediately available to users and decision makers with very little effort.

In addition to these usages, k.Actors is used as a scripting language to automate repetitive tasks (for example to build global high-resolution map outputs describing a single observable, by computing it in multiple local contexts with fully customized model resolution in each) and to build test suites for all aspects of k.LAB.