Tethys 3.1

Tethys 3.1 introduces a new R Programming language client and the ability to interoperate with PAMGuard version 2.02.13 or later.  Both of these functionalities are new and in a beta state, so please report any issues to the Tethys and PAMGuard teams.  The Tethys module for PAMGuard is not enabled by default, but can be turned on with a command line switch at PAMGuard launch.  Please contact the PAMGuard/Tethys teams if you wish to do this.

We have also had requests to store bespoke data files along with some types of documents.  Tethys 3.1 adds a BespokeData element to detection and localization documents.  The BespokeData expects a mandatory Abstract element that describes the custom data.  This is followed by zero or more Data elements.  Each Data element has a mandatory URI which may either be a real URI or a filename and an optional Comment field.  If it is a filename, corresponding files need to be added to the attachment field in the web-base data import tool.  Following the optional Data elements, a UserDefined field permits arbitrary XML to be included.  The intended purpose of this is to provide storage for additional products.  For example, most users will choose to record encounter-level records for echolocation clicks as it would be very easy to store enough clicks to quickly overwhelm any client foolish enough to try to request all clicks.  The clicks associated with a single set of detection effort could still be retained by submitting bespoke data.

The Matlab Client has a couple of changes which may impact users of the “new style” tables that were introduced as an optional output in Tethys 3.  Some of the Matlab tables in these outputs had string entries that were wrapped in side a cell entry.  These have been removed.  Note that tables have not yet been integrated into the queries for localizations, this will be done in a future release.  Selecting the format for species input and output still works as before (dbSpeciesFmt), but this will be deprecated and we prefer that you use the new functions that have been integrated into the query object (see Matlab Client documentation or below for details).

The remainder of this release is to support a new simple query language developed to support the R client, several under the hood changes such as the introduction of an XLST processor, corrects errors discovered in the Tethys 3.0 release, and implements changes to schemata to reflect the evolving proposed ASA/ANSI standards.  Note that while the version of Python (3.9.5) has not changed with this release, some of the bundled packages have.  With the exception of changes to representing the localizations, schemata changes are minor and will not impact most users.   If you have any issues with the new schemata, please contact us, we’re here to help.

The following subsections provide highlights of some of the changes with various Tethys components.

Schema changes:

Calibrations

  • FrequenceyResponse is now optional.
  • Changes to optional QualityAssurance
    • Comment is now optional
    • AlternateCalibration, used to specify a “stand-in” calibration when one is not available, no longer requires a timestamp although one is highly recommended.

Deployments:

  • SamplingData/Channel: Added new element EventTrigger to account for instruments that only record events or data when certain conditions are met (e.g., detection of a specific type of call).  EvenTrigger has the following elements:
    • Description – Standard description: Objectives, Abstract, and Method describing triggering conditions
    • Algorithm – Standard Algorithm descriptor used in other places (e.g. Detections, Localizations).
  • Data/Tracks changes:
    • Added optional element SpeedUnit. Required when track data records instrument speed and must be in units: kn, m/s, km/h.
    • Track/Trackline renamed Track/TrackId,
    • Track/Point changed:
      • Bearing_DegN à Heading_DegN – Renamed and clarified that this heading indicates the direction that the instrument is pointed, not necessarily the direction of travel. Measurement is in degrees counterclockwise from true or magnetic north, as specified in attribute North.
      • CourseOverGround_DegN – Counterclockwise angle relative to magnetic or true North (specify as attribute North) showing direction of movement relative to the ground.
      • Speed_kn à Speed to accommodate non marine deployments. Units is specified in the SpeedUnit specified above.  Clarified that Speed is the speed through the medium in which the instrument is traveling.
      • SpeedOverGround – Specifies the speed at which the instrument is traveling relative to the ground.  Unlike Speed, this will take into account air or water currents which may be providing additional movement relative to the ground.
    • Tracks
  • Data/TrackEffort
    • Added Comment field to OnPath/OffPath specification of effort.

Localizations:

  • Major changes to how localizations are represented to conform with changes in the proposed standard by the ANSI Standards committee that conforms to a subset of ISO 19111:2019. See the draft standards on the Tethys web site for a description of how localizations are represented.
  • It is usually desirable to downsample their instrument telemetry stored in deployments to avoid keeping too many observations. This however can create problems during localizations where we  may wish to store finer grained measurements.  Each localization now supports a InstrumentTelemetry field which supports recording one observation of telemetry at the time of the localization.  InstrumentTelemetry supports the same types of fields as those that appear in the Data/Tracks/Track/Point fields of deployments.
  • Bespoke data field allows specification of one or more non-standard auxiliary data sources. Bespoke data consists of a textual Abstract that describes the data, zero or more Data elements that contain names of files that the Tethys server has stored, and a UserDefined field that permits arbitrary elements to be stored.  BespokeData is optional and may only be specified once per set of localizations.

Detections:

  • Bespoke data field allows specification of one or more non-standard auxiliary data sources. Bespoke data consists of a textual Abstract that describes the data, zero or more Data elements that contain names of files that the Tethys server has stored, and a UserDefined field that permits arbitrary elements to be stored.  BespokeData is optional and may only be specified once per set of detections.

Server

  • Data import
    • Under the hood changes to the import system
    • Added additional document validation rules and corrected one erroneous one of checking if End was before Start.
    • Excel source documents no longer stored in compressed format as it makes it more difficult for people examining source material
    • Ability to store Bespoke documents
    • Fixed issue in retrieval of attached data.
    • Fixed issue with integer fields being converted to floating point
    • Import web tool would sometimes hang. We have identified one issue related to modifying a file after it is already attached and have added a warning when this happens.  We are not aware of other issues, but please alert the Tethys team should you see any isues.
  • Caching – The server uses query caching to improve performance. Prior to this release, we did not retranslate taxonomic information if it had changed between subsequent identical queries.  That is, if we ran the same query twice, once requesting species names in English and the second time in Latin, both returned documents would use English vernacular names.   Mapping of taxonomic names is now done on the retrieved cache value for each query.
  • XQuery REST interface: These are mostly under the hood changes and will not be of interest to anyone but power users who wish to customize how their queries are returned.  Some of these features may be incorporated into the client libraries in a transparent manner in future releases.
    • XML transformation using XSLT – A new facility has been added to queries that is designed to help power users rewrite XML to extract portions of the XML tree or rewrite it entirely. This uses the extensible stylesheet language (XSLT).  Users can provide XSLT stylesheets (up to version 3.0) to provide transformations of XML.  Users interested in learning XSLT may wish to read an XSLT introduction or read one of the many books on XSLT, such as Doug Tidwell’s XSLT: Mastering XML Transformations 2nd (2008, O’Reilly, Sebastopol, CA) or Michael Kay’s XSLT 2.0 and XPath 2.0 4th ed. (2008, Wrox Press, Hoboken, NJ).
    • POST now respects the responseType header and will return results in one of the following formats:
      • application/json – Returns Javascript object notation (JSON) using the BadgerFish convention. Attributes are prefixed with an @, and element content is associated with a $ key.  We use the Python library to accomplish this.  While we do not currently support other variants than Badgerfish, it would be very easy to add any variant supported by xmljson.
      • application/matlab

WebClient

  • IMPORTANT NOTE: Some web browsers (e.g., chrome) are starting to block downloads from servers that are serving plain-text (unencrypted) data unless you grant permission for the download.  If you are using Tethys in secure socket layer mode (started with the -s flag), this will not affect you.  If you use plain text mode, look for an option to retain the download or change the security settings for the Tethys web site in your browser.  If you are thinking about using secure socket layer on the server, read the manual, as using secure socket layer without a registered certificate can cause issues in using Tethys as some clients block certificates that are not trusted.
  • Added view of ensembles to list of collections
  • Import page uses improved drag and drop interface.
  • Added JSON export – XML to JSON uses the Badgerfish convention, see the xmljson project for a description. Adding other translation conventions supported by xmljson would not be difficult if desired.
  • Updated Google maps interface to address Google API changes.

MatlabClient

  • dbGetDetections – Fixed issue that was corrupting indexing columns that allowed cross data-table references in new style tabular output.
  • Multiple functions – Updated BinSize attribute to reflect Tethys 3.0 name: BinSize_min
  • dbSpeciesFmt has been deprecated. It will still work, but please use the following methods associated with the query object (e.g., q = dbInit(…))

Setting functions  – These all take 1 or 2 arguments.  The first argument specifies the type and must be one of {Latin, TSN, Abbrev, Vernacular}. When the type is Abbrev or Vernacular, a second argument must be provided with the name of the abbreviation map or the vernacular language to be used.  The database does not have entries for all languages, and replaces outputs with the Latin name when a vernacular entry does not exist.  We do not recommend using vernacular for specifying species in queries.  At present, the system does not check the second argument for valid entires.

  • setSpeciesIdInput(…)
  • setSpeciesIdOutput(…)

Reporting functions

  • getSpeciesIdInputOutput()
  • getSpeciesIdOutput()
  • getSpeciesIdInput()

JavaClient

  • Client instances now retain state about preferred representation of taxa for both querying and reporting. Additional methods were added (MatlabClient has been updated to use this new functionality).
  • Beta interface for simpler queries. Existing methods either required XQuery or that languages using the JavaClient provide a Javascript object notation (JSON) representation of the criteria to be matched and what is returned.  This required every client to write functions that produced XQuery or JSON queries, such as the dbGetDetections function in the MatlabClient.  New functionality makes it easy to query using a simple declarative string.  The query object supports the following functions that are documented in the JavaClient manual:
    • getDetections
    • getDetectionEffort
    • getDeployments
    • getLocalizations
    • getLocalizationEffort
    • getCalibration

These are simplifications of the more general QuerySimple method.  Note that this functionality is still in beta test and returns XML as opposed to a client language specific data structure.  Most languages have libraries for parsing XML and future versions of Tethys are likely to further develop this interface.

RClient

  • Basic functionality for R through the new Tethys class. See documentation for how to use.  Note that this is still fairly immature and does not have the same level of convenience that is present in the MatlabClient where results are automatically translated to native structures.

DataExplorer

  • Added support for granularity grouped. Grouped represents a set of individual detections of any granularity that have been grouped together.  Examples include situations such as a click train, song, or other groupings (e.g. detections of the same animals picked up on multiple instruments).  Grouped detections may reference the individual detections regardless of their granularity that are part of the group.