cyberinfrastructure working group (CYWG) workshop notes

[alee]

Omnibus discussion for cyberinfrastructure tooling and support in service of the OMF mission.

The Cyberinfrastructure Working Group (CYWG) coordinates development and management of the OMF Science Gateway, an online platform for collaborative governance and standards. CYWG works to to identify, develop, and enhance tools and processes that lower the technical barriers towards adoption of OMF standards and facilitate their use by modeling scientists. CYWG staff assist the EOWG in developing online training modules and making them available on the OMF Science Gateway.

The OMF is a Science Gateway only insofar as it federates access to other science gateways: Keep It Simple

1. standards adoption and/or advancement for each standards category: Accessibility, Documentation, Interoperability, Reproducibility
2. coordinating efforts on interoperability (BMI/CMI), containerization, education and outreach, etc
3. the OMF website itself (see also One site or two #108)
4. serve as a trusted source / reference point that provides a "seal of approval" for other CI services and provide actionable / concrete guidance to improve in relevant areas
5. consider providing curation / member-only services as incentives for membership in addition to public goods that are developed for the broader community

Things to consider to reduce friction or otherwise incentivize adoption:

• template repositories with common project layouts for different modeling frameworks or platforms
• CLI tools to interactively build or augment these repositories
• github apps or actions that augment existing codebases with further-curatable pull requests
• consider MINT project as a metadata service layer to aid in Accessibility (discoverability) currently a tedious process to register the data (is this an area for us to contribute)

carpentries style lessons / workshops, github apps / actions that augment existing codebases through a transparent pull request style process

Other gateways:

• CUAHSI / hydroshare https://hydroshare.org
• TACC https://ptdatax.tacc.utexas.edu
• Ike Wai http://ikewai.org/
• https://mint.isi.edu/ & https://mintproject.readthedocs.io/en/latest/

[cmbarton]

And don't forget the OMF science gateway. This becomes a key tool for supporting the mission and collective engagement.

[cmbarton]

It's not web development as much as conceptualizing a platform to enable stakeholders to have a stake and encourage participation.

[alee]

Summary from Breakout Group 3:

Suzanne:

• consider MINT project as a metadata service layer to aid in Accessibility (discoverability) currently a tedious process to register the data (is this an area for OMF to contribute)
• the OMF is a Science Gateway only insofar as it federates access to other science gateways: Keep It Simple

John:

• serve as a trusted source that provides a "seal of approval" for other CI services and provide actionable / concrete guidance to improve in areas
• use Standards as ways of organizing CI

@cpritcha - Interoperability by definition needs a concerted effort across modeling groups and CI support for validation / certification / integration testing

prioritize tasks

[alee]

Summary from Breakout Group 4:

items to consider:

• building software tools that provide interoperability and integration checks / tests on whether a codebase meets a given interoperability claim (output data types)
• serve as an index to useful compute resources: HPC/HTC, domain-specific modeling best practices or other learning resources
• something similar to the list of interesting jupyter notebooks organized by domain

[alee]

Summary from Breakout Group 1

Chris, Albert, and Tony

• badging OMF certified CI
• identifying commonly used data structures for modeling in different domains and building software to facilitate that (something like tidyverse for models)
• Sensitivity Analysis Library in Python: https://salib.readthedocs.io/en/latest/
• Probabilistic Programming in Python: https://docs.pymc.io/
• model harmonization across model ensembles: beyond standard naming conventions, how to tune model parameterizations so that e.g., a downstream model knows to disable a module or component within it that is unneeded due to an upstream model.
• every model needs data.. how do we build linkages with the data community on data access
• coordination on broadcasting the modeling community's needs to the data community (subsets of data, DOIs)
• toolkit to evaluate uncertainty in model outputs / perform sampling + make proper sensitivity analysis more accessible
• containerization: composing Docker dependencies through multi-stage builds and submitted via automated PRs

Initial priorities: best modeling practices and initial standards, badging initial compliant CIs

Low hanging fruit to consider for additional OMF grants:

• containerization: composing Docker dependencies through multi-stage builds and submitted via automated PRs
• MINT metadata registration service: pick a few exemplar models to manually register with MINT and develop an automated workflow

[alee]

Summary from Breakout Group 2

Phil, Olaf, and Alan

• incentives and benefits of OMF membership: distinguish between public goods and whiteglove cyberinfrastructure / curation services / devops pipelines for continuous integration, automated testing, etc. to contributing members
• ways to lower the burden of building adequate documentation: e.g., something like https://dmptool.org/ or templates for ODD or other documentation protocols
• work to minimize the mundane tasks to adopt the standards
• working with existing source code annotation tools like Doxygen / Javadocs / readthedocs
• identifying applicable / appropriate interoperable model ensembles for given tasks
• look into public private partnerships, cyberinfrastructure or modeling services across academia and industry
• tools to conveniently generate open / interoperable data formats

[alee]

Summary from Breakout Group 4

Isaac: be content curators disseminated via the OMF web presence: YouTube of models, statquest

Benoit: Look into the JOSS review process https://joss.readthedocs.io/en/latest/reviewer_guidelines.html, they have an editorial bot as well https://joss.readthedocs.io/en/latest/whedon.html

Focus on the accessible cookiecutter template repositories. Teach people how to do the right thing at the start.

Provide exemplar models with readable self documented code

[alee]

Summary from Breakout Group 1

George: test harnesses for compliance to open certifications are much easier than managing open source compliance - closed source can be OGC-compliant if it passes the test harness

Come up with a test spec for the presence of metadata elements

Look to Apache Software Foundation, Eclipse, Linux Foundation, for inspiration

Delia: Working Group should also have some capacity devoted to ensuring that the site is secure: cybersecurity concerns

the education component is critical to instill the discipline

Need to identify how the working group will be funded, staffed, managed, maintained.

[alee]

Summary from Breakout Group 2

1. Separate the concept from the technical implementation, and clearly articulate the concepts and principles first, then build the cyberinfrastructure from those concepts. A requirements specification for the template repository - what can we actually do, then build reference implementations of those requirements specs.

Software engineering good practices: how should the model be structured, loose coupling, no need for technical implementation of the concepts

2. Cyberinfrastructure for ontologies / vocabularies for phenomenons, names, spatio-temporal structures, categories of models and domains in which model belong and web services for these vocabularies + definitions + search

General Systems Theory should be capable of providing an ontology that serves any and all domains, an upper ontology that can be applied to all other domains. modeling is the core of systems science research

[alee]

Summary from Breakout Group 3

support for TRACE notebooks? A list of desiderata that a programming framework should provide to facilitate these things. For example, an example TRACE document, hooks that facilitate inspection, ability to test

Rate modeling frameworks on a variety of attributes (testability, inspectability / debugging, etc).

Does OMF provide research software engineering services? Perhaps instead serve as a broker - providing consulting services presents its own unique set of challenges.

Increasing the reliability of ecological models using modern software engineering techniques