Thursday Developer Meeting notes - ADR

Owned by Aron Roberts (Unlicensed)
Last updated: Apr 10, 2012Version comment
8 min read

Rough notes from Developer Meeting sessions on Thursday, 2012-02-09

The following notes for this session are sparse - randomly selective

Debugging, troubleshooting

Debugging in the UI layer

Poster - Firefox plug-in for POSTing
https://addons.mozilla.org/en-US/firefox/addon/poster/

MIME types, permissions
paste/type in, or refer to a file
Note: default will be text/xml, will need to change that to application/xml

Chrome has an equivalent; its UI is not great

In Firebug:

  • To see the JSON request payload being sent by the UI layer to the app layer, set a breakpoint in DataContext.js, near the end of the BuildOpts function (e.g. var buildOpts = function (...)), in the if (data) ... statement.
  • To see the JSON response payload being returned by the app layer to the UI layer, set a breakpoint in DataContext.js, in the BuildOpts function, in the if (responseData && ...) statement.

Debugging in the App layer

Testing if a new field is working in the app layer
Set up new test in cspi-services
maybe not for extension

Inspect uispec, uischema coming out of the app layer
You should at least see the name of the field in there somewhere

(See App layer calls below for more details.)

Logging for troubleshooting

Documenting performance logging
for app layer, services layer
app layer 'perflog' file currently written to /tmp/, location could be changed

services layer logging currently set to TRACE

logging - developer profile v. runtime profile

UI framework allows for logging, JavaScript initialization
Firebug is sufficient generally
JavaScript errors and warnings, Net tab in Firebug to see requests and responses
Only in really complex situations does Yura use logging

Adding a new procedure

Adding a new procedure in the services layer

Choose an existing procedure, and clone it (e.g. with 'svn export' or git equivalent) to a new folder, alongside other procedures' folders in the services tree

Literally do these case-sensitive searches and replaces everywhere (e.g. if cloning 'Movement' procedure to 'Foo' procedure):
Movements -> Foos
movements -> foos
Movement -> Foo
movement -> foo
MOVEMENT -> FOO

The core classes you're going to care about include:
__Resource.java
in the 'service' folder of the module
and a Nuxeo-specific __DocHandler.java file

for authority services
has sub-resources, authority plus the authority item
have twin document handlers, one for the authority and one for the authority item
tend to have only logic around displayname computation, which might go away later on anyway

could create Maven archetypes
one for procedure, other for authority

almost everything follows a pattern

once you're done modifying your service, and it at least builds

  • There is an existing document on the wiki on how to hook a new procedure into the services layer

Adding a new procedure in the application layer

Add a new procedure in the app layer by making the following changes:

  • in tomcat-main/src/main/resources/defaults:
    • add a new base-...xml file
    • add a new domain-...xml file
    • edit base-relations.xml to add new record type as a relate-able record type
      • (Per Chris M, this base-relations.xml file also has an 'all' keyword for all relation - or record? - types, but we don't appear to be using that currently)
    • edit the id / number pattern config file, base-other-idgenerator.xml, to add any automatically generated number patterns needed (also need to make the generator(s) in the services layer; there's a document on the DOC wiki about that)
  • in tomcat-main/src/main/resources:
    • add an xml include line to point to your new procedure's base-* file, to *-tenant.xml (e.g. core-tenant.xml and lifesci-tenant.xml) (in app layer tree) and to our-tenant-tenant.xml (in the mini-build)
    • add an xml include line to point to your new procedure's base-* file, to default.xml (this file appears to still be used in populating the lists of record types, at least in app layer tests)

Inspect uispec, uischema coming out of the app layer

(See App layer calls below for more details.)

Create App layer tests, including test datafiles

Need to add details about adding App layer tests.

Adding a new procedure in the UI layer

Add a new procedure in the UI layer by making the following changes:

  • Add two (2) new HTML template files, cloning and editing from similar files
    • main record page (in defaults/html/pages)
    • advanced search page (in defaults/html/components)
  • Add two (2) new JSON config files in the UI's defaults/config folder, cloning and editing from similar files
    • main record page (...json)
    • secondary tab page (...-tab.json) (this config file is needed for a new procedure, but not for a new authority record type)
  • Add text labels relevant to the new procedure to the default message bundle file, defaults/bundle/core-messages.properties

In addition, for authorities, need to edit js/Demands.js and clone and edit the relevant block, from an existing entry for another authority
per Yura, at some point you won't need to do that, app layer might be able to generate this for the UI

When working on the additions above, fetch the UISpec by making a call to the app layer, after doing app layer config, in order to make editing the HTML templates and message bundle easier. This will show you:

  • Selector names needed in the HTML templates
  • Message label keys needed for bundle

Finally, fetch the UISpec and UISchema by making calls to the app layer, and place those into src/test/uispecs and src/test/uischema. Those static JSON files are used for local testing of the UI layer.

Message bundle overlay - could this be per-tenant for contributions?
App layer might be able to read all files in bundle folder
App layer merges all overlay files just once

  • App layer init doesn't rebuild that - it could be added

Boolean fields

In UI, would add checkbox
Test cases added to ensure that's working
test/js/RendererTest.js

includes required markup for a checkbox
uischema for that field has to have a default value, be a boolean

set datatype to boolean in app layer config, default value is false

Chris identified she needs to make a change to the XML/JSON conversion class to preserve the type as Boolean

Documentation

John K:
Removing lots of old cruft from the DOC wiki in the customizing area
some docs partly useful, but ...

Chris P:
Overview of how to do an extension

  • Creating a tenant
  • Adding extensions in each layer

Documenting how to do server-side debugging

Doc sprint just for cleaning up

Jesse's notes about how he did this, not why
Could be a good roadmap
On deploy wiki, notes on MMI implementation
Acquisition - adding a few fields

App layer reference documentation
Unified collectionspace config

Ray:
A lot of documentation in JIRAs, that doesn't make it to wiki documentation
Also postings on the Talk list

Nate:

  • Document how to enable caching

App layer calls, tips, suggestions

Chris M. demoing UI calls with requests and results for:
loginstatus
(based on accountperms/0 call to the services - who you are currently logged in as)
composite

WebUI.java
Lists most of the calls / URL paths that the app layer accepts

A few additional calls are in
TenantServlet

  • init
  • composite

In TestUIRecords.java
testMisc()
will create 10 records of every object, procedure type
default /generator, may create 20 objects of each

if logged in as a user in a tenant,
may potentially create records including extra extension parts

in the app layer
some calls created as a result of Kasper's requests for QAing
most others used by the UI

'basic' call to return full record without any other data, such as right sidebar data

The app layer's canonical documentation is in the
Unified Config documents on the project wiki

has the default patterns for generation of selectors

In app layer, could add Ant config-deploy to push deploy

Long discussion of <services-tenant-singular> in Ray's base-procedure-claim.xml app layer config file for Claims. Possible need to wildcard suffixes in relation searches in the services, to pick up extension DocTypes that are truly extensions of standard DocTypes but whose names might include a tenant identifier as a suffix.

Return last template used in Core part
Could be used to template - for Concept authority UI tweaks, perhaps even for extending services like Group for purposes like research visits

UI layer tips, suggestions

Suggestion for Yura:
JavaScript file in the UI to compare template v. uispec, identify any mismatches

In UI payloads sent to the app layer
Sends all fields, except for descendants in empty nested structures

UI layer
change HTML template
message bundle overlay file
look in UISpec for generated full name

switch to dropdown requires change to select structure
otherwise, most fields all use the same HTML structure in templates; they just get decorated differently

there is some complexity around HTML positioning
if you're adding a bunch of fields, or moving around fields

Fake static uispecs in the test directory
Local JSON files
test/data directory

Open a file in test/html, get local path to that file, open in Firefox
have to turn off security restriction
/src/main/webapp/defaults/html/record.html?recordtype=objectexit&csid=....

http://issues.collectionspace.org/browse/CSPACE-3763

  • Need to pull more into the mini-build - test tree - to simulate local editing
  • Chris discussing with Yura the possibility of creating generic files in the UI layer from app layer config, if they're missing
    done at init time
    pull out core bit, cspi-schema stuff, to make a standalone utility

that's why we (app) have the OneJAR during the build process, then wrap the WAR around it

Pain points

writing the tests in the services
now easier with richard's changes

ray
also doing things by rote worked
but didn't know why we did things this way
e.g. why we have two services XSD schemas, one for Nuxeo and the other for JAXB, and whether they need to be identical or different

Chris, discussing with Yura the need to do:
Show what instance of the authority you're saving to, on the authority record editor page

Services: could add validation logic in the relations service, to constrain hierarchical authorities to a namespace
(need details)

Community contributions

procedures on a Walker list
rights management
condition checking
valuation control

Nate asked:
When should something be added to an existing schema, or instead added as a new procedure?

Michael:
Appraisers who come up with valuations for a group of objects?

Most procedures are essentially around groups. If the functional people are happier with doing it in extension schema, it's a lot easier. Tradeoff between functionality and time and cost to build and contribute.

First step is to talk about it on the Talk list

Michael:
Stratigraphic information, useful to a thousand museums, but not yet to any others in the CollectionSpace community

Chris P asked about acceptance criteria for new procedures

Angela's primary criteria is, "Is it mentioned in SPECTRUM?"
Beyond that, a base principle is we want to own as little code as possible, due to small size of core development team.
Can do work in the services to make it easier to plug-in new services, most of the complexity - bottleneck - around the JAXB stuff. Beyond that, mostly just need to define new document type in the Nuxeo document bundle.

Chris at Walker:
what's the probability that someone wouldn't have a consultant, but still run a system locally?

Distribute whole VMs?

EC2 scripts, etc. Install up to 10 tenants per VM, then roll over to new hosting instance?

Want some other tenants' vocabularies, authorities ...

People with existing data with complicated migrations probably won't want to borrow vocabularies.

UCB Herbaria see the value of their plant names for other herbaria.
Matching exercise, simple plant names against a vocabulary.

For smaller museums, may want the botanical taxonomy, but not want to maintain it.

Some people just use AAT unmodified, supplementing with their local name authority.
If you can deal with AAT on a read-only basis, could share between tenants, rights-permitting.

Managing rights across tenants for authorities is an interesting challenge.

Could have a shared tenancy for certain things, unsecured access to certain resources.

Possible differentiation of domain and local fields, to help others make more ready use of contributed templates.

Selectors, CSS classes ...

Default selectors could be modified, by app layer code, to add 'domain' or 'local' ...
However, changing default pattern for selector names could cause significant disruption for current implementers.

Javadoc-like (xDoclet-like?) comments in the app layer

Some museums may just want to take changes wholesale.

Biggest cost savings with non-customization.

Deployment tools

Deployment tools, possibly with some use for deploying contributed bundles as well:

Fabric, Python tool
Chris and Nate
for doing deployments http://docs.fabfile.org/

Per Nate, there are similar tools in other languages

At CARET, Chris M's colleagues use rollout tools, disaster control, when a server goes down or is hacked
Get up in minutes on a new VM slice
Puppet http://puppetlabs.com/
can talk to Tony at CARET about this

Field level permissions

Associating roles with templates might be a starting point workaround for approximating field-level permissions

E.g. assigning data entry to a student or volunteer with a limited number of fields available in the template

(For more on this: Friday Developer Meeting notes - ADR)