Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fits #129

Merged
merged 7 commits into from
Jan 8, 2020
Merged

Fits #129

merged 7 commits into from
Jan 8, 2020

Conversation

dannylamb
Copy link
Member

GitHub Issue: (link)

What does this Pull Request do?

This PR installs the work for FITS media that @ajstanley has developed for UPEI's RDM project. Now we can test it 🎉 🚀

What's new?

Points to a dev version of the fits role and adds that role back into the tomcat play so it gets brought in when provisioning.

How should this be tested?

Make a node, add an original file to the node. A FITS media derivative should get made, and you can see all the metadata that gets extracted from it.

Interested parties

@Islandora-Devops/committers @ajstanley @elizoller

@dannylamb
Copy link
Member Author

For some reason, some of the config is getting changed or not imported correctly (I'm not sure how) despite the features being enabled in the right order. I'll sort that out soon, but wanted to get this up now. It's got nothing to do with all the nice work @ajstanley is contributing here.

elizoller
elizoller reviewed Aug 7, 2019
View reviewed changes
post-install.yml
command: "{{ drush_path }} --root {{ drupal_core_path }} -y en islandora_defaults"

- name: Import feature
command: "{{ drush_path }} --root {{ drupal_core_path }} -y fim --bundle=islandora islandora_defaults"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you not enabling islandora_defaults and importing it anymore? Did it move somewhere else? When I ran a fresh build, I lost a lot of the settings like having media use on the media types.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is being enabled (as a module) @dannylamb added it to the drupal_enabled_modules ansible variable, but we probably still need this line to import the feature because it is not imported by default.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, I manually ran drush en and it was already enabled, then I ran the drush fim and it fixed the missing things 👍 (and then a FITS media object was made! )

@dannylamb
Copy link
Member Author

dannylamb commented Aug 7, 2019 via email

@Natkeeran
Copy link
Contributor

Natkeeran commented Aug 26, 2019 •
edited
Loading

  • installs without ansible issues
  • media type is created
  • when an image (tiff and jpg) with Media Use Original was created, it creates the Fits derivative. However, it fails to create the TN and Service files. Not sure if this is a regression issue or unrelated issue.
[2019-08-26 12:26:23] app.INFO: Imagemagick Command: {"cmd":"convert - -thumbnail 100x100 jpeg:-"} []
[2019-08-26 12:26:23] crayfish.cmd_execute_service.ERROR: Process exited with non-zero code. {"exit_code":1,"stderr":"convert-im6.q16: no decode delegate for this image format `' @ error/constitut$
[2019-08-26 12:26:23] app.ERROR: RuntimeException: {"exception":"[object] (RuntimeException(code: 500): convert-im6.q16: no decode delegate for this image format `' @ error/constitute.c/ReadImage/$
[2019-08-26 12:26:23] app.DEBUG: < 500 [] []
[2019-08-26 12:26:24] app.INFO: Matched route "{route}". {"route":"GET_convert","route_parameters":{"_controller":"houdini.controller:convert","_route":"GET_convert"},"request_uri":"http://165.22.$
[2019-08-26 12:26:24] app.DEBUG: Checking for guard authentication credentials. {"firewall_key":"default","authenticators":1} []
[2019-08-26 12:26:24] app.DEBUG: Checking support on guard authenticator. {"firewall_key":"default","authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"} []
[2019-08-26 12:26:24] app.DEBUG: Calling getCredentials() on guard authenticator. {"firewall_key":"default","authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"} []
[2019-08-26 12:26:24] crayfish.syn.jwt_authentication.DEBUG: Token: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpYXQiOjE1NjY4NDAzNzQsImV4cCI6MTU2Njg0NzU3NCwid2ViaWQiOiIxIiwiaXNzIjoiaHR0cHM6XC9cLzE2NS4$
[2019-08-26 12:26:24] app.DEBUG: Passing guard token information to the GuardAuthenticationProvider {"firewall_key":"default","authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"}$
[2019-08-26 12:26:24] app.INFO: Guard authentication successful! {"token":"[object] (Symfony\\Component\\Security\\Guard\\Token\\PostAuthenticationGuardToken: PostAuthenticationGuardToken(user=\"a$
[2019-08-26 12:26:24] app.DEBUG: Guard authenticator set no success response: request continues. {"authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"} []
[2019-08-26 12:26:24] app.DEBUG: Remember me skipped: it is not configured for the firewall. {"authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"} []
  • same issue for video
[2019-08-26 12:36:01] app.DEBUG: Ffmpeg Command: {"cmd":"ffmpeg -i http://165.22.239.57:8000/_flysystem/fedora/2019-08/test.mp4   -vcodec libx264 -preset medium -acodec aac -strict -2 -ab 128k -ac$
[2019-08-26 12:36:03] crayfish.cmd_execute_service.ERROR: Process exited with non-zero code. {"exit_code":1,"stderr":"ffmpeg version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2000-2019 the FFmpeg devel$
[2019-08-26 12:36:03] app.ERROR: RuntimeException: {"exception":"[object] (RuntimeException(code: 500): ffmpeg version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2000-2019 the FFmpeg developers\n  built$
[2019-08-26 12:36:03] app.DEBUG: < 500 [] []
[2019-08-26 12:36:04] app.INFO: Matched route "{route}". {"route":"GET_convert","route_parameters":{"_controller":"homarus.controller:convert","_route":"GET_convert"},"request_uri":"http://165.22.$
[2019-08-26 12:36:04] app.DEBUG: Checking for guard authentication credentials. {"firewall_key":"default","authenticators":1} []
[2019-08-26 12:36:04] app.DEBUG: Checking support on guard authenticator. {"firewall_key":"default","authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"} []
[2019-08-26 12:36:04] app.DEBUG: Calling getCredentials() on guard authenticator. {"firewall_key":"default","authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"} []
[2019-08-26 12:36:04] crayfish.syn.jwt_authentication.DEBUG: Token: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpYXQiOjE1NjY4NDA5NjEsImV4cCI6MTU2Njg0ODE2MSwid2ViaWQiOiIxIiwiaXNzIjoiaHR0cHM6XC9cLzE2NS4$
[2019-08-26 12:36:04] app.DEBUG: Passing guard token information to the GuardAuthenticationProvider {"firewall_key":"default","authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"}$
[2019-08-26 12:36:04] app.INFO: Guard authentication successful! {"token":"[object] (Symfony\\Component\\Security\\Guard\\Token\\PostAuthenticationGuardToken: PostAuthenticationGuardToken(user=\"a$
[2019-08-26 12:36:04] app.DEBUG: Guard authenticator set no success response: request continues. {"authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"} []
[2019-08-26 12:36:04] app.DEBUG: Remember me skipped: it is not configured for the firewall. {"authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"} []

Questions

  • Are FITS fields pseudo fields? (For example, cannot edit them)
  • Only four FITS fields seem to be available as fields (attachment). If so, how is this being selected? (xslt?)
    fits
  • How would multilingual support work for pseudo fields (in case people want to make FITS information available for search or faceting)?

@ajstanley
Copy link
Contributor

@Natkeeran - I haven't seen this. What happens if you turn the context to create FITS off before ingesting a video? CrayFits has no resources shared with the Crayfish cluster, so there shouldn't be a conflict.

@Natkeeran
Copy link
Contributor

@ajstanley
Disabling FITS context does not help with the derivative issue. It may not be a related issue. However, I am not sure exactly what the issue from the houdini log.

[2019-08-26 12:48:27] app.INFO: Matched route "{route}". {"route":"GET_convert","route_parameters":{"_controller":"houdini.controller:convert","_route":"GET_convert"},"request_uri":"http://165.22.239.57:8000/houdini/convert","method":"GET"} []
[2019-08-26 12:48:27] app.DEBUG: Checking for guard authentication credentials. {"firewall_key":"default","authenticators":1} []
[2019-08-26 12:48:27] app.DEBUG: Checking support on guard authenticator. {"firewall_key":"default","authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"} []
[2019-08-26 12:48:27] app.DEBUG: Calling getCredentials() on guard authenticator. {"firewall_key":"default","authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"} []
[2019-08-26 12:48:27] crayfish.syn.jwt_authentication.DEBUG: Token: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpYXQiOjE1NjY4NDE2OTQsImV4cCI6MTU2Njg0ODg5NCwid2ViaWQiOiIxIiwiaXNzIjoiaHR0cHM6XC9cLzE2NS4yMi4yMzkuNTc6ODAwMCIsInN1YiI6ImFkbWluIiwicm9sZXMiOlsiYXV0aGVudGljYXRlZCIsImFkbWluaXN0cmF0b3IiLCJmZWRvcmFhZG1pbiJdfQ.Yz7FTL5SyRUgjfyLmL8RVDDycuhlhiEu1g9Juw0M0SVR3rhMwK-QI7ue6OEXaSrxIXfQ_S9XOffz20YF-6z5voSOkEk9n1JFRj6LFgvVLgNKyRDDXpQZFUhYB_V7b9_JR4jBrSL-0aS6anBkhw9WqwGtrZ_4nIp1r53XXj48FBKeoCgvWX4j0EqaVP5iAA1JfgskwI8c5xhXZC0Xp70zeHNpoayV2JFkAXBFAHYHafE_E44HDvqPyUUcpZlmECHIV9V4bdy_Kgca3pdX4T4qxQmzwBZpSm0WW7utp30bkmFX5tuFqMQFCXLEZ5mnbVhffFfjysYqWEcSYFUHxxrmpw [] []
[2019-08-26 12:48:27] app.DEBUG: Passing guard token information to the GuardAuthenticationProvider {"firewall_key":"default","authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"} []
[2019-08-26 12:48:27] app.INFO: Guard authentication successful! {"token":"[object] (Symfony\\Component\\Security\\Guard\\Token\\PostAuthenticationGuardToken: PostAuthenticationGuardToken(user=\"admin\", authenticated=true, roles=\"authenticated, administrator, fedoraadmin\"))","authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"} []
[2019-08-26 12:48:27] app.DEBUG: Guard authenticator set no success response: request continues. {"authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"} []
[2019-08-26 12:48:27] app.DEBUG: Remember me skipped: it is not configured for the firewall. {"authenticator":"Islandora\\Crayfish\\Commons\\Syn\\JwtAuthenticator"} []
[2019-08-26 12:48:27] app.DEBUG: > GET /houdini/convert [] []
[2019-08-26 12:48:27] app.INFO: Convert request. [] []
[2019-08-26 12:48:27] app.DEBUG: X-Islandora-Args: {"args":"-thumbnail 100x100"} []
[2019-08-26 12:48:27] app.DEBUG: Content Types: [] []
[2019-08-26 12:48:27] app.DEBUG: Content Type Chosen: {"type":"image/jpeg"} []
[2019-08-26 12:48:27] app.INFO: Imagemagick Command: {"cmd":"convert - -thumbnail 100x100 jpeg:-"} []
[2019-08-26 12:48:27] crayfish.cmd_execute_service.ERROR: Process exited with non-zero code. {"exit_code":1,"stderr":"convert-im6.q16: no decode delegate for this image format `' @ error/constitute.c/ReadImage/504.\nconvert-im6.q16: no images defined `jpeg:-' @ error/convert.c/ConvertImageCommand/3258.\n"} []
[2019-08-26 12:48:27] app.ERROR: RuntimeException: {"exception":"[object] (RuntimeException(code: 500): convert-im6.q16: no decode delegate for this image format `' @ error/constitute.c/ReadImage/504.\nconvert-im6.q16: no images defined `jpeg:-' @ error/convert.c/ConvertImageCommand/3258.\n at /var/www/html/Crayfish/Houdini/vendor/islandora/crayfish-commons/src/CmdExecuteService.php:120)"} []
[2019-08-26 12:48:27] app.DEBUG: < 500 [] []

@ajstanley
Copy link
Contributor

The no delegate message seems like its an imagemagick thing - what happens when you run
identify -list format |grep JPG?

@Natkeeran
Copy link
Contributor 

root@clawtest:/var/log/islandora# identify -list format |grep JPG
      JPG* JPEG      rw-   Joint Photographic Experts Group JFIF format (80)

@ajstanley
Copy link
Contributor

weird. at least imagemagick is working. Anything interesting in the karaf logs?

@Natkeeran
Copy link
Contributor

Will do some more testing tomorrow and report back here.

@Natkeeran
Copy link
Contributor

Natkeeran commented Aug 27, 2019 •
edited
Loading

@ajstanley
It does not look like the issue is at the Karaf end. I've attached the log files. Things are failing in milliner as well!

gemini.log
houdini.log
karaf.log
milliner.log

@ajstanley
Copy link
Contributor

ajstanley commented Aug 28, 2019 via email

@dannylamb
Copy link
Member Author

What does the Houdini log say?

@dannylamb
Copy link
Member Author

...and read the upscroll Danny...

Thanks for posting the logs @Natkeeran

@dannylamb
Copy link
Member Author 

[2019-08-27 13:26:54] app.ERROR: RuntimeException: {"exception":"[object] (RuntimeException(code: 500): convert-im6.q16: no decode delegate for this image format `' @ error/constitute.c/ReadImage/504.\nconvert-im6.q16: no images defined `jpeg:-' @ error/convert.c/ConvertImageCommand/3258.\n at /var/www/html/Crayfish/Houdini/vendor/islandora/crayfish-commons/src/CmdExecuteService.php:120)"} []
[2019-08-27 13:26:54] app.DEBUG: < 500 [] []

@dannylamb
Copy link
Member Author

I've got a few other pulls lined up, but I'll give this a test soon.

@dannylamb
Copy link
Member Author

Been a while... I'm testing this again after resolving some conflicts.

@dannylamb
Copy link
Member Author

I can confirm this is working after having pulled in from upstream. Just be sure to clear out your external roles 😺

@seth-shaw-unlv
Copy link
Member

Tried to build the PR today but I keep hitting this error:

TASK [Islandora-Devops.fits : Clean Directory (Failure okay)] ******************
Friday 15 November 2019  12:52:33 -0800 (0:00:01.272)       0:08:23.501 ******* 
failed: [default] (item=absent) => {"changed": false, "item": "absent", "msg": "file (/var/www/html/CrayFits/vendor/symfony/monolog-bundle) is absent, cannot continue", "path": "/var/www/html/CrayFits/vendor/symfony/monolog-bundle", "state": "absent"}
failed: [default] (item=directory) => {"changed": false, "item": "directory", "msg": "file (/var/www/html/CrayFits/vendor/symfony/monolog-bundle) is absent, cannot continue", "path": "/var/www/html/CrayFits/vendor/symfony/monolog-bundle", "state": "absent"}
...ignoring

TASK [Islandora-Devops.fits : Change CrayFits directory ownership, group and permissions] ***
Friday 15 November 2019  12:52:33 -0800 (0:00:00.357)       0:08:23.859 ******* 
changed: [default]

TASK [Islandora-Devops.fits : Open file for logging] ***************************
Friday 15 November 2019  12:52:33 -0800 (0:00:00.302)       0:08:24.161 ******* 
changed: [default]

TASK [Islandora-Devops.fits : Restart apache] **********************************
Friday 15 November 2019  12:52:34 -0800 (0:00:00.281)       0:08:24.443 ******* 
fatal: [default]: FAILED! => {"changed": false, "msg": "Could not find the requested service apache2: host"}

I haven't looked into it but it appears to be an issue with Ubuntu v. CentOS. I'll take a look again on Monday.

@seth-shaw-unlv
Copy link
Member

The restart should probably be a notify: restart apache property of one of the tasks like we do on Crayfish instead of a specific task. Again, I'll look at this more closely on Monday.

@seth-shaw-unlv
Copy link
Member

seth-shaw-unlv commented Nov 18, 2019 •
edited
Loading

So, I got this to install on CentOS after some tinkering with ansible-role-fits.

For building on CentOS:

  • apache conf-enabled folder is /etc/httpd/conf.d/ not /etc/httpd/conf_enabled.
  • simply removing the "Restart apache" task from tasks/build-fits-site.yml allowed that step to pass and nothing broke

After that this appears to work fine. One thing I noticed is that the fields it creates are formatted text but all of the example data values I saw seem better suited to plain-text, if not integer values. Are we using formatted text as a generic catch-all?

Also, none of the FITS results drops into Fedora, this is all currently kept in Drupal (including the FITS XML). Is that the default we want?

@seth-shaw-unlv
Copy link
Member

Tagging @ajstanley for the above questions.

@ajstanley
Copy link
Contributor

@seth-shaw-unlv
The fields are being created dynamically from key => value pairs we're getting from the fits XML, so we don't really know what's coming in (and if it's an integer this time will it always be?) so I'm giving it a default -
https://github.com/roblib/islandora_fits/blob/master/src/Services/XMLTransform.php#L297-L318
I'm giving it a type of 'text', which I incorrectly guessed would create a plain text field, but we're getting formatted instead. I'll build a fresh box tonight and try changing the default type to 'string' to see what happens. It ought to give us plain text. Check this space tomorrow.

As for location, we can set the default file storage from within the action to any schema we like - I tend to default to fedora for that which cannot be replaced and Drupal for that which can.

Thanks for the centos feedback - I'm flying blind on that one.

@ajstanley
Copy link
Contributor

Switched the default storage type to 'string' and the newly created fields are all plain text now.

@seth-shaw-unlv
Copy link
Member

Sounds good. I'm fine with merging once we get the ansible roles working on both Ubuntu and CentOS.

@elizoller
Copy link
Member

Are you still needing a tester? I'm using Ubuntu vm.

@seth-shaw-unlv
Copy link
Member

@elizoller, it works fine for Ubuntu. It is the CentOS installation clean-up we are waiting on.

@ajstanley
Copy link
Contributor

ajstanley commented Dec 5, 2019 via email

@ajstanley
Copy link
Contributor

Still ready for testing - I've built both Centos and Ubunto boxes without any trouble.

@seth-shaw-unlv
Copy link
Member

@ajstanley, tested this morning with CentOS 7 and it failed with:

TASK [Islandora-Devops.fits : Restart apache] **********************************
Thursday 02 January 2020  08:10:00 -0800 (0:00:00.212)       0:19:15.568 ****** 
fatal: [default]: FAILED! => {"changed": false, "msg": "Could not find the requested service apache2: host"}

Using the Islandora-Devops.fits as defined in requirements.yml as:

- src: https://github.com/ajstanley/ansible-role-fits
  name: Islandora-Devops.fits
  version: master

@ajstanley
Copy link
Contributor

ajstanley commented Jan 2, 2020 via email

@seth-shaw-unlv
Copy link
Member

@dannylamb, we merged the associated ansible-role-fits PR. Can you revert the ansible-role-fits URL in requirements.yml back to the official repo?

Note: we can keep the "master" tag as we did not cut a new version of ansible-role-fits, figuring we would do so with the Islandora 8.x-1.1 release.

seth-shaw-unlv
seth-shaw-unlv approved these changes Jan 8, 2020
View reviewed changes
Copy link
Member

@seth-shaw-unlv seth-shaw-unlv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@seth-shaw-unlv seth-shaw-unlv merged commit ab0e8af into dev Jan 8, 2020
@seth-shaw-unlv seth-shaw-unlv deleted the fits branch January 8, 2020 22:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@whikloj whikloj whikloj left review comments

@elizoller elizoller elizoller left review comments

@seth-shaw-unlv seth-shaw-unlv seth-shaw-unlv approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@dannylamb @Natkeeran @ajstanley @seth-shaw-unlv @elizoller @whikloj