Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ETL Pipe for download and ingest EEZ [MARXAN-1618] #1121

Draft
wants to merge 6 commits into
base: develop
Choose a base branch
from

Conversation

aagm
Copy link
Member

@aagm aagm commented Jun 1, 2022

Overview

This pr adds the makefile pipe to manage EEZ marine boundaries.

Designs

Link to the related design prototypes (if applicable)

Testing instructions

Either make a full db restoration:

  1. make clean-slate
  2. make start-api
  3. make seed-api-init-data
  4. make seed-geodb-data

Or run just for Testing the individual pipe:
docker-compose -p marxan-cloud -f ./data/docker-compose-data_management.yml up --no-start --build marxan-seed-data marxan-seed-data && docker-compose -p marxan-cloud -f ./data/docker-compose-data_management.yml run marxan-seed-data make seed-eez

Checklist before submitting

  • Meaningful commits and code rebased on develop.
  • If this PR adds feature that should be tested for regressions when
    deploying to staging/production, please add brief testing instructions
    to the deploy checklist (docs/deployment-checklist.md)
  • Update CHANGELOG file

@aagm aagm requested a review from hotzevzl June 1, 2022 09:08
@vercel
Copy link

vercel bot commented Jun 1, 2022

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated
marxan ✅ Ready (Inspect) Visit Preview Jun 1, 2022 at 2:34PM (UTC)
marxan-storybook ✅ Ready (Inspect) Visit Preview Jun 1, 2022 at 2:34PM (UTC)

@hotzevzl hotzevzl changed the title ETL Pipe for download and ingest EEZ ETL Pipe for download and ingest EEZ [MARXAN-1618] Jun 1, 2022
data/data_download/Makefile Outdated Show resolved Hide resolved
Co-authored-by: andrea rota <47385021+hotzevzl@users.noreply.github.com>
Comment on lines +34 to +43
curl --location --request POST 'https://www.marineregions.org/download_file.php?name=World_EEZ_v11_20191118.zip' \
--header 'Cookie: PHPSESSID=870e305efbc0519d59b361427dbd8336; vliz_webc=vliz_webc1' \
--form 'name="Jen"' \
--form 'organisation="TNC"' \
--form 'email="admin@marxan.com"' \
--form 'country="EEUU"' \
--form 'user_category="academia"' \
--form 'purpose_category="Conservation"' \
--form 'agree="1"' \
--output './World_EEZ_v11_20191118.zip'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry I had missed parts of this PR while I reviewed it - probably my tab wasn't refreshed when you pushed later changes... this bit, I recommend to handle it differently, without hardcoding cookie and most of the form data such as name, org, email, country, user_category, purpose_category

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how to do this... my first thought would be to move the whole curl command to a simple shell script that is committed to the repo with placeholders, and needs to be copied to a specific file (that should be added to .gitignore, similarly to env.default vs .env), mounted in the eez ingestion container, and executed in the eez Makefile

@hotzevzl hotzevzl self-requested a review June 1, 2022 15:21
@hotzevzl
Copy link
Member

hotzevzl commented Jun 1, 2022

@aagm for the failing tests (https://github.com/Vizzuality/marxan-cloud/runs/6691428000?check_suite_focus=true for example), IIRC each migration file is wrapped in a transaction so we may need two migrations in order for the enum to be fully committed by the time it's then used. I don't think (again IIRC, and IIRC the docs are thin on this - I'd need to read the source code...) that using two distinct queryRunner.query() steps in the same migration will be enough.

Or avoiding touching that enum altogether and going the separate table route 😜

@andresgnlez andresgnlez marked this pull request as draft July 31, 2023 06:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Data Everything related data WIP Work in progress
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants