GeoArchive - Zotero.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is an initial attempt at the process for incorporating items representing documents in a GeoArchive collection within the GeoKB. The knowledgebase is the place where we will be able to develop the richest suite of linkages for what we are able to extract or identify within document reference materials. It may also serve as the logical alternative location for metadata associated with documents, providing the future-proofing of using a particular tech base for storing document content with citation metadata.\n",
    "\n",
    "This process caused me to examine a couple of new issues with the GeoKB, establishing some new properties:\n",
    "\n",
    "* I added a GDDID property to provide the linkage to the GeoDeepDive infrastructure where we've run a given document through one or more processing pipelines. This provides the conduit for us to incorporate extracted information into the GeoKB. The formatter URL creates a link that ties into the /articles/ route in the xDD API.\n",
    "* I added a Zotero ID property to house a compound identifier with the unique Library ID and the Item Key for the item containing document metadata. This gives us the hook to keep things in sync over time. These identifiers are also reflected in the reference URL provided to link into the library, but it's the identifiers themselves that are actionable via the API. (Note: ultimately, I still want to establish a hanle server to register these identifiers against creating an evergreen resolver for future-proofing.)\n",
    "* I did a little bit of work on country place names to round out the reference source in the GeoKB and established linkages based on what was previously brought into the metadata from spreadsheet or file system naming convention sources. I opted to use a new \"addresses place\" property here as opposed to a more specific property like \"country\" as I think this is the way we want to model the linkages from a document to different concepts.\n",
    "* I was able to get a handful of matches for some of the more major mining projects to existing \"mine\" items previously pulled in from the GNIS process. This is pretty crude at this point, but I put them in as linkages for demonstration purposes at least. I opted to use a different (also new) property here, addresses feature, as there is a fairly distinct difference in the intent of a mine (or a dam or whatever) kind of place vs. a geopolitical boundary. But I also think we want to use a higher level concept than \"the mine,\" \"the dam,\" etc. for the features that we are able to determine the contents of a document address in some way.\n",
    "* In both cases, place and feature, I included a \"reference statement\" (may eventually be better as a qualifier) indicating that the particular connection to an identified place or mine was made from imperfect metadata. We'll have to play with this idea in practice as we continue to work through use cases.\n",
    "\n",
    "A next step here is to pull in the \"addresses subject\" (another new property) claims from both citation metadata pulled in from the inventory process as well as extractions from DeepDive. I'm going to concentrate on the latter with geologic time as a demonstration.\n",
    "\n",
    "I also hesitated to pull the point coordinates that we have for some of these records in at this point. From a broader perspective across the knowledgebase, those are really point locations likely indicating where the mining property is. They come from the documents and are associated with them as items, but they will be more useful in the context of the mining property itself. The same could be said for the other concepts here as well, so thinking that through is still something we need to do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pyzotero import zotero\n",
    "import pandas as pd\n",
    "from wbmaker import WikibaseConnection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "geokb = WikibaseConnection(\"GEOKB_CLOUD\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Establish connection to Zotero group library from source\n",
    "\n",
    "# Select source item to process\n",
    "source_item_id = geokb.ref_lookup['GeoArchive Collection of National Instrument 43-101 Technical Reports']\n",
    "source_item = geokb.wbi.item.get(source_item_id)\n",
    "\n",
    "# Extract library identifier for Zotero group\n",
    "source_instance_of = source_item.claims.get_json()[geokb.prop_lookup['instance of']][0]\n",
    "# I obviously need to do something better with this once I settle on conventions for different source types\n",
    "reference_url = source_instance_of[\"references\"][0][\"snaks\"][geokb.prop_lookup['reference URL']][0][\"datavalue\"][\"value\"]\n",
    "ni43101_library_id = reference_url.split(\"/\")[4]\n",
    "\n",
    "ni43101_api_key = os.environ['NI43101_KEY']\n",
    "\n",
    "ni43101_library = zotero.Zotero(\n",
    "    ni43101_library_id,\n",
    "    'group', \n",
    "    ni43101_api_key\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# It takes a long time to pull items, and I need to work through my caching strategy a bit better\n",
    "\n",
    "#all_items = ni43101_library.everything(ni43101_library.items())\n",
    "\n",
    "# df_all_items = pd.DataFrame([i[\"data\"] for i in all_items])\n",
    "# df_all_items.to_pickle(\"data/ni43101_library_20200419.p\")\n",
    "df_all_items = pd.read_pickle(\"data/ni43101_library_20200419.p\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "reports = df_all_items[df_all_items.itemType == \"report\"].reset_index()\n",
    "reports[\"project_name\"] = reports.seriesTitle.apply(lambda x: x.replace(\"Project: \", \"\").strip())\n",
    "reports[\"zotero_id\"] = reports.key.apply(lambda x: \":\".join([ni43101_library_id, x]))\n",
    "reports[\"zotero_url\"] = reports.key.apply(lambda x: f\"https://www.zotero.org/groups/{ni43101_library_id}/usgs_ni_43-101_reports/items/{x}/library\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_countries = \"PREFIX%20wd%3A%20%3Chttps%3A%2F%2Fgeokb.wikibase.cloud%2Fentity%2F%3E%0APREFIX%20wdt%3A%20%3Chttps%3A%2F%2Fgeokb.wikibase.cloud%2Fprop%2Fdirect%2F%3E%0A%0ASELECT%20%3Fitem%20%3FitemLabel%20%3FitemAltLabel%0AWHERE%20%7B%0A%20%20%3Fitem%20wdt%3AP1%20wd%3AQ27%20.%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22%20%7D%0A%7D%0A\"\n",
    "df_countries = geokb.wb_ref_data(query=query_countries)\n",
    "df_countries[\"item\"] = df_countries.item.apply(lambda x: x.split('/')[-1])\n",
    "\n",
    "query_us_states = \"PREFIX%20wd%3A%20%3Chttps%3A%2F%2Fgeokb.wikibase.cloud%2Fentity%2F%3E%0APREFIX%20wdt%3A%20%3Chttps%3A%2F%2Fgeokb.wikibase.cloud%2Fprop%2Fdirect%2F%3E%0A%0ASELECT%20%3Fitem%20%3FitemLabel%0AWHERE%20%7B%0A%20%20%3Fitem%20wdt%3AP1%20wd%3AQ229%20.%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22%20%7D%0A%7D%0A\"\n",
    "df_us_states = geokb.wb_ref_data(query=query_us_states)\n",
    "df_us_states[\"item\"] = df_us_states.item.apply(lambda x: x.split('/')[-1])\n",
    "\n",
    "df_countries['aliases'] = df_countries.itemAltLabel.apply(lambda x: [i.strip() for i in x.split(\",\")] if isinstance(x, str) else None)\n",
    "place_lookup = pd.concat([\n",
    "    df_us_states,\n",
    "    df_countries[[\"item\",\"itemLabel\"]],\n",
    "    df_countries[df_countries.aliases.notnull()][[\"item\",\"aliases\"]].explode(\"aliases\").rename(columns={\"aliases\": \"itemLabel\"})\n",
    "]).drop_duplicates().set_index(\"itemLabel\")[\"item\"].to_dict()\n",
    "\n",
    "query_mines = \"PREFIX%20wd%3A%20%3Chttps%3A%2F%2Fgeokb.wikibase.cloud%2Fentity%2F%3E%0APREFIX%20wdt%3A%20%3Chttps%3A%2F%2Fgeokb.wikibase.cloud%2Fprop%2Fdirect%2F%3E%0A%0ASELECT%20%3Fitem%20%3FitemLabel%0AWHERE%20%7B%0A%20%20%3Fitem%20wdt%3AP1%20wd%3AQ3646%20.%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22%20%7D%0A%7D%0A\"\n",
    "df_mines = geokb.wb_ref_data(query=query_mines)\n",
    "df_mines[\"item\"] = df_mines.item.apply(lambda x: x.split('/')[-1])\n",
    "\n",
    "mine_lookup = df_mines.set_index(\"itemLabel\")[\"item\"].to_dict()\n",
    "\n",
    "query_commodity = \"PREFIX%20wd%3A%20%3Chttps%3A%2F%2Fgeokb.wikibase.cloud%2Fentity%2F%3E%0APREFIX%20wdt%3A%20%3Chttps%3A%2F%2Fgeokb.wikibase.cloud%2Fprop%2Fdirect%2F%3E%0A%0ASELECT%20%3Fitem%20%3FitemLabel%20%3FitemAltLabel%0AWHERE%20%7B%0A%20%20%3Fitem%20wdt%3AP1%20wd%3AQ406%20.%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22%20%7D%0A%7D%0A\"\n",
    "df_commodity = geokb.wb_ref_data(query=query_commodity)\n",
    "df_commodity[\"item\"] = df_commodity.item.apply(lambda x: x.split('/')[-1])\n",
    "df_commodity['aliases'] = df_commodity.itemAltLabel.apply(lambda x: [i.strip() for i in x.split(\",\")] if isinstance(x, str) else None)\n",
    "\n",
    "commodity_lookup = pd.concat([\n",
    "    df_commodity[[\"item\",\"itemLabel\"]],\n",
    "    df_commodity[df_commodity.aliases.notnull()][[\"item\",\"aliases\"]].explode(\"aliases\").rename(columns={\"aliases\": \"itemLabel\"})\n",
    "]).drop_duplicates().set_index(\"itemLabel\")[\"item\"].to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "report_tags = reports[[\"key\",\"tags\"]].explode(\"tags\")\n",
    "report_tags[\"tag\"] = report_tags.tags.apply(lambda x: x[\"tag\"] if isinstance(x, dict) else x)\n",
    "report_tags.drop(columns=[\"tags\"], inplace=True)\n",
    "report_tags = report_tags.dropna()\n",
    "report_tags[\"tag_type\"] = report_tags.tag.apply(lambda x: x.split(':')[0])\n",
    "report_tags[\"tag\"] = report_tags.tag.apply(lambda x: x.split(':')[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "unique_commodity_tags = list(report_tags[report_tags.tag_type == \"Commodity\"].tag.unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item</th>\n",
       "      <th>itemLabel</th>\n",
       "      <th>itemAltLabel</th>\n",
       "      <th>aliases</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [item, itemLabel, itemAltLabel, aliases]\n",
       "Index: []"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_commodity[df_commodity.itemLabel == 'zinc']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Zn',\n",
       " 'U',\n",
       " 'Ni',\n",
       " 'PGE',\n",
       " 'Pb',\n",
       " 'PGM',\n",
       " 'La',\n",
       " 'REE',\n",
       " 'potash',\n",
       " 'W',\n",
       " 'V',\n",
       " 'Ti',\n",
       " 'Pd',\n",
       " 'Pt',\n",
       " 'Nb',\n",
       " 'sand',\n",
       " 'silica',\n",
       " 'B',\n",
       " 'P',\n",
       " 'quartz',\n",
       " 'Sb',\n",
       " 'Sn',\n",
       " 'Ta',\n",
       " '',\n",
       " 'Se',\n",
       " 'barite',\n",
       " 'Re',\n",
       " 'Te',\n",
       " 'Hf',\n",
       " 'Th',\n",
       " 'Zr',\n",
       " 'F',\n",
       " 'Mg',\n",
       " 'Ce',\n",
       " 'Sd',\n",
       " 'Sil',\n",
       " 'gypsum',\n",
       " 'Rh',\n",
       " 'Sc',\n",
       " 'Ba',\n",
       " 'S',\n",
       " 'Y',\n",
       " 'Nd',\n",
       " 'Sm',\n",
       " 'Cly',\n",
       " 'TLC',\n",
       " 'bauxite',\n",
       " 'pumice',\n",
       " 'Rb',\n",
       " 'fluorspar',\n",
       " 'peat',\n",
       " 'halloysite',\n",
       " 'phosphate',\n",
       " 'CD',\n",
       " 'travertine',\n",
       " 'LWA',\n",
       " 'TL',\n",
       " 'quartzite',\n",
       " 'N',\n",
       " 'wollastonite',\n",
       " 'Na',\n",
       " 'O',\n",
       " 'Al,',\n",
       " 'Si']"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[i for i in unique_commodity_tags if i not in commodity_lookup]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_zotero_id = \"PREFIX%20wdt%3A%20%3Chttps%3A%2F%2Fgeokb.wikibase.cloud%2Fprop%2Fdirect%2F%3E%0A%0ASELECT%20%3Fitem%20%3Fzotero_id%0AWHERE%20%7B%0A%20%20%3Fitem%20wdt%3AP97%20%3Fzotero_id%20.%0A%7D%0A\"\n",
    "df_zotero_id = geokb.wb_ref_data(query=query_zotero_id)\n",
    "df_zotero_id[\"item\"] = df_zotero_id.item.apply(lambda x: x.split('/')[-1])\n",
    "wb_items_lookup = df_zotero_id.set_index(\"zotero_id\")[\"item\"].to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "diamond Q453\n",
      "Au Q329\n",
      "Au Q329\n",
      "Cu Q315\n",
      "Au Q329\n"
     ]
    }
   ],
   "source": [
    "for index, row in reports.head().iterrows():\n",
    "    for tag in row.tags:\n",
    "        if tag[\"tag\"].startswith(\"Commodity:\"):\n",
    "            commodity = tag[\"tag\"].split(\":\")[1]\n",
    "            print(commodity, commodity_lookup[commodity])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_reference = geokb.datatypes.Item(\n",
    "    prop_nr=geokb.prop_lookup['data source'],\n",
    "    value=source_item_id\n",
    ")\n",
    "\n",
    "for index, row in  reports.iterrows():\n",
    "    item_references = geokb.models.References()\n",
    "    item_references.add(source_reference)\n",
    "\n",
    "    item_qid = wb_items_lookup[row.zotero_id] if row.zotero_id in wb_items_lookup else None\n",
    "\n",
    "    if item_qid:\n",
    "        item = geokb.wbi.item.get(item_qid)\n",
    "    else:\n",
    "        item = geokb.wbi.item.new()\n",
    "\n",
    "    item.labels.set('en', row[\"title\"])\n",
    "    item.descriptions.set('en', \"an NI 43-101 Technical Report pulled from the GeoArchive collection\")\n",
    "\n",
    "    claims = geokb.models.Claims()\n",
    "\n",
    "    claims.add(\n",
    "        geokb.datatypes.Item(\n",
    "            prop_nr=geokb.prop_lookup['instance of'],\n",
    "            value=geokb.class_lookup['NI 43-101 Technical Report'],\n",
    "            references=item_references\n",
    "        )\n",
    "    )\n",
    "\n",
    "    claims.add(\n",
    "        geokb.datatypes.ExternalID(\n",
    "            prop_nr=geokb.prop_lookup['Zotero ID'],\n",
    "            value=row.zotero_id\n",
    "        )\n",
    "    )\n",
    "\n",
    "    claims.add(\n",
    "        geokb.datatypes.URL(\n",
    "            prop_nr=geokb.prop_lookup['reference URL'],\n",
    "            value=row.zotero_url\n",
    "        )\n",
    "    )\n",
    "\n",
    "    if isinstance(row.callNumber, str) and row.callNumber.startswith(\"gddid:\"):\n",
    "        gddid = row.callNumber.split(\":\")[-1].strip()\n",
    "        claims.add(\n",
    "            geokb.datatypes.ExternalID(\n",
    "                prop_nr=geokb.prop_lookup[\"GDDID\"],\n",
    "                value=gddid\n",
    "            )\n",
    "        )\n",
    "\n",
    "    if row.project_name in mine_lookup:\n",
    "        mine_ref_statement = geokb.datatypes.String(\n",
    "            prop_nr=geokb.prop_lookup['reference statement'],\n",
    "            value=\"Linkage to this mine was established with a simple name match from imperfect metadata identifying the mineral development project\"\n",
    "        )\n",
    "        mine_connection_refs = geokb.models.References()\n",
    "        mine_connection_refs.add(source_reference)\n",
    "        mine_connection_refs.add(mine_ref_statement)\n",
    "\n",
    "        claims.add(\n",
    "            geokb.datatypes.Item(\n",
    "                prop_nr=geokb.prop_lookup['addresses feature'],\n",
    "                value=mine_lookup[row.project_name],\n",
    "                references=mine_connection_refs\n",
    "            )\n",
    "        )\n",
    "\n",
    "    if row.place in place_lookup:\n",
    "        place_ref_statement = geokb.datatypes.String(\n",
    "            prop_nr=geokb.prop_lookup['reference statement'],\n",
    "            value=\"Linkage to this mine was established with a simple name match from imperfect metadata identifying the geographic area of the mining project\"\n",
    "        )\n",
    "        place_connection_refs = geokb.models.References()\n",
    "        place_connection_refs.add(source_reference)\n",
    "        place_connection_refs.add(place_ref_statement)\n",
    "\n",
    "        claims.add(\n",
    "            geokb.datatypes.Item(\n",
    "                prop_nr=geokb.prop_lookup['addresses place'],\n",
    "                value=place_lookup[row.place],\n",
    "                references=place_connection_refs\n",
    "            )\n",
    "        )\n",
    "\n",
    "    item.claims.add(claims=claims)\n",
    "\n",
    "    response = item.write(summary=\"Added initial NI 43-101 item from Zotero library\")\n",
    "    print(row[\"title\"], response.id)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "geokb",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "73445f04a821fa03c75b5864e622841100a6f63483b936d10ffa5383ef328dbe"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}