Shortcode search

[MyriaCore]

Closes #7. Blocked by #8 (since this PR includes commits from #8). Review blocks merge of #8.

While this MR is being reviewed, users can install my fork via Ulauncher's plugin install system, since the shortcode-search branch has been merged into master as of MyriaCore#2.

New Features / Changes

• EmojiSpider.py now scrapes pages on emojipedia.org to retrieve shortcode information
• Users can search shortcodes by beginning their search with :. Otherwise, CLDR names will be searched as per usual.
• When searching for shortcodes, the shortcode that matched is displayed instead of the emoji's name
• I sprinkled a few comments here and there to make the codebase a bit more readable 😄
• Fixes skin tone bug by properly sourcing skin tone data

Technical Information

I've added a shortcode table to the emoji.sqlite database:

[MyriaCore]

Name-encoding bug

As you can see, the blood type emojis don't have any shortcodes associated. However, the shortcodes are on emojipedia, and should be present.

It appears this is happening because the CLDR Shortnames (as listed on unicode.org) use punctuation that isn't correctly url-encoded, so when the spider tries to access them, it accesses a dead link. The full name of 🅰️ is A button (blood type), which turns into a button (blood type), and then a-button-(blood-type) when URL-ized by name_to_shortcodes.

[MyriaCore]

I'm not 100% sure on how I want UX to work here, but I had a few ideas. We'll use the 😄 emoji as an example:

A: How to tell if the user is searching normally or searching 'by shortcode'

1. The user has a preferences setting that tells us this (this is current behavior)
2. If the first character of the search is :, then we assume the user is searching by shortcode. Otherwise, we'll search by CLDR name.
   EX: The user types em :smile, and gets 😄 back. The user then types em grinning face with, then selects 😄, and then receives the intended emote.

B: What to display on the search listing when searching by shortcode

1. The CLDR shortname should be displayed (this is current behavior)
2. The shortname that matched the user's query should be displayed.

Right now, I'm partway through implementing a setup with A.1 + B.1, but I'm honestly liking A.2 + B.2 better. I just don't quite know how I'd implement B.2.

[MyriaCore]

Implementing B.2 - Ideas

1. 🏗️ Add a new table meant specifically for tracking shortcodes. Since skin_tone.name and emoji.name are used as unique identifiers, this new table might do the same. It'd have 2 columns in this case - shortcode.name and shortcode.code. This would allow a single emoji to have multiple entries - it'd just provide entries with different shortcode.codes that share the same shortcode.name.
2. 🚧 ATM, the emoji shortcodes are stored as space-seperated values in a single entry. So, maybe they could be parsed out by main.py, and we could use whichever shortcode matches the query and just display that.

So, idea 1 would provide a more robust implementation of this feature, but idea 2 would be way easier, and we'd make less heavy architecture changes when doing so.

[MyriaCore]

B.2 Implementation 1 - Searching the new DB structure

Here's the current query string that's used by main.py to search the DB when searching with shortcodes:

SELECT em.name, em.code, em.keywords,
       em.icon_apple, em.icon_twemoji, em.icon_noto, em.icon_blobmoji,
       skt.icon_apple AS skt_icon_apple, skt.icon_twemoji AS skt_icon_twemoji,
       skt.icon_noto AS skt_icon_noto, skt.icon_blobmoji AS skt_icon_blobmoji,
       skt.code AS skt_code
FROM emoji AS em
  LEFT JOIN skin_tone AS skt 
    ON skt.name = em.name AND tone = ?
WHERE shortcodes LIKE %?%
LIMIT 8

An individual entry retrieved w/ this query would return a shortcodes dictionary entry, which should be a space-seperated string of shortcodes.

With B.2 Impl 1's proposed changes, the search query would likely have to be something more like this:

SELECT em.name, em.code, em.keywords,
       em.icon_apple, em.icon_twemoji, em.icon_noto, em.icon_blobmoji,
       skt.icon_apple AS skt_icon_apple, skt.icon_twemoji AS skt_icon_twemoji,
       skt.icon_noto AS skt_icon_noto, skt.icon_blobmoji AS skt_icon_blobmoji,
       skt.code AS skt_code, sc.code AS shortcode
FROM emoji AS em
  LEFT JOIN skin_tone AS skt 
    ON skt.name = em.name AND tone = ?
  LEFT JOIN shortcode AS sc 
    ON sc.name = em.name
WHERE shortcode LIKE ?%
LIMIT 8

Not 100% if this actually works, since I had to familiarize myself with how LEFT JOIN works, but I'm pretty sure this should be all good.

[MyriaCore]

At this point, I think I'm probably ready to draft A.2 + B.2.1 as a full solution.

[MyriaCore]

I'm noticing something super funky in the DB. It basically has no data on skin tone variations. Not super sure why.

[gornostal]

Hi @MyriaCore
I really appreciate your work!

I wasn't able to review your PRs yet. I hope I can find some time for that soon.

[MyriaCore]

Thanks @gornostal! Please review #8 first, as this PR contains code from there.

[MyriaCore]

At this point, unless I find some more bugs, the implementation is pretty much done. This PR should be ready for review.

[gornostal]

Thanks a lot for these improvements and sorry it took so long!
Great work 👍