Create Windows installer

[J5lx]

Original outdated description

Now that I'm going to get busier again, I'd like to get some of the working stuff out that I've been working on lately, starting with this. As I already mentioned on IRC, I dug up some of my old NSIS stuff and adapted it to create a Windows installer for Pencil2D. Right now, there are some things that could possibly be added (checking for an existing installation, for instance), but the actual (un)installation routine seems to be working splendidly at this point. The only thing that definitely needs improvement before this can reasonably be released is the graphics; I'm by no means a professional designer and I'd say it shows :) While I'm at it, time for some screenshots:

• Language selection:
  
• Installer welcome page:
  
• Installer license confirmation:
  
• Installer mode selection:
  
• Installer destination selection:
  
• Installer start menu folder selection:
  
• Installer progress page:
  
• Installer finish page:
  
• Look, we even got ourselves an uninstall entry, complete with meta information:
  
• Uninstaller welcome page:
  
• Uninstaller confirmation page:
  
• Uninstaller progress page:
  
• Uninstaller finish page:
  

If anyone wants to try it themselves, look here: https://drive.google.com/open?id=1vR04yL4LUrbvPwRFPi1ajS7Una0CN01s. Please note that the contents of these "demo" installers are slightly different from the current releases, most notably I was too lazy to download the FFmpeg binaries.

To create the installer, installer.nsi needs to be processed by makensis like this:

• Windows:

  makensis /DBITS=32|64 /DVERSION=1.2.3 /DINSTALLFILES=path\to\directory\to\be\installed\relative\to\installer.nsi path\to\util\installer\installer.nsi
  

• *nix (for cross-compilation etc.)

  makensis -DBITS=32|64 -DVERSION=1.2.3 -DINSTALLFILES=path/to/directory/to/be/installed/relative/to/installer.nsi path/to/util/installer/installer.nsi
  

Due to the NSIS compiler treating all paths as relative to the main script, the resulting installer can then be found in the same directory as installer.nsi.

As I've previously said, I consider the installer itself pretty much complete, but the things I'd like to get ready before merging are:

• Build it automatically with GitHub Actions
• Make the icon a little cleaner. It currently looks kind of not-so-pretty at small sizes, this is probably solvable by embedding more pre-rendered sizes in the .ico file, I'll just have to recall how to do that again.
• Add options for additional shortcuts (desktop etc.)
• Revisit UI strings
• Mechanisms to translate UI strings via Transifex
• Install runtime libraries
• file icons?
• detect previous installation and handle upgrades
• Address TODOs in code
• Implement update check (maybe)

Let me know what you think, be it about the whole thing overall or about some of the individual aspects!

closes #1762

[Kaikogf]

Great !

[Jose-Moreno]

@J5lx Well that's cool! thank you for the hard work 😄 , but I'd like to ask, what are exactly the advantages of having and installer as opposed to have a "portable" version?

[J5lx]

Maybe it’s just me, but over time I got the impression that a) there are a lot more non-technical people that it often seems and b) many of these non-technical people have no idea how to deal with any sort of download that is not itself a document or an executable ­– including ZIP files – because they usually don’t come in contact with them. The fact that Windows calls ZIP files compressed folders doesn’t make it any better since it leads people into thinking there’s absolutely no need to extract them entirely, even when Windows tells them otherwise. Actually it seems that an awful lot of the comments on our download page are somehow related to ZIP file extraction (people extracting only some of the files or not knowing how to deal with the archive altogether, etc). Speaking of it, there are also some comments about missing MSVC runtime libraries, maybe I can find a way to detect whether the needed MSVC runtime is already installed and otherwise ask the users whether they want to install it too.

tl;dr; I’m pretty confident that an installer is a good idea, but please let me know if you have any concerns. Of course this PR doesn’t mean we should stop providing the portable ZIP files altogether, it’s more about offering an option that is more straightforward for less technical users.

[Jose-Moreno]

@J5lx You're right, in that regard an installer would be great for our Windows non-technical users (mostly kids), what could be done similarly for Mac though? and is it worth to do try that for Linux? (I like the app image, but the only installer-like thingy I know is those RPM packages and I don't know if it's FOSS-y enough lol)

In my case I love using the portable, because I hate installing things on windows since well ...it's windows and you get all the bloated goodies for installing on it, most of my FOSS apps are portable unless I have to specifically install it for some sort of obscure reason, but even then all of those packages sure give the option for most users to pick between an installer and a portable version, so I agree it would be a good thing for P2D.

As a related question, would it be possible to make this installer have a VCredist check? that is actually the number 2 problem with anyone installing on Windows, and the solution is far more technical for the average consumer than extracting a compressed folder.

[chchwy]

@J5lx good news to have an installer!

@Jose-Moreno Just as @J5lx said, I've been answering the "missing files" questions many times. A bit surprised that some of the people (not many but some) have no idea about zip & unzip. Also, I believe that the installer is able to handle vc redistributable installation, just need to find out how to do it.

But yeah, personally I prefer a portable version if I have choices. We could probably provide both.

Another good idea is giving Windows Store a go.
For example, Krita and Paint.net are available on Windows Store now
https://blog.getpaint.net/2017/09/29/paint-net-is-now-available-on-the-windows-store/
https://krita.org/en/item/krita-available-from-the-windows-store/

In the blog posts, they both mentioned one-click-installation and auto-update, which is good for those non-technical people.

[scribblemaniac]

Very nice! As mentioned, a significant number of people were having difficulties with the zipped executable, so I definitely second the usefulness of this, even if I would opt for the portable version personally. Another step to increasing usability would be static building for Windows, which if I understand what it is correctly, would allow for just the exe file to be distributed. Windows Store is also good.

@Jose-Moreno Mac/Linux should not require a similar thing. Zip files are extracted when opened, rather than "peeked-into" as in Windows. Also for both Mac's .app and Linux's AppImage formats, all of the libraries and resources are self-enclosed, so there's no issues with people not copying some important part of it. RPM packages are fairly FOSS-y imho (although some package managers do one better by compiling from the source).

[J5lx]

I don’t know about Mac, but for Linux our current AppImages should suffice, really. Distribution packages are nice, of course, but they are sometimes tricky to get right and are better taken care of by people who actually know their stuff, aka package maintainers of the respective distribution. That’s why mapreri as a Debian person is doing the deb package and I as an Arch user am doing the Arch one.

Regarding static binaries, I can see why they would be nice but I personally don’t really like fat binaries. I’m also not sure what to do with Qt’s plugin architecture (esp. system plugin) and the FFmpeg binary in that case.

As for runtime distribution, it is 100% possible (there are complete solutions available) and it’s totally on my todo list at this point. Getting it into the Microsoft store would probably be a good idea, too, but naturally we’ll have to see how exactly that works and all and whether they take fees or anything. IIRC there was also some talk about getting Pencil into the Mac store some time ago on IRC, we should probably check that out as well.

Also, I actually prefer installation over portable, mainly because it’s so goddamn quick and easy to get just about anything installed on Arch with just the press of one button (kinda), but also because it feels cleaner than having loose app folders floating around.

[chchwy]

At the moment, publishing an app on Windows Store costs 19 USD, it's a one-time fee and no renewal charge. But for Mac Store, it costs 99 USD per year, which might be considerable.

[Kaikogf]

[The green line and the cross red line over Pencil2D's icon tell a little bit of the story here: they are made by Pencil2D, no mistake! 👍 I like it ]

The Installer program is Much more intuitive. Great

[scribblemaniac]

How about these for the install/uninstall icons?

SVG versions here

[Kaikogf]

The green lines are not intuitive. My thoughts are that you can make an OK mark (V) within the same style as this last red cross uninstaller icon (I like it more than the previous one)

[J5lx]

@scribblemaniac I really like those! Personally I find both the check mark ✓ and your arrow-pointing-onto-tray (or whatever one might call it) equally comprehensible, so I’m not sure if the the check mark is better as Kaíko suggested. Also @Kaikogf, I’m afraid I have to admit I used GIMP for the original icons for its guides and scaling features. Shame on me! :)

[Kaikogf]

@J5lx
Now I can see what the green mark meant. So knowing that it started to make sense using it, but I would guess that some people would not grasp it at first sight.

[scribblemaniac]

Check

SVG is here

[J5lx]

I have now replaced the images with those by @scribblemaniac. Looks good IMO:

[Kaikogf]

Looks good really!

I have tested the DEMO Installer and it is working just fine! 👍
I have though two suggestions here: after the succeeded installation of Pencil2D it was not created a shortcut on the desktop (win8.1x64bits).
I know It is fairly easy to create one, but who knows ... just wanted to say something!

After the uninstallation of Pencil2D there was left its main folder with subfolders.. A very few programs that I know are capable of deleting its own folder after its uninstallation and I found it to be clean

;)

best regards

[Kaikogf]

I am waiting to see Portuguese (Portugal) to be implemented here... just to check, once again, the translation

[J5lx]

That the installer does not create a desktop shortcut is 100% correct intended behaviour. If you ask me, desktop symbols are an annoying pest and only an excuse for not having an efficient launcher, and they should be purged from this world :) But of course not everyone shares that opinion, so I’ll see what I can do about that. As for the leftover folder, that should not happen if you went through the uninstallation routine properly (i.e. quit Pencil2D before uninstallation etc.). If it does, I’d consider it a bug, but it never happened to myself, so I’ll need some more details from you:

• In which mode did you run the installer (current user / all users)?
• Where did you install the program (C:\…)
• Which directories and files exactly are still present after uninstallation? E.g.

  C:\path\to\instdir\
  C:\path\to\instdir\Pencil2D.exe
  C:\path\to\instdir\audio\
  C:\path\to\instdir\audio\qtaudio_wasapi.dll
  C:\path\to\instdir\audio\qtaudio_windows.dll
  ⋮
  

Regarding the missing Portuguese (Portugal), that’s because Pencil2D itself currently doesn’t include it, but I noticed it’s already fully translated on Transifex, so we can easily add it to the installer as well (just add !insertmacro MUI_LANGUAGE "Portuguese" in installer.nsi).

[Kaikogf]

"The annoying pest" has no more room for another shortcut jejeje... what a desktop

I run both methods, All Users and Current User
Installed the program to its default path (kept the one of the launcher installation for both methods)

I tried the demo again for checking... this is what was left after uninstalling Pencil2D [All Users]
Pencil2D path: C:\Programs\Pencil2D

folders

When installing it for Current User, Pencil2D will be hidden on the AppData folders, so this made me wonder if a beginner (non-technical) tries it he/she will have a hard time getting inside Pencil2D installation folder, mainly if he wants to find it for adding a shortcut else where (Windows here) since Pencil2D's launcher doesn't create one
Pencil2D path: C:\Users\Carlos\AppData\Local\Pencil2D
The left folders are the same as the previous image, nothing inside of them

While installing Pencil2D I did not create a shortcut on the Main Menu (checked mark box)

One thing I would like to point out is that every time after installing Pencil2D and opening it for the first time, my HUION pen doesn't work, meaning its tip moves all over the UI but nothing reacts to it, no selection no drawing, no moving, etc. After closing and restarting it again it works fine

To be honest I am afraid you will say that all, from top to bottom, is normal

[J5lx]

Okay, thanks for the info, I’ll see what I can do about it.

Also, valid point about Pencil not being easily accessible in the AppData folder when no shortcuts were created. I’ll definitely add options for additional shortcuts.

[J5lx]

FTR, I have now added pre-scaled versions in 16x16, 32x32 and 48x48 (as per recommendation from Microsoft) to the icon file, so the icon now looks much, much better at smaller sizes.

[J5lx]

@Kaikogf I finally managed to obtain a legal copy of Win8.1 (which was a little tricky since MS apparently doesn't want people to create new installations of it) to have a look at your issue, but I was not able to reproduce it. I tried installing and uninstalling both per-user and system-wide, both with and without shortcuts, and in all of the cases all files and folders were completely removed without any leftovers. I also looked for hints on the internet but I didn't find any, so I'll have to ask you a few more questions and hope that this gets us any closer to the solution.

1. What locale (i.e. display language, number format etc.) are you using? (Portuguese (Portugal) maybe?) Are there any characters in your file system paths that are not part of the English alphabet, e.g. letters with diacritics or maybe even characters from scripts other than the Latin script?
2. You said that very few programs you know of are capable of cleanly deleting their own folder. Could you list some of the programs that you found to be capable of this? Is it possible that there were no problems originally and that the problems only started occurring at a certain point in time?
3. Do you have any other programs installed that you suspect might interfere with the uninstallation, particularly the kind that integrates deeply with the system (such as anti-virus solutions)?

[MrStevns]

Excellent, great job finishing the remaining work!

As for future changes and continued contributions to this setup, how does one edit such an installation setup? Is everything written by hand or do you have a tool to generate most of it?

[J5lx]

In general, nearly all of it is written by hand, except for localisation and files deployed by windeployqt. The source file for the files copied by windeployqt is generated at build-time (see the harvest_files stuff in create-package.sh), but for testing locally it’s also possible to just comment out <ComponentGroupRef Id="windeployqt" /> in pencil2d.wxs and skip those files altogether. The localisation files (.xlf) are converted to WiX’s own format at build-time as well (just like the .ts files for Pencil2D itself), using the Tikal tool from the Okapi project. Of course that means it’s a bit more tiresome to do it locally, but has the advantage that the CI will pick up changes made to existing translations by the Transifex bot with no additional action required. And of course, the translation source file is likewise generated from the original .wxl file using Tikal.

In summary, among the files that are part of this PR, only the .xlf localisation files are generated, all other files are hand-written.

In general, the process of building the installer locally isn’t terribly difficult. Of course, you first need to have WiX and Tikal available locally. The latter is just a zip download from okapiframework.org, the former can be installed from the command line using the dotnet/wix/nuget commands in install-dependencies.sh. Outline of the build steps:

1. Build the (release) DLL from the pencil2d.pro project (like any other qmake project). Probably needs to be built against MSVC, MinGW likely won’t work.
2. Create the windeployqt source file (or comment out the line mentioned above)
3. Convert translations using Tikal (and manually update the Culture and Language attributes, since Tikal doesn’t know how to handle those)
   • Of course, depending on what you’re working on it’s also possible to exclude the localisation files by commenting them out in pencil2d.bundle.wxs
4. Build the MSI installation database from pencil2d.wxs
5. Build the .exe installation bundle from pencil2d.bundle.wxs

The actual commands for all of that can be found in create-package.sh. For incremental builds during development, you would normally just repeat steps 4 and 5 as needed (and maybe 1 if you make any changes to pencil2d.cpp). Tbh, I’ve also been thinking about potentially extracting the packaging steps performed by the CI into something that can be run locally, but I’ve decided not to do include that in this PR in order to not needlessly inflate its scope.

Here is a basic overview of the source files:

• pencil2d.{cpp,def,pro}: Custom installation logic – mainly reading installation options from existing installations and updating the progress text
• pencil2d.wxs: MSI installation database
• pencil2d.bundle.wxs: .exe installation bundle (bundles our MSI and the MSVC redistributable)
• pencil2d*.{thm,png}, cog*.{png,svg}: Bundle theme source files. WiX also has a theme viewer available from their GitHub releases (part of the additional tools download) which can be used to preview changes to the theme while working on it.
• pencil2d.wxl, okf_xml_wxl.fprm, translations/*: Localisation-related files
• win_pclx?_icon.{ico,xcf}: File type icon. As noted in one of my previous comments above, this is loosely based on the macOS file type icons we already have.

WiX’s documentation can be found here.

Perhaps I should also add a writeup of this for our developer docs, but for now I hope this helps with reviewing at least.

[J5lx]

PR has been rebased on master. As a small addendum to my previous comment, another useful tool while working on the installer is Orca from the Windows SDK which can be used to inspect the generated MSI.

[J5lx]

I’ve added some proper technical documentation about the installer now. It’s a bit more in-depth than my previous comment in this thread.

[MrStevns]

I've looked through the code and it looks fine to me.
From what I can read though, Wix can be a pretty sizable beast to maintain, and I can imagine that from looking at all the code that you've written already.

As I recall you decided on Wix because it's what you've used before, right?

With all the effort put into this PR, I wouldn't want to see it go to waste but i can't help but ask.

Did you consider an installer that might have required less work to maintain?

Both Nullsoft Scriptable Install System and Inno Setup should be easier to maintain afaik but of course might not be as flexible is writing everything yourself.
Maintaining install scripts is imo. as fun as maintaining build scripts, eg. not fun at all. I'd just rather we spend energy elsewhere than on maintaining this if it means less time to work on actual features for the application 😅

Overall good job completing this PR, it's been in a draft state for quite some time and it's a much asked feature, so I think our users will definitely appreciate this!

Edit:

Ah.. as I can read from the history, you did in fact you NSIS in the beginning! my bad... why did you change software?

[J5lx]

As I recall you decided on Wix because it's what you've used before, right?

No, that was NSIS (as you’ve already realised).

Wix can be a pretty sizable beast to maintain

In which way? Code size? I agree that it takes more code than I’d like, but a lot of that custom code is relatively cosmetic – theming, progress reporting, localisation and the like - whereas with NSIS a lot of the extra code that I had to write was related to the actual installation logic. Properly registering the uninstaller, getting the MSVC redistributable installed or even just proper handling of 64 bit systems (!) are all things that I had to take care of manually in NSIS. In fact, the entire uninstallation routine has to be specified manually, separate from the installation routine. Now I just tell WiX, here’s my application metadata, here’s my redistributable, this is the word size my program was built for, and these are the files, and then WiX makes sure everything ends up in the right place. And this isn’t even accounting for file type registration and the desktop icon stuff which I never got around to when I was still working with NSIS.

Did you consider an installer that might have required less work to maintain?

What specifically do you consider to be maintenance-intensive? Just the overall size? Or something in particular?

For what it’s worth, the NSIS work that I had before was not much different in size from the WiX project, it’s just that a lot of it was dealing with really basic stuff whereas with WiX it’s mainly on the cosmetic side (as mentioned above). In my experience, this actually makes WiX more pleasant to work with. What is admittedly a bit of a hassle right now is the manual build steps, if that’s what you mean. I’ve thought about writing a script or something for that, but didn’t do so in the end because I wanted to focus on the installer itself.

I agree that there are more exciting things to work on than packaging, but NSIS and InnoSetup aren’t perfect either. IMO the simplicity of those two systems can be deceptive, because it pushes complexity from the framework into the actual installer.

I guess it also depends on what you want out of an installer. For the (vast?) majority of our users, the installer will be the first thing they interact with after downloading the program, so I want that to be a somewhat decent experience. Of course, if you don’t see value in that it’s also possible to do a super barebones installer that does the absolute minimum to get the program installed – even in WiX. But personally, I’d rather put a little effort in to make it a decent experience. And even then, what I did here is nothing compared to the complexity of other real world installers such as those of GIMP and Firefox (which use InnoSetup and NSIS, respectively).

Actually, I think this is the most important thing I’d like to say with this comment – at the end of the day you can do simple installers in WiX and very complex ones in InnoSetup and NSIS. It depends more on what you’re trying to do than the tool you use.

why did you change software?

Well I guess I already answered this pretty much. Maybe it’s just because it’s been so long but to be honest, I struggle to think of substantial reasons to prefer NSIS over WiX in our case. IMO NSIS is easier to get started with, but WiX makes it easier to get a solid end result. “Requires much work to maintain” is hard to address because it’s relatively vague, but if there’s anything in particular that worries you I’ll see what can be done about it.

[MrStevns]

In which way? Code size? ....

Code size is one thing, but that's not my concern. No it's a combination of different things that could increase the friction when having to update this in the future.

• The steps involved to setup the installer
• That almost everything is written by hand, 11 files excluding icons and images that potentially requires changes between releases.
• WiX not being well documented, although it seems to still be somewhat preferred amongst windows developers still, so that might not be a real concern.

Maybe my expectations are just too high as for what an installer framework might do for you. If Windows developers are used to this kind of work, then I'm not one to argue against it, since I'm not one myself 😅

Of course, if you don’t see value in that it’s also possible to do a super barebones installer that does the absolute minimum to get the program installed – even in WiX. But personally, I’d rather put a little effort in to make it a decent experience. And even then, what I did here is nothing compared to the complexity of other real world installers such as those of GIMP and Firefox (which use InnoSetup and NSIS, respectively).

It's not so much that I don't see the value in making the installation process decent, because I do (I'm all in for it actually 😀 ), I'm just surprised by the work required to get a decent windows installation experience. When I read about windows installers, it feels like this field hasn't seen much if any innovation in decades.

Actually, I think this is the most important thing I’d like to say with this comment – at the end of the day you can do simple installers in WiX and very complex ones in InnoSetup and NSIS. It depends more on what you’re trying to do than the tool you use.

Fair point, I have no further comments on the subject, we can merge the PR 🎉

[J5lx]

• The steps involved to setup the installer

Fair, can’t argue with that. Perhaps I should come up with some sort of helper script after all… But that’s for another time.

• WiX not being well documented

I think part of that comes from the fact that WiX was originally little more than a build system for Windows Installer, so you’re kind of expected to be familiar with that. But yeah, the documentation could be better.

• That almost everything is written by hand

Also fair, though I feel like this is something that a lot of those frameworks have in common. Windows provides some installer functionality through Windows Installer, but even then you still kinda have to do the UI yourself. Frameworks will usually provide some sort of default UI, but as soon as you need any sort of custom functionality (such as the desktop icon stuff in our case), you’re back to rolling your own, often dealing directly or indirectly with the old win32 API (because you want your installer to work everywhere without pulling in an entire external GUI framework, and win32 is the lowest common denominator). The Firefox folks even ended up creating their own UI setup from scratch where they embed an Internet Explorer control (yes, Internet Explorer, because, again, lowest common denominator…) to load some HTML and JavaScript for the UI.

When I read about windows installers, it feels like this field hasn't seen much if any innovation in decades.

To be fair, I get the impression all of this becomes a lot more convenient in newer Windows versions that support “packaged” apps (i.e. AppX/MSIX and the app store stuff). It’s just that we agreed to keep our stuff working on Windows 7 (and scribble even suggested XP in the other thread), so we can’t use any of that newer OS-level tech. Until then, I believe the upcoming WiX 5 will at least include improvements to get rid of some of the boilerplate, so there’s that.

we can merge the PR

Wonderful – I want to go over the changes myself one more time in the next few days in case I notice anything new after not actively working on it for a while, then I’ll go ahead with the merge.

[J5lx]

Still need to set up the translations on Transifex, will do that later