What tools do you use to test Portability?

Discuss anything related to portable freeware here.
Message
Author
User avatar
JohnTHaller
Posts: 714
Joined: Wed Feb 10, 2010 4:44 pm
Location: New York, NY
Contact:

Re: What tools do you use to test Portability?

#31 Post by JohnTHaller »

For PortableApps.com, I use a series of clean virtual machines within VirtualBox. They're considered clean as they're a standard Windows install with just the defaults and all default Windows updates installed. The only added bit is the VirtualBox drivers to enable a fully functional testing environment. Virtual machines for Windows 2000, XP, Vista, 7, 8, 8.1, and 10 in both 32-bit and 64-bit variants are kept up to date on our main testing rig. For testing portability, I do a Regshot of the registry and all local files on C: before and after running an app. Within the app, I try to make sure all the main functionality is accessed, especially ensuring that options are changed and saved, any "views" are altered where applicable, and that at least one file is opened/saved. Options and views ensure we catch any apps that might split settings between INI and registry. File open/save ensures we catch GTK and Qt apps that might store registry entries you might not expect.
PortableApps.com - The open standard for portable software | Support Net Neutrality

CornHolio the Gringo
Posts: 23
Joined: Sat Jan 03, 2015 2:01 am

Re: What tools do you use to test Portability?

#32 Post by CornHolio the Gringo »

Heya JohnTHaller.
I love all the effort You give into portableapps.com.
I myself has always trusted Your portable sofware.
Its seems "clean" and the default settings are always set for Privacy.

Love You man...

User avatar
JohnTHaller
Posts: 714
Joined: Wed Feb 10, 2010 4:44 pm
Location: New York, NY
Contact:

Re: What tools do you use to test Portability?

#33 Post by JohnTHaller »

CornHolio the Gringo wrote:Heya JohnTHaller.
I love all the effort You give into portableapps.com.
I myself has always trusted Your portable sofware.
Its seems "clean" and the default settings are always set for Privacy.

Love You man...
Thanks. Glad you appreciate it. We try to present the best possible version of each app we package, even passing updated icons and bug fixes upstream. As for clean, we scan every app with ~40 antivirus engines, digitally sign, and confirm file hashes in the platform for a reason. :)
PortableApps.com - The open standard for portable software | Support Net Neutrality

User avatar
tomcat
Posts: 10
Joined: Sun Sep 14, 2014 9:24 am
Location: Europe
Contact:

Re: What tools do you use to test Portability?

#34 Post by tomcat »

Well, I have a naive query.

It seems to me that just to test whether an app is portable, you don’t need anything. Just double click it and see if (UAC and) an installer appears. If it does, it’s not portable and you can block the installer.
Alternatively and better still, try to run it from a USB stick, the definition of portability, and again see what happens.

The problem I see is that some portable programs, that is the non-stealth ones, still write to the registry despite being portable. And to test stealth you do need a registry monitor of some sort. But stealth is different from portability. Are we mixing the two?

So, what am I understanding wrong?

User avatar
webfork
Posts: 10818
Joined: Wed Apr 11, 2007 8:06 pm
Location: US, Texas
Contact:

Re: What tools do you use to test Portability?

#35 Post by webfork »

tomcat wrote: Fri Apr 05, 2019 8:00 am It seems to me that just to test whether an app is portable, you don’t need anything. Just double click it and see if (UAC and) an installer appears. If it does, it’s not portable and you can block the installer.
Alternatively and better still, try to run it from a USB stick, the definition of portability, and again see what happens.

The problem I see is that some portable programs, that is the non-stealth ones, still write to the registry despite being portable. And to test stealth you do need a registry monitor of some sort. But stealth is different from portability. Are we mixing the two?
I haven't done much with UAC so I'm not sure what system calls go into that or what that specifically indicates, but portability is fundamentally about a program's settings. If all settings are saved to the local folder, but a few registry writes for minor or unimportant or functions like window location, that's still considered acceptable.

As far as stealth (at least on our site), writing anywhere other than a few system temp folders means you're not stealth (more here).

There's a third category as well that I think you may have come across called "noinstall" which means there's no program installer (usually getting around the need for admin access), but can still save settings to various places on the drive. Those tend to work fine on a USB drive test but every machine you plug-in to will require modifying settings all over again.

User avatar
webfork
Posts: 10818
Joined: Wed Apr 11, 2007 8:06 pm
Location: US, Texas
Contact:

Re: What tools do you use to test Portability?

#36 Post by webfork »

JamieLanger wrote: Tue Apr 09, 2019 1:01 am Any application that can hold its configs and setups separately can be portable.
I'm not sure what you mean by "separately" but I haven't found settings portability that functions well outside of the application folder.

User avatar
Midas
Posts: 6705
Joined: Mon Dec 07, 2009 7:09 am
Location: Sol3

Re: What tools do you use to test Portability?

#37 Post by Midas »

Theoretically, with programs that rely on '-confdir' style command parameters, you could adopt a sidecar config folder scheme for storing settings while keeping portability -- similar to the way Jportable is usually set. I fail to see any major benefits to such setup, but it's a possibility.

bitcoin
Posts: 285
Joined: Sun Dec 31, 2017 6:32 pm

Re: What tools do you use to test Portability?

#38 Post by bitcoin »

tomcat wrote: Fri Apr 05, 2019 8:00 am Well, I have a naive query.

It seems to me that just to test whether an app is portable, you don’t need anything. Just double click it and see if (UAC and) an installer appears. If it does, it’s not portable and you can block the installer.
Alternatively and better still, try to run it from a USB stick, the definition of portability, and again see what happens.

The problem I see is that some portable programs, that is the non-stealth ones, still write to the registry despite being portable. And to test stealth you do need a registry monitor of some sort. But stealth is different from portability. Are we mixing the two?

So, what am I understanding wrong?
some of the programs that claim to be portable or appear to be portable (no installer) create folders for program settings at places like Win7>Users>AppData and leave them there

so if for example you use your USB drive with portable programs on someone else's computer (maybe you are a technician or just borrow a friend's computer) its leaving that junk there

appsuser
Posts: 136
Joined: Fri Feb 08, 2008 11:51 pm

Re: What tools do you use to test Portability?

#39 Post by appsuser »

Wonder if anyone has tried the new Windows Sandbox feature of Windows 10 (Windows 10 Pro or Enterprise only) to test for Portability?

https://techcommunity.microsoft.com/t5/ ... a-p/301849

User avatar
webfork
Posts: 10818
Joined: Wed Apr 11, 2007 8:06 pm
Location: US, Texas
Contact:

Re: What tools do you use to test Portability?

#40 Post by webfork »

appsuser wrote: Mon Nov 25, 2019 8:02 am Wonder if anyone has tried the new Windows Sandbox feature of Windows 10 (Windows 10 Pro or Enterprise only) to test for Portability?

https://techcommunity.microsoft.com/t5/ ... a-p/301849
As Microsoft has a tendency to flirt with given features and then pull them later, I've focused on using tools and programs outside their sphere. That said, if I hear that it's a better/faster/smarter tool than my current system, I'll certainly risk it.

User avatar
vevy
Posts: 795
Joined: Tue Sep 10, 2019 11:17 am

Re: What tools do you use to test Portability?

#41 Post by vevy »


User avatar
webfork
Posts: 10818
Joined: Wed Apr 11, 2007 8:06 pm
Location: US, Texas
Contact:

Re: What tools do you use to test Portability?

#42 Post by webfork »

vevy wrote: Sat Oct 31, 2020 11:34 am Potentially useful:

https://sourceforge.net/projects/filegrab/
Yeah definitely ... great suggestion.
FileGrab is a tool that monitors a Windows filesystem for newly created files and copy those files to another location. It can be useful for honeypots, malware analysis, investigation scenarios and so on.

Features

Run in background (hidden Window)
Send files to a FTP server
Monitors all filesystems or specified path
Regular expressions support

User avatar
webfork
Posts: 10818
Joined: Wed Apr 11, 2007 8:06 pm
Location: US, Texas
Contact:

Re: What tools do you use to test Portability?

#43 Post by webfork »

An update on my portable testing process that is a very long time in the making. I wanted to balance my efforts between something accessible, secure, and effective. The process below is a look at what I've put together, but if you want something a bit more involved or comprehensive, Midas posted about a more involved testing system you might prefer.

This post could also function with some tweaks as an introduction to the Windows Sandbox, Windows registry, and bundleware/malware analysis.

---

Before you begin

Testing rig: Multi-core with a fast hard drive. While most testing setups benefit from a faster computer, the process below relies primarily on hard drive speed. I recommend a machine with at least 2 cores and 2 (most modern i5 processors have this). Benchmarking note: The process below was using a 3-year old Intel i7 with an SSD, and was able to generate a system snapshot in about 2 minutes. Aside from a slight interface hesitation, I really couldn't tell the difference between the Sandbox and my regular computer.

Windows Sandbox (first mentioned here) - by some accounts this is limited to Windows 10 Pro but I'm not 100% about that. Why Windows Sandbox? It has the advantage of being wonderfully simple, but misses some of the advantages of virtualization, including an overall better environment for testing (as you'll see below and if you look around in the forums, sandboxes like this one reject some software functions).
Steps:
  1. Gather your testing software and post it to a folder. Really the only System Explorer is necessary, but the other recommended tools below reflect a lot of conversations on the site over time, most earlier in this thread. A more detailed breakdown on these other functions can be given if anyone is interested.
    • Before and after system snapshots: System Explorer – to track system activity within the Windows instance
    • File analysis / hash / VT /etc.: - SigcheckGUI
    • Find net connections (phone home): simplewall (with all filters enabled so that you get prompted for any connection)
    • File extraction - Universal Extractor 1 and Universal Extractor 2
    • Dependency checks (e.g. Java, Visual C++ Redistributable): TBA, was PeStudio but that gives me a "wait ..." error for many of the program elements.
    • Run as limited user - the “runas” command (discussed here) to test whether or not admin is required or even necessary (it’s good security to always run a program with the fewest system rights)
  2. Start up the Windows Sandbox instance and copy over all files from your prepared folder. There's a shared clipboard for files and other content between your regular computer and the sandbox so it really is just CTRL+C -> CTRL+V to transfer files.
  3. Run System Explorer and press the + in the top-right corner to add the "snapshots" tab

    System Explorer Snapshots tab
    Image

    _
  4. Create a Snapshot with Files+Registry

    System Explorer snapshot creation
    Image

    _
  5. Run the program in question, change a few settings to make sure you trigger an edit to the settings file or entry
  6. Close and create another snapshot in System Explorer
  7. Select both snapshots and select "Compare snapshots"


    Issues

    Limited system functionality - two programs I tested had issues because they interact with basic windows function like the mouse and antivirus. One was EitherMouse, which lets you operate two mice on the same computer, and the other was ConfigureDefender, which modifies antivirus settings. While EitherMouse functioned in a limited way, ConfigureDefender would not start.

    Note that simplewall worked fine on the Windows Sandbox, so this may have been a unique incompatibility with ConfigureDefender.

    Results

    EitherMouse details showing it's modifications to the registry for both mouse settings and the program itself

    System Explorer test results for EitherMouse
    Image
    _
    Other notes

    As noted elsewhere, once you close the Windows Sandbox instance, you lose all data so make sure to copy over anything you want to keep to the host machine (files and text).


    Related: Intro to file posting accessories
Last edited by webfork on Sat Jan 02, 2021 1:44 pm, edited 4 times in total.
Reason: tons of minor edits, image fixes, and Defender is AV not Firewall

User avatar
Midas
Posts: 6705
Joined: Mon Dec 07, 2009 7:09 am
Location: Sol3

Re: What tools do you use to test Portability?

#44 Post by Midas »

Great tutorial, thanks. 8)

This kind of knowledge gathering has to be hands down the best illustration of the virtues or our little community.

User avatar
webfork
Posts: 10818
Joined: Wed Apr 11, 2007 8:06 pm
Location: US, Texas
Contact:

Re: What tools do you use to test Portability?

#45 Post by webfork »

Midas wrote: Sat Jan 02, 2021 6:31 am Great tutorial, thanks. 8)

This kind of knowledge gathering has to be hands down the best illustration of the virtues or our little community.
Thanks -- and this is definitely not my creation, rather a new take on a bunch of tweaks on a lot of other work by other users over the years. The simplicity of Windows Sandbox really brought it all together.

Post Reply