CLI Database Discussions

Discuss anything related to command line tools here.
Post Reply
Message
Author
vevy
Posts: 678
Joined: Tue Sep 10, 2019 11:17 am

CLI Database Discussions

#1 Post by vevy » Mon Jan 27, 2020 2:46 pm

[Mod note: access details for cli.portablefreeware.com can be found at the following post: viewtopic.php?p=95530#p95530]


NOTE: This post refers to and uses elements from CLI Discussions Resources.


Current priorities:
  • Mini-requests
  • #23: Developer field.
  • #43 to #49: Additional statuses.

Mini-requests:
  • If an icon file is chosen, there is no way to remove it during adding or editing. Same for screenshot? (Andrew: KIV)
  • [_size] tag should also apply to line spacing. (Andrew: KIV)
  • Links to fix (usually "cli." is unduly added to them): (Andrew: Done)
    • ✅ Edit entry: "Portable Freeware Update" Link.
    • DB search: "Forum topics matching your query"
    • ✅ Entry Comments: user profile links.
  • ✅ Minor correction to span.use-case: padding: 0px 4px 2px 4px; (Andrew: Fixed)
  • ✅ In edit page, "Beta releases should be posted to..." message should be removed for CLI. (Andrew: Fixed)
  • ✅ In All use cases page, hover text (title) is just the first letter of the use case. (Andrew: Fixed)
  • ✅ Use cases assignment isn't reflected in all tools. For example, click any of the use cases under curl or Wget. (Andrew: Fixed)
-------------------------------------------------------------------------
Priority: 1=min, 3=max, p=postponed

IDRequestDescriptionPriorityAndrew
General Requests
#01 Clearer edit logs Currently, entry log history is diff-ed by characters, which makes figuring the changes out very difficult. Better do it by filed or paragraph. 2 KIV
#02 Adapt main page links to CLI. Like "About", "Useful Links", etc 2p Updated to point to main site.
#05 Promote the best tools. Make a system for promoting the better tools (Editor's pick, voting, etc) ? KIV. The star system (currently only activated for main site) is scored based on internal and external voting.
#06 Sorting by last edited Sorting entries by last entry update date (as logged). 2 KIV
#79 Show number of tools in top bar. Currently, I have to do an empty search to get how many tools are there. 1
Search
#19 One-word synonyms Handle one-word synonyms: like audio=soundmusic. 1p
#20 Handle aliases Use cases like overlay video on another=picture in picture effect and formats like jpg=jpeg. 1-2p
#21 Partial matches. "implement" should also match "implementation". 2-3p
#22 Search terms separately i.e. not necessarily in order 2p
#81 Prioritize title matches Tools with {search_term} in their title should be prioritized. 3
#82 Prioritize use case matches Tools with {search_term} in their use cases should be prioritized. 2-3
#83 Prioritize recent tools Recently-updated tools should be prioritized. 2
#66 Sources/Developers table All the sources sifted for Windows CLI tools.
These could be developer sites, Github pages, directories, etc.
This allows us to know what was checked and when, and whether it needs to be checked regularly.
2
#67 Index Table Could be dynamically linked to the database or static.
CTRL+F for whether a tool exists in the database and its basic info.
1-2p
Field Requests
General
#50 Parent>Child detail inheritance System to allow child entries to get details from parent tool (collection) ✅
#64 Implement filters as text first. Especially for maintenance filters. e.g. Instead of a checkbox for "Child tool", append "child:1" to search.
mp3 license:foss child:1 extract:"."
Easier to implement (?), less UI clutter.
KIV
#80 Reverse Boolean Filters. e.g. os=off for tools that are NOT open source. 2
#54 Templates as unexpanded tags Going with your suggestion about [warning] tags, and in the same spirit of unifying template wording and facilitating global changes, add templates in general as tags.

e.g. [extraction_see_parent] in #53 and expand automatically.

There are ways to expose the wording to the contributor if you wish to discuss.
2p
#18 Field incomplete message If a filterable field is optional, show a message atop the "filter results page" that says: "Not all tools contain this type of information. Results may be incomplete." KIV
Title
#55 Inherited Possible use: Automatically include parent name between brackets touch [Yori]. 1p
#35 Software title: Suggestions To avoid duplicate entries. Make suggestions clickable into a new tab. ✅
Categories
#10 Filterable Useful with large categories when combined with other filters. 1-2 KIV
Website URL
#56 Always inherited. ✅
Size
#12 Filterable -- 1 KIV
#57 Conditionally inherited (if child field is empty) ✅
License
#07 Filterable e.g. only FOSS results. ✅
#58 Conditionally inherited ✅
#78 Split license types into radio button groups Public-domain/Open Source (PD, GPL, BSD, etc).
Free.
Conditionally Free (Free personal, etc)
Limited Free (Freemium, Liteware/Demo, etc.)
TBC
?
Description
Writes settings to
How to extract
Version
#68 "v" in Version field Insert automatically in entry view? 1p
Release date
#08 Filterable For example, tools released on or after 2018-01-01 1-2 KIV
#61 Release date Always inherited. ✅
Download URL
Runs on
#13 Filterable Postponed till such data is entered satisfactorily. 1p KIV
#59 Conditionally inherited ✅
Icon
#72 Accept .ico 1
#60 Conditionally inherited ✅ Done
#04 Stock icon for TUI tools. Image Free for commercial. Only attribution. 1 KIV
Screenshot
Additional status
#36 Move Additional statuses Next to the title. This should better help clarity and will save space. ✅ Done
#14 Filterable Tool, package, Win. package, discontinued, etc. 2 KIV
#17 Clickable badges Make badges (additional status/warning) clickable (to a page filtered by these badges e.g. Win packages only). This will have the side benefit of making categories like "Collection" and "Unix Ports" unneeded. 1-2 KIV
#85 Split Badges from Additional status into 3 groups:
1. Tool level
2. Tool type
3. Tool origin
2
Tool level
#38 Collection Image ✅
#37 Package Image ✅
#40 Tool Image ✅
#46 Subcommand (Internal Command)
Image
2-3
Tool type
#86 CLI Assumed by default. No badge. =
#43 CLI-mode GUI
Image
2-3
#44 TUI
Image
2-3
#45 Batch
Image
2-3
#48 Alternative Shell
Image
2-3
#49 Shell Extension
Image
2-3
Tool origin
#87 Native Windows Assumed by default. No badge. =
#41 Unix Port Image ✅
#47 Windows-Included
Image
2-3
Additional Status
#88 Active Assumed by default. No badge. ==
#42 Discontinued ✅
#89 Discontinued
Image
1-2
Dependencies
#15 Filterable Needs the field to have a Boolean option of Y/N first (like Stealth). Needs the information to be entered first. 1p KIV
Stealth
#16 Filterable Needs the data first. 1p KIV
Suggested By
#69 Conditionally inherited 2
Parent ID
#65 Suggest parent name Upon entering parent ID (to confirm correct ID). 1-2
#63 Filter by IsChild Useful for maintenance. KIV
Use Cases
#11 Filterable -- 2p KIV
#84 Show number of tools between brackets e.g. search text (15) 1-2
Developer/Author
#73 Create (Optional) 2
#23 Suggestions Show suggestion like "Title" field, but clicking a suggestion enters it into the field (not disabled like "Title").
#09 Filterable All tools by a given developers. Useful for maintenance and usage. 1-2 KIV
#62 Conditionally inherited? p (implement field first).
Download Link (Archive)
#70 Create (Optional) Archive.org or so in case dev site goes offline. 2
Documentation Link
#24 Create (Optional) -- 1p
Documentation Text
#25 Create (Optional) Collapsed by default. 1p
Warnings
#74 Create (Optional) Boolean. As text or badge with hover text.
#75 Filterable
#26 IsWarningNet 1-2
#27 IsWarningSecurity 1-2
#28 IsWarningDecompression 1-2
#29 IsNotArchiver Useful because of the difference in behavior with compression tools in GUI vs CLI. 1
#30 IsNotCompressor Useful because of the difference in behavior with compression tools in GUI vs CLI. 1
Blurb
#31 Create (Optional) One-liner like GNU blurbs, Debian package desc. 1
IsInIndex
#32 Create. Boolean, optional. Added to the CLI Tool Index or not. Useful for managing the additions of tools. 1-2p
Available in/Part of/Get it from/Parent Entry
#33 Create. Optional. Still WIP.
Enter delimited IDs of parent tools, e.g. 6,9,11 Will put a field with sentence: Available in UnxUtils, GnuWin, Gow.
2p
Notes
#34 Create (Optional) Edit box. Not shown in entry view.
GUI Front-ends
#51 Create (Optional) Edit box. Link or just name the front-ends available for a tool. 1-2
Noob-friendliness
#76 Create (Optional) 3-level listbox (Easy, intermediate, Hard). Average votes. 1-2
Changelog Link
#77 Create (Optional) Many tools have buried changelogs (NEWS, CHANGES, CHANGELOG, Release Notes, etc) 1-2


------------------------------------------------------------------


Original Post:
Spoiler!   
Hi, everyone

I know Andrew is not big on CLI tools, but they are very useful. I keep discovering new tools and some of them are jaw-dropping in functionality and keep wondering: what else is there that I could be missing?

There are many sites that list CLI software among GUI software. Alternativeto.net has tags that help separate such tools, but their database doesn't have that many.

Also, some software authors have many capable CLI tools on their sites.

What I mean is, they are all-over the place and deserve a separate database.

Is there something like that out there? Eager to here your recommendations (and ideas) :)
Last edited by vevy on Wed Nov 04, 2020 11:19 pm, edited 116 times in total.
"Is there a Windows-included tool for this task?"
"I only want open-source tools"
"I want a tool that is still actively developed"
"So many to choose from!"
and many more!
Support easy-to-do filters and badges!

User avatar
SYSTEM
Posts: 1954
Joined: Sat Jul 31, 2010 1:19 am
Location: Helsinki, Finland

Re: Is there a public directory/repo for command line tools?

#2 Post by SYSTEM » Mon Jan 27, 2020 8:10 pm

https://tinyapps.org/ isn't exclusively for command-line apps, but a large portion of their catalog is command-line.
My YouTube channel | Release date of my 13th playlist: August 24, 2020

vevy
Posts: 678
Joined: Tue Sep 10, 2019 11:17 am

Re: Is there a public directory/repo for command line tools?

#3 Post by vevy » Mon Jan 27, 2020 8:57 pm

Thanks, System. Nice site, but, by my quick look, it doesn't seem to have many tools (for example ptime, rxrepl, busybox, xpdf, etc) and old versions of others.

Any other suggestions are appreciated. :)
"Is there a Windows-included tool for this task?"
"I only want open-source tools"
"I want a tool that is still actively developed"
"So many to choose from!"
and many more!
Support easy-to-do filters and badges!


vevy
Posts: 678
Joined: Tue Sep 10, 2019 11:17 am

Re: Is there a public directory/repo for command line tools?

#5 Post by vevy » Tue Jan 28, 2020 2:16 pm

billon wrote:
Tue Jan 28, 2020 12:05 pm
https://www.robvanderwoude.com/
Good find. It has some nice tools and references many other authors. It says though:
This list doesn't even attempt to be complete, as there are literally thousands and thousands of batch file utilities available on the web.

The limited space for the descriptions of these tools doesn't always do them justice, so check out the authors' web sites for more details.
I propose we make a (separate) database here for CLI tools. Current database functions as usual while the other serves as a comprehensive directory for all these good tools that risk going into obscurity.

Andrew, you just create a new database for us and we will populate it :) . Little work is needed on your part. I don't suppose that is too much to ask? Just link it at the main site's top bar (Next to Latest, Categories) and we will do the work.

What do our veterans think? Is it worth a shot?
"Is there a Windows-included tool for this task?"
"I only want open-source tools"
"I want a tool that is still actively developed"
"So many to choose from!"
and many more!
Support easy-to-do filters and badges!

TP109
Posts: 561
Joined: Sat Apr 08, 2006 7:12 pm
Location: Midwestern US

Re: Public directories/repos for command line tools (and suggestion for a secondary Portable Freeware database)

#6 Post by TP109 » Tue Jan 28, 2020 4:57 pm

A search for command line tools in the TPFC Forums brought up 208 pages of results, so command line tools have been an ongoing discussion for awhile. At one time, there was an entry in the FAQs titled, "Why don't you accept command-line apps?", but it's no longer listed on the FAQ page .

The thread below was dedicated to discussing command line tools in 2016.
TPFC Forums - Great CLI Programs

The page below has a list of where to find some command line tools.
https://cects.com/find-win32-cli-utilities/
vevy wrote:
Tue Jan 28, 2020 2:16 pm
I propose we make a (separate) database here for CLI tools. Current database functions as usual while the other serves as a comprehensive directory for all these good tools that risk going into obscurity.
I use command line tools on a daily basis and a database for command line tools would be interesting and useful, but I'm not sure if it would be feasible or appropriate for TPFC to take on that task. I recall that possibility being discussed before, but I haven't located the exact thread.

Specular
Posts: 422
Joined: Sun Feb 16, 2014 10:54 pm

Re: Public directories/repos for command line tools (and suggestion for a secondary Portable Freeware database)

#7 Post by Specular » Tue Jan 28, 2020 7:24 pm

I think the idea of an index of CLI programs is an interesting one but I also wonder how that would gel with carrying over the existing nature of the PFDB, in that most programs, descriptions and instructions are written and intended for a more general user audience, while CLI tools each have a learning curve of their own and I wonder what such a database would look like if carrying over PFDB principles.

Even things as simple as entry descriptions for example over time are often edited to simplify the verbiage yet with CLI programs it necessitates a better understanding of the technical details of the program, any additional dependencies (some of these are moving targets when changes occur for example), 'easy to use' would also need be judged differently and generally users be expected to be more familiar with command-line use and its limitations (which can trip up even experienced users, in areas like path escaping).

I think PFDB has a pretty clued-in, poweruser-leaning audience based on comments and thread postings but I'd be curious how entries and expectations would be handled.

Edit: and I'd also add that it's uncertain how many would be willing to maintain updates to such a DB, given for the audience we have it's a more niche area and keeping track of many updates is something that tends to be picked up by a smaller handful of users and sporadic contributors. Proposing something is obviously simpler than maintaining it.

User avatar
webfork
Posts: 9523
Joined: Wed Apr 11, 2007 8:06 pm
Location: US, Texas
Contact:

Re: Is there a public directory/repo for command line tools?

#8 Post by webfork » Tue Jan 28, 2020 8:17 pm

vevy wrote:
Tue Jan 28, 2020 2:16 pm
What do our veterans think? Is it worth a shot?
Some input off the top of my head: the truth is that most CLI tools are portable, so it could make sense to include here. There's some amazing CLI software that could definitely benefit from some time and attention. Still, there are some concerns:
  • Advocates - Ultimately a lot of good software arrived because it had a champion. I can definitely get behind great CLI tools like 7zip's command-line tools, SFK, and PDFBox. All those are amazing programs, but I don't really have a toolset beyond that.
  • Maintenance - We have a group that's been here for a long time that helps keep entries up to date, works on descriptions, and generally tries to make sure that we find mirrors for dead software. A software listing without a fairly dedicated crew probably will be a flash in the pan.
  • Platforms - Does this include non-Windows CLI tools?
  • Dependencies - CLI software does have the same problems that regular software has, including dependencies (cygwin and Java) and system configuration (PowerShell vs. CMD) and all of that would definitely need to be part of any comprehensive site.
Whatever the case, it's a great idea I hope someone can execute on.

User avatar
Andrew Lee
Posts: 2507
Joined: Sat Feb 04, 2006 9:19 am
Contact:

Re: Public directories/repos for command line tools (and suggestion for a secondary Portable Freeware database)

#9 Post by Andrew Lee » Wed Jan 29, 2020 1:57 am

At one time, there was an entry in the FAQs titled, "Why don't you accept command-line apps?", but it's no longer listed on the FAQ page.
As the database evolved into being managed by the community (via voting), at one point I think we decided that the good "taste" of the community will take care of this.

I remember we used to have a lot of debates not only about CLI, but also .Net, Java, etc apps (Electron-based apps were still a rarity then). Now there are a sprinkle of such apps in the database on a case-by-case basis.

I kinda like this more than the black-and-white approach, much like how I didn't agree with the hardline view that only stealth will do for portability. Life is more interesting this way. :D

Personally, I still think we should have a damn good reason for including a CLI app, since these are mostly portable by default, with very few exceptions (and those exceptions are IMHO plain evil :evil:).

User avatar
Midas
Posts: 5526
Joined: Mon Dec 07, 2009 7:09 am
Location: Sol3

Re: Public directories/repos for command line tools (and suggestion for a secondary Portable Freeware database)

#10 Post by Midas » Wed Jan 29, 2020 4:57 am

Andrew Lee wrote: I kinda like this more than the black-and-white approach, much like how I didn't agree with the hardline view that only stealth will do for portability. Life is more interesting this way.

I see what you have done there... And although I am a fan of (some) CLI tools -- FFmpeg and Youtube-dl immediatly spring to my mind from having used them in the last few hours -- I wholeheartedly agree. Principles are good, intelligent discretion is even better. 8)

Apart from those, I'd recommend anyone interested in CLI tools to start by checking the following projects:

vevy
Posts: 678
Joined: Tue Sep 10, 2019 11:17 am

Re: Public directories/repos for command line tools (and suggestion for a secondary Portable Freeware database)

#11 Post by vevy » Wed Jan 29, 2020 1:32 pm

Good discussion. Here are my thoughts and responses (TP109, Specular, webfork, Andrew):
Also, Thanks for the links, TP109 and Midas. :wink:
  • Windows CLI tools. Unix users have it made when it comes to CLI tools and repos. :)
  • Separate Database. Rationale: Keep the purpose and scope of the main database untouched while also not be constrained by its exact paradigm. Databases with mixed CLI and GUI apps are already there on many bigger sites like Softpedia; so there is no need to repeat that.
  • Less maintenance. CLI tools often reach maturity much faster than GUI tools due to their relative simplicity. Many (Most ?) CLI tools are not updated that often or at all because they still function well even after many years and OS versions.
  • Effort is cumulative. Even if it becomes a "flash in the pan" effort, the work done is not lost and can serve a base for future improvements. Also many entries won't need updating as in the previous point.
  • The audience is there and need is there. This site is known among "power users" and is well primed to unify the fragmented efforts like the sites mentioned before.
  • Some sites and tools are going offline. f2ko.de is not accessible anymore and many good but unmaintained tools on google sites, private sites, etc won't be there forever. There is a need to collect them and provide archive.org links for example.
  • There is a place for native, ported and platform-dependent (Cygwin, Java) tools. Dependencies should be mentioned when appropriate. This is already the case in the regular database (although I think most tools don't have real dependency issues.)
Last edited by vevy on Wed Jan 29, 2020 2:07 pm, edited 2 times in total.
"Is there a Windows-included tool for this task?"
"I only want open-source tools"
"I want a tool that is still actively developed"
"So many to choose from!"
and many more!
Support easy-to-do filters and badges!

vevy
Posts: 678
Joined: Tue Sep 10, 2019 11:17 am

Re: Public directories/repos for command line tools (and suggestion for a secondary Portable Freeware database)

#12 Post by vevy » Wed Jan 29, 2020 2:03 pm

More thoughts:
  • I envision a category for "Collection" CLI tools. Things like Cygwin, Busybox, Swiss File Knife, etc that provide many tools/function under one umbrella. Maybe even split them into "Platforms" like Cygwin, "Collections" like UnxUtils, GnuWin32, Busybox, Sysinternals and multi-function tools like Swiss File Knife and Yori(?)
  • Maybe we should use tags more in the new database (like 2ry categories). For example, Xpdf may have tags like "convert", "pdf-to-text", "pdf-to-html". Such function tags would help the users find alternatives for a given task. This is a lot of work though. Still, lets lay the foundation since we are planning. :D
  • Should we add apps that have a CLI component like Inkscape and Handbrake, or should we just refer to their regular database entry, or ignore them altogether?
"Is there a Windows-included tool for this task?"
"I only want open-source tools"
"I want a tool that is still actively developed"
"So many to choose from!"
and many more!
Support easy-to-do filters and badges!

Specular
Posts: 422
Joined: Sun Feb 16, 2014 10:54 pm

Re: Public directories/repos for command line tools (and suggestion for a secondary Portable Freeware database)

#13 Post by Specular » Wed Jan 29, 2020 10:59 pm

vevy wrote:
Wed Jan 29, 2020 1:32 pm
  • Less maintenance. CLI tools often reach maturity much faster than GUI tools due to their relative simplicity. Many (Most ?) CLI tools are not updated that often or at all because they still function well even after many years and OS versions.
I'd say it's a bit more of a mix of update frequency, not unlike GUI programs. There are definitely some that go for long stretches without updates though, particularly for niche use. Some of my most used CLI programs are updated very frequently: ffmpeg (updated daily), youtube-dl (frequently), ImageMagick (every few days).

That said I don't tend to update them myself as regularly as updates are available but it'd become a pretty continuous task for whoever was editing such entries.

One other thing is who would be voting here on such programs? As there are a variety of useful GUI programs that at one point or another didn't receive enough votes and I wonder how CLI programs would fare with this userbase. I know popular programs such as those listed so far would likely be voted upon but I wonder how others that may be popular among a certain audience but not necessarily here would be received. Not sure anyone could determine that beforehand necessarily though :P
vevy wrote:
Wed Jan 29, 2020 2:03 pm
  • Should we add apps that have a CLI component like Inkscape and Handbrake, or should we just refer to their regular database entry, or ignore them altogether?
This is a good question. Depends how the entries are written maybe, whether it's just a very similar description of the GUI entry with a CLI screenshot or if entries are envisioned to have more about the CLI specific aspects, like perhaps brief mention about syntax or similar (or whether that aspect would be assumed the user would have to go and find in some documentation, which might only exist in a /help command).
Last edited by Specular on Fri Feb 07, 2020 3:24 am, edited 1 time in total.

User avatar
Midas
Posts: 5526
Joined: Mon Dec 07, 2009 7:09 am
Location: Sol3

Re: Public directories/repos for command line tools (and suggestion for a secondary Portable Freeware database)

#14 Post by Midas » Thu Jan 30, 2020 4:57 am

:idea: Instead of imposing on Andrew Lee the burden of running yet another site for CLI tools, let me suggest here that someone with a little web savvy could setup a Github repo modeled after TPFC database for the effect.

E.g., Scoop (https://scoop.sh/) seems to have had some success with this approach...

vevy
Posts: 678
Joined: Tue Sep 10, 2019 11:17 am

Re: Public directories/repos for command line tools (and suggestion for a secondary Portable Freeware database)

#15 Post by vevy » Thu Jan 30, 2020 11:47 am

Specular wrote:
Wed Jan 29, 2020 10:59 pm
I'd say it's a bit more of a mix of update frequency, not unlike GUI programs. There are definitely some that go for long stretches without updates though, particularly for niche use. Some of my most used CLI programs are updated very frequently: ffmpeg (updated daily), youtube-dl (almost daily), ImageMagick (every few days).
True. I have a couple of thoughts:
  • There is a lot of value in the listing itself, regardless of keeping up with updates: to make the tool itself discoverable to the user.
  • We can just follow stable updates (every few weeks/months for ffmpeg for example.
  • We can simply link to the download page, rather than specific files, so that the user will always be getting the latest update. Again there is value in creating a database of the tools themselves rather than scrambling after version updates.

    This is one of the reasons I am suggesting a new database not bound by the exact rules of the main one. :)
Specular wrote:
Wed Jan 29, 2020 10:59 pm
One other thing is who would be voting here on such programs?
Well, it is up to the boss, but I would suggest easing up or removing this restriction at first while filling up the database. Let's say:
  • new tools are added without needing votes.
  • if an app gets two downvotes (because it doesn't work properly for example), it becomes private until further notice.
"Is there a Windows-included tool for this task?"
"I only want open-source tools"
"I want a tool that is still actively developed"
"So many to choose from!"
and many more!
Support easy-to-do filters and badges!

Post Reply