New suggestion for CLI database - Roast me!

[Andrew Lee]

An idea came to me last night that I just could not shake off.. so I need you guys to knock some sense into me.

With all the discussions and arguments we had over the CLI database these past months, it has always been at the back of my mind to figure out what's the best way going forward. A nagging feeling I've always had is the current mainline database format is not very suitable for CLI apps for so many reasons that I shall not detail here.

A long time ago, I made a little app called Media Cookbook that never really attracted much attention. However, I have found it useful enough to be using it even today (I also add use doskey to automate common CLI tasks). But I have always been keenly aware of its inadequacies (eg. too much work to add minor variations of a task), but I find a recipe-based approach is still very attractive for me.

Then last evening it hit me. I think what is needed for the CLI site is not an app-centric database, but a recipe-centric database. Let me explain.

A recipe is simply a short description of what you are trying to achieve, and the CLI for doing it. So for example:

Add subtitle to MP4 ffmpeg -i <infile.mp4> -i <infile.srt> -c copy -c:s mov_text <outfile.mp4> Remove execution permission for all files in given folder icacls <folder> /deny <User>:(CI)(OI)(X) /t Replace text in file sed -i 's/<oldtext>/<newtext>/g' <file.txt> -o <file.txt> awk '{sub(/2019/,2020); print . "<file.txt>" }' file.txt Download YouTube video youtube-dl "<video-url>

The description/title is sorta like a free-text version of vevy's use case. Under each title are multiple community-added recipes, which can be public/hidden based on our existing voting system.

Note that "ffmpeg" in the first recipe is optionally linked to the version I used successfully for my recipe. The system can automatically compile a list of all the ffmpeg's in the system and show the full list with an optional button, so this solves the problem of curating different versions of a CLI tool.

The search function will return a list of matching titles, so searching for "youtube" or "video" will return "Download YouTube video" and all matching titles/recipes. So surfacing the right recipes is pretty straightforward and could be made as targeted as you want.

For contribution, one can add a new CLI for the recipe eg. "Add subtitle to MP4", so the latter can have multiple CLIs for performing that function, which can then be voted upon. Or one can create a new recipe altogether if none of the existing ones match the new function.

Note that a recipe can mash together multiple CLIs, or could even be a Powershell script. The only constraint is the target function should be as single-purpose as possible (I believe the technical term is "orthogonal") , so no "I wrote this script that can juggle 4 balls, eat lunch and perform squats at the same time".

The database does not need to be comprehensive. The key idea is having problems that are worth solving, the CLIs to solve them, and the right args to use. CLIs are different in this regard to GUIs. Most of the time, you need to mash a number of different args from different recipes to do what you want.

The major difference of this approach is that the focus now is _not_ on new versions of CLI tools, but new recipes. A recipe can indicate that it only works with > certain version because of a new arg etc. The front page will list new and updated recipes instead of new versions of CLIs.

OK, that's about all that I have at the moment. Roast my idea pls!

[Specular]

Under each title are multiple community-added recipes, which can be public/hidden based on our existing voting system.

For contribution, one can add a new CLI for the recipe eg. "Add subtitle to MP4", so the latter can have multiple CLIs for performing that function, which can then be voted upon. Or one can create a new recipe altogether if none of the existing ones match the new function.

Note that a recipe can mash together multiple CLIs, or could even be a Powershell script. The only constraint is the target function should be as single-purpose as possible...

The database does not need to be comprehensive. The key idea is having problems that are worth solving, the CLIs to solve them, and the right args to use.

It's an interesting idea. In an ideal scenario it would also force users submitting to consider what problems are useful solving, particularly if they've encountered them before and the programs/commands used to achieve it. A few sites I've seen now and then posted to Hacker News focus on useful command line one-liners (though your idea seems to be about focusing on the problem to be solved first than solely the methods to achieve it), but I got the sense those sites are more Linux-centric. I remember a few of us talked here a year or more ago about keeping notes on often-used CLI one liners and your 'recipe' term.

Two things come to mind. One would be that for any submission there could be a smaller and streamlined/longer but more robust/etc version of commands for a recipe. How would alternative versions of commands be handled, would they all be shown under the recipe (problem to be solved) entry? I suppose alternatives could be debated in the comments for such recipes though that wouldn't be ideal for longer forms or for attachments like script files (but OTOH using the forums for discussions per-recipe could get messy with the multitude of potential combinations - edit: or not in reality? Idk, would depend how it went ).

The other is given that with virtually any CLI program's basic arguments one could create a recipe what do you feel would direct users to only post 'problems that are worth solving', since I'd imagine with such constraints some users (*cough* from the comprehensive camp *cough* ) may want to post all default args under new recipes (since it could be argued that anything is worth solving to someone)... which functionally would be not dissimilar to the recent debate over Busybox's functions as separate entries, where such an alternative DB could be potentially filled with pages of recipes for the sake of covering everything.

I mean, there's voting but to actually give such recipes enough visibility to be voted upon in the first place they'd have to be visible for enough time so I just wonder how anything broader in the 'usefulness' metric of recipe submission would be handled or dissuaded (though potentially it may be less of an issue or not one at all if searching/filtering/browsing were based around problems to be solved, I think I'd be interested to see what such a layout would look like).

[thepiney]

I really like the example for "icacls" linking to the site with a description, syntax and parameters so it's easier/faster to fine tune the recipe or clarify what the arguments mean.

Also, your note about linking to the version that was successfully used is a great idea. Would a "note" at the end of the "recipe" be added as well? Could have something about why that version was used, ie. new arguments as you mentioned and/or possible issues with some versions that may cause corrupted outputs or such?

Could the database be setup to work with either method or would that be too much extra effort?

As some recipes are more or less detailed, I could see it working with one-liner recipes as Specular mentioned or multiple-line recipes for some. The idea of scripts really hits that point well. I met an IT manager a few years ago that had a "recipe" for --> "How do I get paid for doing a job while I play online games at work?"... He automated his entire work day with several CLI batch files and scripts that he ran using Windows Task Scheduler.

You could also add links to other recipes as well --> "Download YouTube video" could have a link to say "Convert video to different format" or "Add subtitle to MP4" although that could get to be a lot of links added to the entry.

All in all, I like the idea a lot. It would definitely be interesting to see how it would work out.

[Andrew Lee]

Two things come to mind. One would be that for any submission there could be a smaller and streamlined/longer but more robust/etc version of commands for a recipe. How would alternative versions of commands be handled, would they all be shown under the recipe (problem to be solved) entry? I suppose alternatives could be debated in the comments for such recipes though that wouldn't be ideal for longer forms or for attachments like script files (but OTOH using the forums for discussions per-recipe could get messy with the multitude of potential combinations - edit: or not in reality? Idk, would depend how it went ).

Let's call Download YouTube video a "task", and youtube-dl <url> a "recipe". For each task, there could be multiple recipes submitted by different users. In fact, I have deliberately included 2 different recipes for the task Replace text in file.

So I could upvote or downvote competiting recipes, so the better ones float to the top. Conceivably, recipes below a certain threshold are hidden and can be expanded by a [Show all] button.

The other is given that with virtually any CLI program's basic arguments one could create a recipe what do you feel would direct users to only post 'problems that are worth solving', since I'd imagine with such constraints some users (*cough* from the comprehensive camp *cough* ) may want to post all default args under new recipes (since it could be argued that anything is worth solving to someone)... which functionally would be not dissimilar to the recent debate over Busybox's functions as separate entries, where such an alternative DB could be potentially filled with pages of recipes for the sake of covering everything.

Voting becomes very important here. Suppose I create a task called "Create directory" with the recipe mkdir <dirname>. Then I tell everyone about this new task I have created in the forum and pls go vote. If no one finds it interesting enough to upvote that single recipe (I hope!), then that task will stay private and not appear in normal listing and searches.

Also, your note about linking to the version that was successfully used is a great idea. Would a "note" at the end of the "recipe" be added as well? Could have something about why that version was used, ie. new arguments as you mentioned and/or possible issues with some versions that may cause corrupted outputs or such?

Yes, I think having an optional note for each recipe would be very useful to list important info such as version req, new args, recipe constraints, input format requirement and so on.

Also, I am thinking each recipe will have its own comment section that is collapsed initially but can be expanded at a click for discussion about that particular recipe. This will reuse the current database entry comment system, so you get the downvote function of comments as well.

Could the database be setup to work with either method or would that be too much extra effort?

The frontend and database structure would be quite different, though a lot of backend logic could be reused. For example, a lot of stuff like icons and screenshots which are not suitable for CLI will no longer be present. But scoring, voting, searching etc. in the backend could all be adapted to the new format.

You can think of recipes as existing database entries ("Note" will simply be some existing field for the entry, repurposed). Category => Task. A lot of GUI-specific fields in the database can be dropped. That's the database model. Frontend will be quite visually quite different, probably very textual, with AJAX thrown in where possible to make things more responsive (eg. typing "video" in the search bar will immediately list all tasks with the word "video" in them in a dropdown box). This makes sense because the search results will be constrained to that single field with pretty short and concise wordings, something that's a lot harder to do with the GUI site.

[vevy]

Hi, Andrew. Welcome to the forums!

Unfortunately, while your idea could be useful, we are afraid it may not fit well with our community. Maybe www.dostips.com/forum would be a better place for your idea!

Don't let that discourage you from doing useful stuff here! Looking forward to your contributions!

------------------------------------------------------------------------------------------------------------------

Now to business:

A few thoughts:

• If you are in the audience for a recipe, you fall in either one of two categories:
  1. You don't what tool to use for what you want to achieve (Batch / Find And Edit Lines in TXT file - Stack Overflow).
  2. You know the tool but don't how to make it do the task: (mp4 - Use ffmpeg to add text subtitles - Stack Overflow).

• In the first case, you don't have a solid thread in you hand as a starting point. You can, for example, ask for recommendations, search the web using descriptive keywords. If you have a specific term for the task, then better.

• For the second, you have the name of the tool. You can take tool-specific actions: reading the included help or the docs, asking for help, searching for help topics by others, watch tutorial videos, etc.

But how are users to find good tools for these recipes?

1. The thing is, "popular" tools aren't always the best ones. An they are certainly not the only ones worthy of being a candidate solution/recipe.
2. A user usually has a bunch of tools they know. Either the well-known ones or by exploring or through work or search or asking/recommendations, etc.
3. What my work has been revolving around for months is to gather the tools and assigning them the functions they perform.
4. If you know what tools can to task X, you can find the ones worthy of being in a recipe. Otherwise, we are missing out on a lot.
5. Maybe there is an unknown wrapper around FFmpeg that abstracts its syntax making the creation of different recipes a breeze. Maybe there is one that cares about features FFmpeg had to de-emphasize for greater compatibility/appeal to its big audience.
6. What I care about is providing the community (including myself) with the building tools and objective points of information with which such a solution can be integrated on a capable foundation. That's why I put so much emphasis on proper categorization and assigning attributes. Even though your idea wasn't specifically on my mind, consider the following:
   • The recipe page can be the top part of the individual use case pages. Contributed recipes then tools.
   • They can have their own database and UI that draws from the comprehensive database behind the scenes.
   • Recipes can be linked from a field of tools. You click a recipe to go to its page and see how that tool (and others) can be used.
     icacls has the recipe "Remove execution permission for all files in given folder". If you want to find alternatives to suggest a different recipe, you can find under icacls the use case "change file ACL permissions". Click it to find less-advertised alternatives like Repacls.

------------------------------------------------------------------------------------------------------------------

• You suggest solving the first by providing categorized tasks (how to do x) suggested by the users and solving the second by listing under those tasks, the appropriate syntax (or recipe) for the tools the user suggests for solving the task. At the heart of that is the idea of crowd-sourcing/democratizing the best solutions. While I am not sure there are enough involved users to better separate the good from the bad clearly enough (one L5 user can and will probably decide the placement of many recipes), I do think that this democratic way is the primary way to go.
• A good positive of your idea is that it would also expose which tools are simpler to use for a given task. I would like that.
• Even emphasizing appropriate versions can fit in that design (or pretty much any other to be honest).
• Also, you went full vevy describing the idea. You never go full vevy!

[vevy]

...since I'd imagine with such constraints some users (*cough* from the comprehensive camp *cough* ) may want to post all default args under new recipes (since it could be argued that anything is worth solving to someone)... which functionally would be not dissimilar to the recent debate over Busybox's functions as separate entries, where such an alternative DB could be potentially filled with pages of recipes for the sake of covering everything.

Hey!
BusyBox functions is pretty much in the same spirit of what Andrew is suggesting: solution-based categorization. If you click on the use case "extract strings from binary", you should find busybox.exe strings as an alternative to Sysinternals strings.exe.

An important clarification for Andrew and everyone: I am not suggesting forcing things on people. If the community doesn't want to see, e.g. batch files or subcommands in the database by default, then simply don't! Hide them by default! Just lay the foundation so that those who find these useful can find them.

though potentially it may be less of an issue or not one at all if searching/filtering/browsing were based around problems to be solved

THANK YOU! Good searching, filtering, and browsing (categorization) do help things, don't they?

[Andrew Lee]

Also, you went full vevy describing the idea. You never go full vevy!

You lost me here Care to explain?

[vevy]

Also, you went full vevy describing the idea. You never go full vevy!

You lost me here Care to explain?

Detailed description anticipating and pre-solving issues! Just joking!

[Midas]

I, for one, like the IDEA very much -- almost as much as the discussion it generated...

[Andrew Lee]

OK, I have put together a prototype of what I have in mind. Please visit cmd dot portablefreeware dot com to check it out. Let's try to not post links to the prototype in case the search engines pick them up.

Please note that this is currently a "storyboard" level prototype. None of the edit/voting/commenting functions actually work, only viewing and searching. The entries are coming from a real database though (manually entered using the DB backend, good for refining the scheme and figure out what is/is not practical/possible).

Points of interest:

- Under the task "Rename files using regular expressions", there are 3 public solutions, and 1 private. You can click through below to reveal the private one (and vote to make it public on a real site).

- Fake comments can be found under the "Convert AVI to MP4 losslessly" and "Normalize volume of M4A/MP4 file losslessly" solutions.

- The main listing lists new solutions by most recently posted. If a task has more than 1 solution, it will be shown in brackets (eg. "Rename files using regular expressions (3)"). Click through to see all solutions for that task.

- Where available, link to the forum topic can be found under the tool reference. eg. click through to references for "brename", "youtube-dl", "ffmpeg" etc.

- Version, license, extraction info are purposely left out. I just feel they are not so important in CLIs. Maybe I am wrong. Some of this info could potentially be added to the app reference.

- webfork asked if we will run into the same problem with use cases i.e. too many to list. My thinking is a task should only be added if it is something worth solving to you personally. So a "mkdir" task should likely not be added, or if added, not be upvoted. There is also another angle to it, which is when submitting a new CLI, maybe find a task that will showcase its essence and make it more likely to come up in searches. That should be sufficient, rather than exhaustively list all the tasks it is capable of.

I know this is coming from a totally different direction from the app-based database that we have so far, so please feel free to roast the idea. I have no attachment to the prototype at all, so pls don't think I'd be offended if you poke holes at it. I'd much rather find out early on that it's a dud, then spend a ton of time on it and find nobody is interested later on.

Thanks!

[Midas]

OK, great.

So I'll just do a bit of summing up before I add my own first impressions and suggestions.

1. This is (will become) what I'd call a solution based index -- being that problems solved takes precedence over the tool(s) used to solve them;

2. Regarding the issue addressed by webfork and Andrew in the last bullet-point above (and from the example given), I'd expressly call it the minimum relevance and difficulty threshold rule -- like the platform isn't really meant to exemplify basic program operation but, instead, targets how to use them to solve specific problems and the more non-immediately-evident they are, the better.

If we agree on those points, here's some notes about the preliminary mock-up:

- I expected to see a (mandatory?) plain language, if necessarily succinct (to be perfected over time), statement of the problem being handled in addition to the main header; it would help clarify said header and deal with ambiguities that are bound to arise in time and more complex cases; worst case scenario, it'd just contain a repetition of the main header but, be it as it may, I wouldn't worry too much about redundancy here.

- First and foremost, my initial impression was that "Solution" didn't feel relevant enough as a (secondary) header; I'd rather have something like a mention of the tool used as well as the permalink right there.

- Talking of permalinks, I'd would prefer if they'd be a little more fixed-length (and descriptive as well), based on a string or maybe a hash, like for example in Stack Exchange (e.g., https://stackoverflow.com/questions/8318911/) or YouTube (where https://www.youtube.com/watch?v=J7UwSVsiwzI and https://youtu.be/J7UwSVsiwzI are both valid).

- I really dig the "copy" button in the example box but, as platforms keep evolving and changing, I feel something more explicit like the forum "select all" generated by forum code tags (which also contain a permalink and will do precisely that) would also be welcome.

Code: Select all

some text

- Overall, I feel the layout should be somewhat simplified for easier readability/interactivity, although I have no ready-made suggestions for that. Any web-designers to the rescue?

[Andrew Lee]

- Overall, I feel the layout should be somewhat simplified for easier readability/interactivity, although I have no ready-made suggestions for that. Any web-designers to the rescue?

First off, I couldn't agree with you more on this. Even I found the layout and color combination somewhat lacking, but tried as I did, I couldn't improve on it There is a reason why we pay good money to web designers!

Before specific responses, I need a gut-feel response from you. Which approach do you like better? Tool-based, solutions-based, or neither (sufficient that we just discuss in the forum)?

2. Regarding the issue addressed by webfork and Andrew in the last bullet-point above (and from the example given), I'd expressly call it the minimum relevance and difficulty threshold rule -- like the platform isn't really meant to exemplify basic program operation but, instead, targets how to use them to solve specific problems and the more non-immediately-evident they are, the better.

That's the idea. For example, a simple rename ("ren") is probably not that interesting, but regex rename is, and the list of CLIs that let you do it.

- I expected to see a (mandatory?) plain language, if necessarily succinct (to be perfected over time), statement of the problem being handled in addition to the main header; it would help clarify said header and deal with ambiguities that are bound to arise in time and more complex cases; worst case scenario, it'd just contain a repetition of the main header but, be it as it may, I wouldn't worry too much about redundancy here.

If you could provide a simple mockup (even using tables or just plain-text), I think it will help greatly.

- First and foremost, my initial impression was that "Solution" didn't feel relevant enough as a (secondary) header; I'd rather have something like a mention of the tool used as well as the permalink right there.

I think I get what you are saying. I will makes some changes.

- Talking of permalinks, I'd would prefer if they'd be a little more fixed-length (and descriptive as well), based on a string or maybe a hash, like for example in Stack Exchange (e.g., https://stackoverflow.com/questions/8318911/) or YouTube (where https://www.youtube.com/watch?v=J7UwSVsiwzI and https://youtu.be/J7UwSVsiwzI are both valid).

So something like "/solution/rename+files+with+regex"?

- I really dig the "copy" button in the example box but, as platforms keep evolving and changing, I feel something more explicit like the forum "select all" generated by forum code tags (which also contain a permalink and will do precisely that) would also be welcome.

I just checked, and I don't think it's a permalink. It's Javascript code that highlights the text. So you can't really copy the link and give it to someone else. It won't work. So, I guess you are saying instead of "Copy", have a "Select all" function instead?

[Midas]

First off, I couldn't agree with you more on this. Even I found the layout and color combination somewhat lacking, but tried as I did, I couldn't improve on it There is a reason why we pay good money to web designers!

Fine. So we agree we disagree.

Before specific responses, I need a gut-feel response from you. Which approach do you like better? Tool-based, solutions-based, or neither (sufficient that we just discuss in the forum)?

I have no strong feeling either way so I'm willing to place my trust in your more experienced advice. I guess my final view will come from how easy it'll be to find anything and subsequently 'grok' the result...

If you could provide a simple mockup (even using tables or just plain-text), I think it will help greatly.

My design skills are pretty slim, too, but I'll see if I can come up with something... stay tuned.

I think I get what you are saying. I will makes some changes.

Great, I see you've done them already.

No permalink, though -- which should point to TPFC internal resources and not elsewhere, IMHO.

Because that will make maintenance simpler -- having an entry for a tool with all the required external links and then using that entry as a proxy everywhere needed.

And like in the regular database, external links should be marked and displayed in new tabs/windows.

So something like "/solution/rename+files+with+regex"?

Not really, because that way privileges readability over ease of use.

Please note how stackoverflow.com/questions/8318911/why-does-html-think-chucknorris-is-a-color and stackoverflow.com/questions/8318911/ both work, with the readable part being optional; the main point here is the fixed length of the ID.

I just checked, and I don't think it's a permalink. It's Javascript code that highlights the text. So you can't really copy the link and give it to someone else. It won't work. So, I guess you are saying instead of "Copy", have a "Select all" function instead?

You're right, of course. But the main point isn't the permalink but a different copy strategy, with different operational and visual feedback. I think it doesn't hurt to keep both.

[Andrew Lee]

Fine. So we agree we disagree.

Wait, I thought we both agree my design skill sucks?

My design skills are pretty slim, too, but I'll see if I can come up with something... stay tuned.

Less so the design bit, more the "succinct language" bit.

No permalink, though -- which should point to TPFC internal resources and not elsewhere, IMHO.

I am having trouble deciding where to point to. The small permalink icon that we have currently is more a link that can be copy-and-paste into a forum post. But it doesn't seem right to put that in the solution title because if you click through, it will just come back to the solution itself.

The alternative seems to point it to the app reference. But my idea was that it could be possible to have multiple apps involved in a solution, so the layout already caters for multiple references (under the "References" section). So again, I am stuck here.

Ideas?

Please note how stackoverflow.com/questions/8318911/why-does-html-think-chucknorris-is-a-color and stackoverflow.com/questions/8318911/ both work, with the readable part being optional; the main point here is the fixed length of the ID.

Ah, so "/?id=123456" and "/?id=000056" etc.? I guess I am still a little confused here. Besides aesthetics, what other benefit does this bring?

You're right, of course. But the main point isn't the permalink but a different copy strategy, with different operational and visual feedback. I think it doesn't hurt to keep both.

OK, got it. Should be an easy addition.

[Midas]

Less so the design bit, more the "succinct language" bit.

Oh, that. Above all, that will be a matter of our own editing skills. Once the design has been settled, maybe we should come up with some guidelines.

I am having trouble deciding where to point to. The small permalink icon that we have currently is more a link that can be copy-and-paste into a forum post. But it doesn't seem right to put that in the solution title because if you click through, it will just come back to the solution itself.

The alternative seems to point it to the app reference. But my idea was that it could be possible to have multiple apps involved in a solution, so the layout already caters for multiple references (under the "References" section). So again, I am stuck here.

Ideas?

Hmm, I see. Let's mull it over, see if anything shines through.

Ah, so "/?id=123456" and "/?id=000056" etc.? I guess I am still a little confused here. Besides aesthetics, what other benefit does this bring?

I may be alone here, but mainly memorability and easier visual checking.

I mean, when things are setup like this, the significant part is just a (small) part of the URL (with the remainder being invariant), so less info is required to reference/retrieve, and you can easily build automation on top (like scripting and such).

OK, got it. Should be an easy addition.

Great! I'm sure others will have more to add.