Ideas for my next freeware project

[JnLlnd]

Hello,

I just finished my first freeware project (CSV Buddy), posted here a few days ago.

I'll make sure I have time to maintain it, following suggestions or bug reports I may receive. But it is time for me to think about my next freeware project.

Is this the right forum to bounce such ideas I have or should I do it in the Chit-Chat area?

[donald]

How about a podcast suite?
Features could include RSS Podcast downloads with rules (both auto and manual), Audio cutting, Streaming (& stream recording), Playing MP3s. (Have not spotted built in audio cutting in any podcatcher)

Examples of these features in other apps.

Re: Podcast downloads
See Roberts Podcatcher This app did a never-ending download when auto download was allowed not fun but a common problem since no podcast software I have tested has a set of rules to go with auto-download.

Gpodder This app works great most of the time but can hang and is hard to use from some mp3 players also Windows has a hard time copying it.

Ziepod This app downloads to a specific path and duplicates the downloaded file/s have not found a relative path setting.

Re: Audio cutting
See AudioBook Cutter FE This app cuts audio files at regular intervals during silence. This is great for longer audio files some podcasts are 2-3 hours.

Re: Streaming
See Screamer Radio This app is great streamer and stream recorder, and it is easy to add streams.

Re: Playing MP3s
CoolPlayer This app can minimize down to a sliver and can be placed over the title-bar of a web-browser. (using the pre-installed skins)

[JnLlnd]

Thanks for your suggestion Donald. But, I already have some ideas that are more in continuity with my previous project.

In fact, I have two spin-off ideas from my CSV Buddy project. CSV Buddy is a Windows portable app that reads, edits and writes CSV files. A secondary feature of this app is to export the CSV data to various formats including HTML page.

My first idea is to focus the new app on HTML page creation. This "HTML generator" would read CSV files (with the same read options found in CSV Buddy), edit, sort data, etc. The main new feature would be an editor/wizard to support the creation of the HTML template. A template consists of three sections: page header (what precedes the list), the list itself (the rows of data) and the page footer. In the row template, variables are linked to fields in the list and are replaced with the field's content for each row. Other variables could be inserted in the header or footer (page title, source, dates, file name, etc.).

The other idea seems more of a challenge to me. In a nutshell, the app would scrape an HTML page containing a structured list, build a table with the data in this list and save it to a CSV format. This "HTML extractor" wizard would help analyze the HTML source to mark the list beginning and end, and to markup each field in the list. Of course, the licence agreement would state that this app must be used only to extract data legally available to the user.

I know these kinds of tools exist web-based or as part of a database or HTML editors. But I don't know if they exist in the world of portable software? And if there is an interest in it?

Any comment / expression of interest / ideas would be appreciated.

[Craunch]

My suggestion would be a program that can convert between CSV, JSON and XML. They are all intended to make it easier for the output from one program to be readable by another, but I keep finding that what one program outputs is one that the next program I want to use cannot read.

[tproli]

@ Craunch Have you tried this?
http://shancarter.github.io/mr-data-converter/

[Craunch]

@tproli thanks for that link, but it only converts from CSV to JSON etc, but not from JSON etc to CSV.

At the moment I am more interested in getting things into CSV which can be a bit messy. I could write this myself, but if I did it would almost certainly be written in Java as a command line utility which would exclude it from this site on two counts.

There is also http://www.json2csv.com/, but I dislike relying on internet sites to remain available if I need to do something. I much prefer having my software on my computer.

[Midas]

Hi there, JnLlnd. Nice of you to ask fellow TPFCers for input on your projects.

The other idea seems more of a challenge to me. In a nutshell, the app would scrape an HTML page containing a structured list, build a table with the data in this list and save it to a CSV format. This "HTML extractor" wizard would help analyze the HTML source to mark the list beginning and end, and to markup each field in the list. Of course, the licence agreement would state that this app must be used only to extract data legally available to the user.

I know these kinds of tools exist web-based or as part of a database or HTML editors. But I don't know if they exist in the world of portable software? And if there is an interest in it?

Any comment / expression of interest / ideas would be appreciated.

• I personally like this idea. One of the gaps I have felt in the world of portability is for a nice and easy Web scraper. Programs like Cintanotes come really close -- but when they turn commercial, the need to insure client satisfaction tends to get in the way of the KISS philosophy. I have used both the MS MHT and Mozilla's MAFF format for personal web archiving (in this respect, MixBrowser, a commercial Trident based front-end, once looked like it was advancing in the right direction; as does Mayakron's freeware CHreader -- also at Softpedia), but every ready made solution falls short one way or the other: no metadata (mainly, URL and date), or no HTML, or no export/import of collected data, etc...

My suggestion would be a program that can convert between CSV, JSON and XML. They are all intended to make it easier for the output from one program to be readable by another, but I keep finding that what one program outputs is one that the next program I want to use cannot read.

• This surely looks like an interesting feature set, especially if it could improve on the editing facilities of a tiny little pearl like XML Marker (sadly not free, not portable and not developed anymore). Free Virtual Tree Navigator has some ideas of its own to offer here, too, although it's close to unusable for the laymen...

[JnLlnd]

@Craunch: Have you tried the XML export feature in CSV Buddy? It does convert your CSV data to the XML format. You must ensure that field names and field data comply with the rules of XML syntax.

This simple example, where each record has two fields named "Field1" and "Field2", shows the XML output format:

<?xml version='1.0'?>
<XMLExport>
<Record>
<Field1>Value Row 1 Col 1</Field1>
<Field2>Value Row 1 Col 2</Field1>
</Record>
<Record>
<Field1>Value Row 2 Col 1</Field1>
<Field2>Value Row 2 Col 2</Field1>
</Record>
</XMLExport>

Not enough time to read/reply to other comments. I'll be back later. Have a good day!

[Craunch]

@JnLlnd CSV Buddy is a nice piece of work, and I must admit to not having spotted the XML export feature which could come in useful. But I need to get JSON or XML data into CSV in the first place. This shouldn't be particularly difficult, but it does get a bit messy if the JSON/XML data has any sort of nested structure. A good example of this would be a Firefox bookmark backup file, which has a tree structure in JSON.

Keep up the good work, see you later.

[JnLlnd]

@Craunch: I never used JSON before but adding "Export to JSON" to CSV Buddy's export tab seems quite easy (a few hours of work). But the opposite (JSON -> CSV), as you mention, is more complex if not impossible in the case of hierarchical data, CSV being so flat...

Converting HTML, XML or JSON to CSV are all variations of the same challenge: parsing semi-freeform or hierarchical data. It means analyzing the source to get field names, finding record delimiters, and start/end of each fields. Interesting enough. If I go in this direction, I would write a text-based parser/wizard (ie you would work in source code) to stay in the simple side of things (but this would target users with some knowledge of the source language). There are very sophisticated web-based application that support WYSIWYG parsing (like [url="http://www.automationanywhere.com/solut ... ataExt.htm"]Automation Everywhere]) but they cost much $$$.

@Midas: Thank you for your suggestions and references. I'll look at the examples you mention and come back to you after I make my mind on them. But a question: are you interested in web scraping for any kind of content or specifically for structured data (that could be saved as CSV, for example)?

No interest yet for an HTML generator from s CSV table?

[Craunch]

Hierarchical JSON --> CSV is not impossible - you just need a good flattening algorithm

Let me give an example based on something you use every day, the Windows filing system.

Currently the directory that I downloaded CSVBuddy to contains 1 files and a subdirectory with 2 files:

Code: Select all

csvbuddy.zip
CSVBuddy-0_8_1-BETA\CSVBuddy-0_8_1-BETA-32-bits.exe
CSVBuddy-0_8_1-BETA\CSVBuddy-0_8_1-BETA-64-bits.exe

If I give their full pathnames they are:

Code: Select all

E:\Craunch\downloads\CSVBuddy\
E:\Craunch\downloads\CSVBuddy\CSVBuddy-0_8_1-BETA\CSVBuddy-0_8_1-BETA-32-bits.exe
E:\Craunch\downloads\CSVBuddy\CSVBuddy-0_8_1-BETA\CSVBuddy-0_8_1-BETA-64-bits.exe

Hierarchically this can be represented as, and this is similar to how the data would be represented in JSON:

Code: Select all

Drive = E:
    Dir1 = Craunch
        Dir2 = Downloads
            Dir3 = CSVBuddy
                Filename = csvbuddy.zip
                Dir4 = CSVBuddy-0_8_1-BETA
                    Filename = CSVBuddy-0_8_1-BETA-32-bits.exe
                    Filename = CSVBuddy-0_8_1-BETA-64-bits.exe

In tabular form this could appear as:

Code: Select all

Drive Dir1    Dir2      Dir3     Dir4                Filename
E:    Craunch Downloads CSVBuddy                     csvbuddy.zip
E:    Craunch Downloads CSVBuddy CSVBuddy-0_8_1-BETA CSVBuddy-0_8_1-BETA-32-bits.exe
E:    Craunch Downloads CSVBuddy CSVBuddy-0_8_1-BETA CSVBuddy-0_8_1-BETA-64-bits.exe

or as CSV:

Code: Select all

Drive,Dir1,Dir2,Dir3,Dir4,Filename
E:,Craunch,Downloads,CSVBuddy,,csvbuddy.zip
E:,Craunch,Downloads,CSVBuddy,CSVBuddy-0_8_1-BETA,CSVBuddy-0_8_1-BETA-32-bits.exe
E:,Craunch,Downloads,CSVBuddy,CSVBuddy-0_8_1-BETA,CSVBuddy-0_8_1-BETA-64-bits.exe

I hope this has given you something to think about

[Midas]

@Midas: [...] But a question: are you interested in web scraping for any kind of content or specifically for structured data (that could be saved as CSV, for example)?

• Short answer is: general content -- but including structured data.
  
  Longer rationale: my main concern has always been keeping data generally accessible and dodging application/format lock-in; I just can't stand the notion that anything a user has worked on at some point in time might be rendered irretrievable/inaccessible because software developers thought that their program functionality could/should override user data.
  
  So interoperability, normalization/standardization, refresh and migration (or entry/exit) paths are always one of my main concerns when evaluating software. CSV, as a long-standing open data standard, plays a part there, as do most text based (human readable) data formats. It's also the main drive of my interest in portable software.
  
  By way of example, I still have email messages in my system(s) that are over ten years old...

[JnLlnd]

• Short answer is: general content -- but including structured data.

If I develop an app, it would focus on structured data. But I understand ang agree with your concerns about data accessibility.

Hierarchical JSON --> CSV is not impossible - you just need a good flattening algorithm
...
I hope this has given you something to think about

Good point! Notes taken

[guinness]

Good to see a new developer amongst us.

[JnLlnd]

Thank you guiness. I took a look at your site. I'll give a try to your installer for sure.